Training Over-parameterized Models with Non-decomposable Objectives

1 Introduction . The misclassification error is the canonical performance measure in most treatments of classification problems [ 92 ] . While appealing in its simplicity , practical machine learning applications often come with more complex and nuanced design goals . For example , these may include minimizing the worst-case error across all classes [ 64 ] , satisfying a given precision or recall target [ 106 ] , enforcing minimal coverage for minority classes [ 31 ] , or imposing group-fairness constraints [ 105 ] . Unlike the misclassification error , such objectives are non-decomposable , i.e. , they can not be expressed as a sum of losses over individual samples . This poses a non-trivial optimization challenge , which is typically addressed by reducing the problem into a sequence of cost-sensitive learning tasks [ 12 , 25 , 4 , 16 , 71 ] . Such reduction-based approaches have been successfully employed in many open-source libraries [ 3 , 2 , 1 ] to provide drop-in replacements for standard loss functions . In this paper , we point out that the standard approach of re-weighting the training loss to incorporate label costs can produce unsatisfactory results when used with high capacity models , which are often trained to memorize the training data . Such over-parameterized models are frequently encountered in the use of modern neural networks , and have been the subject of considerable recent study [ 107 , 73 , 66 ] . As a remedy , we provide new calibrated losses for cost-sensitive learning that are better equipped at training over-parameterized models to optimize non-decomposable metrics , and demonstrate their effectiveness on benchmark image classification tasks . Our main contributions are as follows : ( i ) we illustrate the pitfalls of using loss re-weighting in over-parameterized settings , particularly with diagonal class weights ( Section 3 ) . ( ii ) we propose new logit-adjusted losses for cost-sensitive learning for both diagonal and nondiagonal gain matrices , and show that they are calibrated ( Section 4 ) . ( iii ) we demonstrate that our losses provide significant gains over loss re-weighting in training ResNet models to optimize worst-case recall and to enforce coverage constraints ( Section 5 ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( iv ) we show that the proposed approach compares favorably to post-hoc correction strategies , and can be further improved by distilling scores from a teacher model ( Section 6 ) . 1.1 Related Work . We cover prior work on class-imbalanced learning , cost-sensitive learning , and complex metrics . Learning under class imbalance . A common setting where metrics beyond the misclassification error have been studied is the problem of class imbalance [ 49 , 11 , 33 ] . Here , the label distribution P ( y ) is skewed , and one seeks to ensure that performance on rare classes do not overly suffer . To this end , common metrics include the average of per-class recalls ( also known as the balanced accuracy ) [ 7 , 62 ] , quadratic mean of per-class accuracies [ 51 ] , and the F-score [ 54 ] . A panoply of different techniques have been explored for this problem , spurred in part by recent interest in the specific context of neural networks ( where the problem is termed long-tail learning ) [ 91 , 9 ] , with the focus largely on optimizing balanced accuracy . An exhaustive survey is beyond the scope of this paper ( see , e.g , . [ 33 , 39 ] ) , but we may roughly identify three main strands of work : data modification [ 48 , 11 , 93 , 103 , 108 ] , loss modification [ 107 , 18 , 10 , 88 , 37 , 79 , 97 , 63 , 21 , 45 , 94 ] , and prediction modification [ 28 , 43 , 76 , 60 , 13 , 40 , 110 ] . A related recent thread seeks to improve performance on rare subgroups [ 82 , 83 , 86 ] , as a means to ensure model fairness [ 24 ] . Amongst loss modification techniques , two strategies are of particular relevance . The first is reweighting techniques [ 100 , 65 , 18 ] . As has been noted [ 55 , 96 , 61 ] ( and as we shall subsequently verify ) , these may have limited effect on training samples that are perfectly separable under a given model class ( as is the case with overparameterized models ) . The second is margin techniques , which enforce a class-dependent margin in the loss [ 10 , 88 , 79 , 63 , 94 ] . This seeks to ensure that rare classes are separated with greater confidence , to account for the higher uncertainty in their decision boundary . Cost-sensitive learning . Cost-sensitive learning is a classical extension of standard multiclass classification , wherein errors on different classes incur different costs [ 26 , 23 , 85 ] . Strategies for this problem largely mirror those for class imbalance , which can be seen as imposing a particular set of costs dependent on the label frequencies [ 58 ] . In particular , loss modification techniques based on asymmetric weights [ 57 , 98 , 104 , 5 , 19 , 22 , 85 , 112 ] and margins [ 61 , 44 , 36 , 45 ] have been explored , with the latter proving useful on separable problems . Amongst these , Lin et al . [ 57 ] handle a general cost matrix for multiclass problems , but require the class scores to sum to 0 , a constraint that might be difficult to impose with neural networks . Khan et al . [ 44 ] propose a multiclass loss similar to the logit adjustment idea used in this paper , but only handle a specific type of cost matrix ( e.g. , in their setup , the default cost matrix is a matrix of all 1s ) . The standard multiclass loss of Crammer and Singer [ 17 ] can also be extended to handle a general cost matrix [ 90 ] , but unfortunately is not calibrated [ 78 ] . Complex metrics . There has been much work on extending the class imbalance literature to handle more complex metrics and to impose data-dependent constraints . These methods can again be divided into loss modifications [ 38 , 74 , 41 , 69 , 42 , 31 , 25 , 4 , 16 , 71 , 27 , 59 , 50 ] and post-hoc prediction modifications [ 102 , 68 , 46 , 70 , 72 , 32 , 20 , 67 , 101 , 89 ] , and differ in how they decompose the problem into simpler cost-sensitive formulations ( see [ 71 ] for a unified treatment of common reduction strategies ) . Most of these papers experiment with simple models , with the exception of Sanyal et al . [ 84 ] , who use re-weighting strategies to train shallow DNNs , Kumar et al . [ 50 ] , who train CNNs on binary-labeled problems , and Eban et al . [ 25 ] , who train InceptionNets to optimize an AUC-based ranking metric . There have also been other attempts at training neural networks with ranking metrics [ 87 , 35 , 8 , 77 ] . In contrast , our focus is on handling a general family of evaluation metrics popular in multiclass classification problems . 2 Non-decomposable Objectives . Notations . Let [ m ] = { 1 , . . . , m } . Let m be the ( m 1 ) -dimensional simplex with m coordinates . Let 1m 2 Rm denote an all 1s vector of size m , and diag ( u1 , . . . , um ) denote a m ⇥m diagonal matrix with diagonal elements u1 , . . . , um . For vectors a , b 2 Rm , a/b denotes element-wise division . Let softmaxi ( s ) = exp ( si ) Pm j=1 exp ( sj ) denote a softmax transformation of scores s 2 Rm . Consider a multiclass problem with an instance space X ✓ Rd and a label space Y = [ m ] . Let D denote the underlying data distribution over X ⇥ [ m ] , DX denote the marginal distribution over the Algorithm 1 Reductions-based Algorithm for Maximizing Worst-case Recall ( 1 ) Inputs : Training set S , Validation set Sval , Step-size ! 2 R+ , Class priors ⇡ , CS-loss ` G Initialize : Classifier h0 , Multipliers 0 2 m for t = 0 to T 1 do Update : t+1i = t i exp ⇣ ! bCii [ ht ] ⇡i ⌘ , 8i , where bCij [ h ] = 1|Sval| P ( x , y ) 2Sval 1 ( y = i , h ( x ) = j ) t+1i = t+1iPm j=1 t+1 j , 8i G = diag ( t+11 /⇡1 , . . . , t+1m /⇡m ) Cost-sensitive Learning ( CSL ) : st+1 2 argmins 1 |S| P ( x , y ) 2S ` G ( y , s ( x ) ) // Replaced by few steps of SGD ht+1 ( x ) 2 argmaxi2 [ m ] s t+1 i ( x ) , 8x end for return hT instances X , and ⇡i = P ( y = i ) denote the class priors . We will use pi ( x ) = P ( y = i|x ) to denote the conditional-class probability for instance x . Our goal is to learn a classifier h : X ! [ m ] , and will measure its performance in terms of its confusion matrix C [ h ] , where Cij [ h ] = E ( x , y ) ⇠D [ 1 ( y = i , h ( x ) = j ) ] . Complex learning problems . In the standard setup , one is often interested in maximizing the overall classification accuracy acc [ h ] = P i Cii [ h ] . However , in many practical settings , one may care about other metrics such as the recall for class i , reci [ h ] = Cii [ h ] ⇡i , the precision on class i , preci [ h ] = Cii [ h ] P j Cji , or the proportion of predictions made on class i , i.e . the coverage , covi [ h ] = P j Cji . Below are common examples of real-world design goals based on these metrics . Example 1 ( Maximizing worst-case recall ) . In class-imbalanced settings , where the classifier tends to generalize poorly on the rare classes , we may wish to change the training objective to directly maximize the minimum recall across all classes [ 12 ] : max h min i2 [ m ] Cii [ h ] ⇡i . ( 1 ) Example 2 ( Constraining per-class coverage ) . Another consequence of label imbalance could be that the proportion of predictions that the classifier makes for the tail classes is lower than the actual prevalence of that class . The following learning problem seeks to maximize the average recall while explicitly constraining the classifier ’ s coverage for class j to be at least 95 % of its prior ⇡j [ 16 , 71 ] : max h 1 m mX i=1 Cii [ h ] ⇡i s.t . mX i=1 Cij [ h ] 0.95⇥ ⇡j , 8j 2 [ m ] . ( 2 ) Other examples include constraints on recall and precision ( see Table 1 ) , the F-measure , and the area under the ROC and PR curves , as well as fairness constraints like equal opportunity , which are computed on group-specific versions of the confusion matrix [ 25 , 71 , 15 , 32 , 4 ] . Reduction to cost-sensitive learning . All the problems described above are non-decomposable , in the sense that , they can not be written as minimization of a sum of errors on individual examples , and hence can not be tackled using common off-the-shelf learning algorithms . The dominant approach for solving such problems is to formulate a sequence of cost-sensitive objectives , which do take the form of an average of training errors [ 70 , 12 , 4 , 16 ] . For example , the robust learning problem in ( 1 ) can be equivalently re-written as the following saddle-point optimization problem : max h min 2 m mX i=1 i Cii [ h ] ⇡i , ( 3 ) where i is a multiplier for class i . One can then find a saddle-point for ( 3 ) by jointly minimizing the weighted objective over 2 m ( using e.g . exponentiated-gradient updates ) and maximizing the objective over h. Notice that for a fixed , the optimization over h is a cost-sensitive ( or a gain-weighted ) learning problem : max h X i , j GijCij [ h ] , ( 4 ) where Gii = i⇡i and Gij = 0 , 8i 6= j are the rewards associated with predicting class j when the true class is i . We will refer to G 2 Rm⇥m as the “ gain ” matrix , which in this case is diagonal . In practice , the cost-sensitive learning problem in ( 4 ) is usually replaced with multiple steps of gradient descent , interleaved with the updates on . Algorithm 1 provides an outline of this procedure . Similar to ( 1 ) , the constrained learning problem in ( 2 ) can be written as an equivalent ( Lagrangian ) min-max problem with Lagrange multipliers 2 Rm+ : max h min 2Rm+ 1 m mX i=1 1 ⇡i Cii [ h ] + mX j=1 j ⇣ mX i=1 Cij [ h ] 0.95⇡j ⌘ . One can find a saddle-point for this problem by jointly minimizing the Lagrangian over 2 Rm+ ( using e.g . gradient updates ) and maximizing it over h , with the maximization over h taking the form of a cost-sensitive learning problem . In this case , the gain matrix G is non-diagonal and is given by Gii = 1 m⇡i + i , 8i and Gij = j , 8i 6= j . See Table 1 for the form of the gain matrix for other common constraints . In Appendix A , we provide an outline of this procedure ( Algorithm 2 ) , and also elaborate on how this reduction-based approach can be extended to optimize other common evaluation metrics such F-measure [ 25 , 71 ] . Cost-sensitive losses . A standard approach for solving the cost-sensitive learning problem in ( 4 ) is to use a surrogate loss function ` : [ m ] ⇥ Rm ! R+ that takes a label y and a m-dimensional score u 2 Rm , and outputs a real value ` ( y , u ) . One would then minimize the expected loss L ( s ) = Ex , y [ ` ( y , s ( x ) ) ] over a class of scoring function s : X ! Rm that map each instance to an m-dimensional score . The final classifier h⇤ can then be obtained from the learned scoring function s⇤ by taking an argmax of its predicted scores , i.e . by constructing h⇤ ( x ) 2 argmaxi2 [ m ] s⇤i ( x ) . In practice , we are provided a finite sample S = { ( x1 , y1 ) , . . . , ( xn , yn ) } drawn from D , and will seek to minimize the average loss on S , given by bL ( s ) = 1 |S| P ( x , y ) 2S ` ( y , s ( x ) ) . We will also assume access to a held-out validation sample Sval = { ( x1 , y1 ) , . . . , ( xnval , ynval ) } . One common sanity check for a “ good ” loss function is to confirm that it is classification calibrated ( or Fisher consistent ) for the learning problem of interest [ 6 ] . Loosely speaking , a surrogate loss ` is classification calibrated if the minimizers of the conditional risk Ey|x ( ` ( y , s ( x ) ) ) over all scoring functions s : X ! Rm coincide with the Bayes-optimal prediction for x . For the cost-sensitive objective in ( 4 ) , the Bayes-optimal classifier is given below [ 53 ] . Proposition 1 . The optimal classifier for ( 4 ) for a general gain matrix G 2 Rm⇥m is of the form : h⇤ ( x ) 2 argmaxy2 [ m ] Pm i=1 Giy pi ( x ) = argmaxy2 [ m ] ( G > p ( x ) ) y . All proofs can be found in Appendix B .

Summary: The paper considers multiclass classification with "non-decomposeable" objectives: that is, objectives that are not the sum of individual terms over sample points. The focus is on problems that can be reformulated in terms of a single saddle point structure. These are often recast as a sequence of cost-sensitive problems, but this strategy can be bad when the learning method applied at each step tends to achieve zero error. The authors show that considering instead a sequence of problems with non-standard losses losses leads to significantly better empirical results. A partial justification for the non-standard losses is that they are calibrated (so their optima correspond to optimal classifiers).