Training Over-parameterized Models with Non-decomposable Objectives

1 Introduction . The misclassification error is the canonical performance measure in most treatments of classification problems [ 92 ] . While appealing in its simplicity , practical machine learning applications often come with more complex and nuanced design goals . For example , these may include minimizing the worst-case error across all classes [ 64 ] , satisfying a given precision or recall target [ 106 ] , enforcing minimal coverage for minority classes [ 31 ] , or imposing group-fairness constraints [ 105 ] . Unlike the misclassification error , such objectives are non-decomposable , i.e. , they can not be expressed as a sum of losses over individual samples . This poses a non-trivial optimization challenge , which is typically addressed by reducing the problem into a sequence of cost-sensitive learning tasks [ 12 , 25 , 4 , 16 , 71 ] . Such reduction-based approaches have been successfully employed in many open-source libraries [ 3 , 2 , 1 ] to provide drop-in replacements for standard loss functions . In this paper , we point out that the standard approach of re-weighting the training loss to incorporate label costs can produce unsatisfactory results when used with high capacity models , which are often trained to memorize the training data . Such over-parameterized models are frequently encountered in the use of modern neural networks , and have been the subject of considerable recent study [ 107 , 73 , 66 ] . As a remedy , we provide new calibrated losses for cost-sensitive learning that are better equipped at training over-parameterized models to optimize non-decomposable metrics , and demonstrate their effectiveness on benchmark image classification tasks . Our main contributions are as follows : ( i ) we illustrate the pitfalls of using loss re-weighting in over-parameterized settings , particularly with diagonal class weights ( Section 3 ) . ( ii ) we propose new logit-adjusted losses for cost-sensitive learning for both diagonal and nondiagonal gain matrices , and show that they are calibrated ( Section 4 ) . ( iii ) we demonstrate that our losses provide significant gains over loss re-weighting in training ResNet models to optimize worst-case recall and to enforce coverage constraints ( Section 5 ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( iv ) we show that the proposed approach compares favorably to post-hoc correction strategies , and can be further improved by distilling scores from a teacher model ( Section 6 ) . 1.1 Related Work . We cover prior work on class-imbalanced learning , cost-sensitive learning , and complex metrics . Learning under class imbalance . A common setting where metrics beyond the misclassification error have been studied is the problem of class imbalance [ 49 , 11 , 33 ] . Here , the label distribution P ( y ) is skewed , and one seeks to ensure that performance on rare classes do not overly suffer . To this end , common metrics include the average of per-class recalls ( also known as the balanced accuracy ) [ 7 , 62 ] , quadratic mean of per-class accuracies [ 51 ] , and the F-score [ 54 ] . A panoply of different techniques have been explored for this problem , spurred in part by recent interest in the specific context of neural networks ( where the problem is termed long-tail learning ) [ 91 , 9 ] , with the focus largely on optimizing balanced accuracy . An exhaustive survey is beyond the scope of this paper ( see , e.g , . [ 33 , 39 ] ) , but we may roughly identify three main strands of work : data modification [ 48 , 11 , 93 , 103 , 108 ] , loss modification [ 107 , 18 , 10 , 88 , 37 , 79 , 97 , 63 , 21 , 45 , 94 ] , and prediction modification [ 28 , 43 , 76 , 60 , 13 , 40 , 110 ] . A related recent thread seeks to improve performance on rare subgroups [ 82 , 83 , 86 ] , as a means to ensure model fairness [ 24 ] . Amongst loss modification techniques , two strategies are of particular relevance . The first is reweighting techniques [ 100 , 65 , 18 ] . As has been noted [ 55 , 96 , 61 ] ( and as we shall subsequently verify ) , these may have limited effect on training samples that are perfectly separable under a given model class ( as is the case with overparameterized models ) . The second is margin techniques , which enforce a class-dependent margin in the loss [ 10 , 88 , 79 , 63 , 94 ] . This seeks to ensure that rare classes are separated with greater confidence , to account for the higher uncertainty in their decision boundary . Cost-sensitive learning . Cost-sensitive learning is a classical extension of standard multiclass classification , wherein errors on different classes incur different costs [ 26 , 23 , 85 ] . Strategies for this problem largely mirror those for class imbalance , which can be seen as imposing a particular set of costs dependent on the label frequencies [ 58 ] . In particular , loss modification techniques based on asymmetric weights [ 57 , 98 , 104 , 5 , 19 , 22 , 85 , 112 ] and margins [ 61 , 44 , 36 , 45 ] have been explored , with the latter proving useful on separable problems . Amongst these , Lin et al . [ 57 ] handle a general cost matrix for multiclass problems , but require the class scores to sum to 0 , a constraint that might be difficult to impose with neural networks . Khan et al . [ 44 ] propose a multiclass loss similar to the logit adjustment idea used in this paper , but only handle a specific type of cost matrix ( e.g. , in their setup , the default cost matrix is a matrix of all 1s ) . The standard multiclass loss of Crammer and Singer [ 17 ] can also be extended to handle a general cost matrix [ 90 ] , but unfortunately is not calibrated [ 78 ] . Complex metrics . There has been much work on extending the class imbalance literature to handle more complex metrics and to impose data-dependent constraints . These methods can again be divided into loss modifications [ 38 , 74 , 41 , 69 , 42 , 31 , 25 , 4 , 16 , 71 , 27 , 59 , 50 ] and post-hoc prediction modifications [ 102 , 68 , 46 , 70 , 72 , 32 , 20 , 67 , 101 , 89 ] , and differ in how they decompose the problem into simpler cost-sensitive formulations ( see [ 71 ] for a unified treatment of common reduction strategies ) . Most of these papers experiment with simple models , with the exception of Sanyal et al . [ 84 ] , who use re-weighting strategies to train shallow DNNs , Kumar et al . [ 50 ] , who train CNNs on binary-labeled problems , and Eban et al . [ 25 ] , who train InceptionNets to optimize an AUC-based ranking metric . There have also been other attempts at training neural networks with ranking metrics [ 87 , 35 , 8 , 77 ] . In contrast , our focus is on handling a general family of evaluation metrics popular in multiclass classification problems . 2 Non-decomposable Objectives . Notations . Let [ m ] = { 1 , . . . , m } . Let m be the ( m 1 ) -dimensional simplex with m coordinates . Let 1m 2 Rm denote an all 1s vector of size m , and diag ( u1 , . . . , um ) denote a m ⇥m diagonal matrix with diagonal elements u1 , . . . , um . For vectors a , b 2 Rm , a/b denotes element-wise division . Let softmaxi ( s ) = exp ( si ) Pm j=1 exp ( sj ) denote a softmax transformation of scores s 2 Rm . Consider a multiclass problem with an instance space X ✓ Rd and a label space Y = [ m ] . Let D denote the underlying data distribution over X ⇥ [ m ] , DX denote the marginal distribution over the Algorithm 1 Reductions-based Algorithm for Maximizing Worst-case Recall ( 1 ) Inputs : Training set S , Validation set Sval , Step-size ! 2 R+ , Class priors ⇡ , CS-loss ` G Initialize : Classifier h0 , Multipliers 0 2 m for t = 0 to T 1 do Update : t+1i = t i exp ⇣ ! bCii [ ht ] ⇡i ⌘ , 8i , where bCij [ h ] = 1|Sval| P ( x , y ) 2Sval 1 ( y = i , h ( x ) = j ) t+1i = t+1iPm j=1 t+1 j , 8i G = diag ( t+11 /⇡1 , . . . , t+1m /⇡m ) Cost-sensitive Learning ( CSL ) : st+1 2 argmins 1 |S| P ( x , y ) 2S ` G ( y , s ( x ) ) // Replaced by few steps of SGD ht+1 ( x ) 2 argmaxi2 [ m ] s t+1 i ( x ) , 8x end for return hT instances X , and ⇡i = P ( y = i ) denote the class priors . We will use pi ( x ) = P ( y = i|x ) to denote the conditional-class probability for instance x . Our goal is to learn a classifier h : X ! [ m ] , and will measure its performance in terms of its confusion matrix C [ h ] , where Cij [ h ] = E ( x , y ) ⇠D [ 1 ( y = i , h ( x ) = j ) ] . Complex learning problems . In the standard setup , one is often interested in maximizing the overall classification accuracy acc [ h ] = P i Cii [ h ] . However , in many practical settings , one may care about other metrics such as the recall for class i , reci [ h ] = Cii [ h ] ⇡i , the precision on class i , preci [ h ] = Cii [ h ] P j Cji , or the proportion of predictions made on class i , i.e . the coverage , covi [ h ] = P j Cji . Below are common examples of real-world design goals based on these metrics . Example 1 ( Maximizing worst-case recall ) . In class-imbalanced settings , where the classifier tends to generalize poorly on the rare classes , we may wish to change the training objective to directly maximize the minimum recall across all classes [ 12 ] : max h min i2 [ m ] Cii [ h ] ⇡i . ( 1 ) Example 2 ( Constraining per-class coverage ) . Another consequence of label imbalance could be that the proportion of predictions that the classifier makes for the tail classes is lower than the actual prevalence of that class . The following learning problem seeks to maximize the average recall while explicitly constraining the classifier ’ s coverage for class j to be at least 95 % of its prior ⇡j [ 16 , 71 ] : max h 1 m mX i=1 Cii [ h ] ⇡i s.t . mX i=1 Cij [ h ] 0.95⇥ ⇡j , 8j 2 [ m ] . ( 2 ) Other examples include constraints on recall and precision ( see Table 1 ) , the F-measure , and the area under the ROC and PR curves , as well as fairness constraints like equal opportunity , which are computed on group-specific versions of the confusion matrix [ 25 , 71 , 15 , 32 , 4 ] . Reduction to cost-sensitive learning . All the problems described above are non-decomposable , in the sense that , they can not be written as minimization of a sum of errors on individual examples , and hence can not be tackled using common off-the-shelf learning algorithms . The dominant approach for solving such problems is to formulate a sequence of cost-sensitive objectives , which do take the form of an average of training errors [ 70 , 12 , 4 , 16 ] . For example , the robust learning problem in ( 1 ) can be equivalently re-written as the following saddle-point optimization problem : max h min 2 m mX i=1 i Cii [ h ] ⇡i , ( 3 ) where i is a multiplier for class i . One can then find a saddle-point for ( 3 ) by jointly minimizing the weighted objective over 2 m ( using e.g . exponentiated-gradient updates ) and maximizing the objective over h. Notice that for a fixed , the optimization over h is a cost-sensitive ( or a gain-weighted ) learning problem : max h X i , j GijCij [ h ] , ( 4 ) where Gii = i⇡i and Gij = 0 , 8i 6= j are the rewards associated with predicting class j when the true class is i . We will refer to G 2 Rm⇥m as the “ gain ” matrix , which in this case is diagonal . In practice , the cost-sensitive learning problem in ( 4 ) is usually replaced with multiple steps of gradient descent , interleaved with the updates on . Algorithm 1 provides an outline of this procedure . Similar to ( 1 ) , the constrained learning problem in ( 2 ) can be written as an equivalent ( Lagrangian ) min-max problem with Lagrange multipliers 2 Rm+ : max h min 2Rm+ 1 m mX i=1 1 ⇡i Cii [ h ] + mX j=1 j ⇣ mX i=1 Cij [ h ] 0.95⇡j ⌘ . One can find a saddle-point for this problem by jointly minimizing the Lagrangian over 2 Rm+ ( using e.g . gradient updates ) and maximizing it over h , with the maximization over h taking the form of a cost-sensitive learning problem . In this case , the gain matrix G is non-diagonal and is given by Gii = 1 m⇡i + i , 8i and Gij = j , 8i 6= j . See Table 1 for the form of the gain matrix for other common constraints . In Appendix A , we provide an outline of this procedure ( Algorithm 2 ) , and also elaborate on how this reduction-based approach can be extended to optimize other common evaluation metrics such F-measure [ 25 , 71 ] . Cost-sensitive losses . A standard approach for solving the cost-sensitive learning problem in ( 4 ) is to use a surrogate loss function ` : [ m ] ⇥ Rm ! R+ that takes a label y and a m-dimensional score u 2 Rm , and outputs a real value ` ( y , u ) . One would then minimize the expected loss L ( s ) = Ex , y [ ` ( y , s ( x ) ) ] over a class of scoring function s : X ! Rm that map each instance to an m-dimensional score . The final classifier h⇤ can then be obtained from the learned scoring function s⇤ by taking an argmax of its predicted scores , i.e . by constructing h⇤ ( x ) 2 argmaxi2 [ m ] s⇤i ( x ) . In practice , we are provided a finite sample S = { ( x1 , y1 ) , . . . , ( xn , yn ) } drawn from D , and will seek to minimize the average loss on S , given by bL ( s ) = 1 |S| P ( x , y ) 2S ` ( y , s ( x ) ) . We will also assume access to a held-out validation sample Sval = { ( x1 , y1 ) , . . . , ( xnval , ynval ) } . One common sanity check for a “ good ” loss function is to confirm that it is classification calibrated ( or Fisher consistent ) for the learning problem of interest [ 6 ] . Loosely speaking , a surrogate loss ` is classification calibrated if the minimizers of the conditional risk Ey|x ( ` ( y , s ( x ) ) ) over all scoring functions s : X ! Rm coincide with the Bayes-optimal prediction for x . For the cost-sensitive objective in ( 4 ) , the Bayes-optimal classifier is given below [ 53 ] . Proposition 1 . The optimal classifier for ( 4 ) for a general gain matrix G 2 Rm⇥m is of the form : h⇤ ( x ) 2 argmaxy2 [ m ] Pm i=1 Giy pi ( x ) = argmaxy2 [ m ] ( G > p ( x ) ) y . All proofs can be found in Appendix B .

Summary: The author proposed a new approach for optimizing non-decomposable metrics that focuses on over-parameterized models (i.e., deep learning models). Rather than performing reduction and reweighting as in the traditional models for non-decomposable metrics, the proposed model modifies the logits of the over-parameterized models. The author also shows that the logit modifications are calibrated with respect to the original metric. Finally, the author shows the effectiveness of the proposed models on CIFAR and TIN datasets.