Label-Imbalanced and Group-Sensitive Classification under Overparameterization

1 Introduction . 1.1 Motivation and contributions . Equitable learning in the presence of data imbalances is a classical machine learning ( ML ) problem , but one with increasing importance as ML decisions are adapted in increasingly more complex applications directly involving people [ 3 ] . Two common types of imbalances are those appearing in label-imbalanced and group-sensitive classification . In the first type , examples from a target class are heavily outnumbered by examples from the rest of the classes . The standard metric of average misclassification error is insensitive to such imbalances and among several classical alternatives the balanced error is a widely used metric . In the second type , the broad goal is to ensure fairness with respect to a protected underrepresented group 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( e.g . gender , race ) . While acknowledging that there is no universal fairness metric [ 27 , 13 ] , several suggestions have been made in the literature including Equal Opportunity favoring same true positive rates across groups [ 15 ] . Methods for imbalanced data are broadly categorized into data- and algorithm- level ones . In the latter category , belong cost-sensitive methods and , specifically , those that modify the training loss to account for varying class/group penalties . Corresponding state-of-the-art ( SOTA ) research is motivated by observations that classical methods , such as weighted cross-entropy ( wCE ) fail when training overparameterized deep nets without regularization and with train-loss minimization continuing well beyond zero train-error , in the so-called terminal phase of training ( TPT ) ( [ 43 ] and references therein ) . Intuitively , failure of wCE when trained in TPT is attributed to the failure to appropriately adjust the relative margins between different classes/groups in a way that favors minorities . To overcome this challenge , recent works have proposed a so-called logit-adjusted ( LA ) loss that modifies the cross-entropy ( CE ) loss by including extra additive hyper-parameters acting on the logits [ 24 , 8 , 32 ] . Even more recently , [ 54 ] suggested yet another modification that introduces multiplicative hyperparameters on the logits leading to a class-dependent temperature ( CDT ) loss . Empirically , both adjustments show performance improvements over wCE . However , it remains unclear : Do both additive and multiplicative hyper-parameters lead to margin-adjustments favoring minority classes ? If so , what are the individual mechanisms that lead to this behavior ? How effective are different adjustments at each stage of training ? This paper answers the above questions . Specifically , we argue that multiplicative hyperparameters are most effective for margin adjustments in TPT , while additive parameters can be useful in the initial phase of training . Importantly , this intuition justifies our algorithmic contribution : we introduce the vector-scaling ( VS ) loss that combines both types of adjustments and attains improved performance on SOTA imbalanced datasets . Finally , using the same set of tools , we extend the VS-loss to instances of group-sensitive classification . We make multiple contributions as summarized below ; see also Figure 1 . ● Explaining the distinct roles of additive/multiplicative adjustments . We show that when optimizing in TPT multiplicative logit adjustments are critical . Specifically , we prove for linear models that multiplicative adjustments find classifiers that are solutions to cost-sensitive support-vector-machines ( CS-SVM ) , which by design create larger margins for minority classes . While effective in TPT , we also find that , at the start of training , the same adjustments can actually harm minorities . Instead , additive adjustments can speed up convergence by countering the initial negative effect of the multiplicative ones . The analytical findings are consistent with our experiments . ● An improved algorithm : VS-loss . Motivated by the unique roles of the two different types of adjustments , we propose the vector-scaling ( VS ) loss that combines the best of both worlds and outperforms existing techniques on benchmark datasets . ● Introducing logit-adjustments for groupimbalanced data.We introduce a version of VS-loss tailored to group-imbalanced datasets , thus treating , for the first time , loss-adjustments for label and group imbalances in a unifying way . For the latter , we propose a new algorithm combining our VS-loss with the previously proposed DRO-method to achieve state-of-the-art performance in terms of both Equal Opportunity and worst-subgroup error . ● Generalization analysis / fairness tradeoffs . We present a sharp generalization analysis of the VS-loss on binary overparameterized Gaussian mixtures . Our formulae are explicit in terms of data geometry , priors , parameterization ratio and hyperparameters ; thus , leading to tradeoffs between standard error and fairness measures . We find that VS-loss can improve both balanced and standard error over CE . Interestingly , the optimal hyperparameters that minimize balanced error also optimize Equal Opportunity . 1.2 Connections to related literature . CE adjustments . The use of wCE for imbalanced data is rather old [ 53 ] , but it becomes ineffective under overparameterization , e.g . [ 6 ] . This deficiency has led to the idea of additive label-based parameters ιy on the logits [ 24 , 8 , 50 , 32 , 52 ] . Specifically , [ 32 ] proved that setting ιy = log ( πy ) ( πy denotes the prior of class y ) leads to a Fisher consistent loss , termed LA-loss , which outperformed other heuristics ( e.g. , focal loss [ 28 ] ) on SOTA datasets . However , Fisher consistency is only relevant in the large sample size limit . Instead , we focus on overparameterized models . In a recent work , [ 54 ] proposed the CDT-loss , which instead uses multiplicative label-based parameters ∆y on the logits . The authors arrive at the CDT-loss as a heuristic means of compensating for the empirically observed phenomenon of that the last-layer minority features deviate between training and test instances [ 25 ] . Instead , we arrive at the CDT-loss via a different viewpoint : we show that the multiplicative weights are necessary to move decision boundaries towards majorities when training overparameterized linear models in TPT . Moreover , we argue that while additive weights are not so effective in the TPT , they can help in the initial phase of training . Our analysis sheds light on the individual roles of the two different modifications proposed in the literature and naturally motivates the VS-loss in ( 2 ) . Compared to the above works we also demonstrate the successful use of VS-loss in group-imbalanced setting and show its competitive performance over alternatives in [ 45 , 18 , 40 ] . Beyond CE adjustments there is active research on alternative methods to improve fairness metrics , e.g . [ 23 , 56 , 29 , 41 ] . These are orthogonal to CE adjustments and can potentially be used in conjunction . Relation to vector-scaling calibration . Our naming of the VS-loss is inspired by the vector scaling ( VS ) calibration [ 14 ] , a post-hoc procedure that modifies the logits v after training via v → ∆ ⊙ v + ι , where ⊙ is the Hadamard product . [ 55 ] shows that VS can improve calibration for imbalanced classes , but , in contrast to VS calibration , the multiplicative/additive scalings in our VS-loss are part of the loss and directly affect training . Blessings/curses of overparameterization . Overparameterization acts as a catalyst for deep neural networks [ 38 ] . In terms of optimization , [ 47 , 42 , 20 , 2 ] show that gradient-based algorithms are implicitly biased towards favorable min-norm solutions . Such solutions , are then analyzed in terms of generalization showing that they can in fact lead to benign overfitting e.g . [ 4 , 16 ] . While implicit bias is key to benign overfitting it may come with certain downsides . As a matter of fact , we show here that certain hyper-parameters ( e.g . additive ones ) can be ineffective in the interpolating regime in promoting fairness . Our argument essentially builds on characterizing the implicit bias of wCE/LA/CDT-losses . Related to this , [ 46 ] demonstrated the ineffectiveness of ωy in learning with groups . 2 Problem setup . Data . Let training set { ( xi , gi , yi ) } ni=1 consisting of n i.i.d . samples from a distribution D over X × G × Y ; X ⊆ Rd is the input space , Y = [ C ] ∶= { 1 , . . . , C } the set of C labels , and , G = [ K ] refers to group membership among K ≥ 1 groups . Group-assignments are known for training data , but unknown at test time . For concreteness , we focus here on the binary setting , i.e . C = 2 and Y = { −1 , +1 } ; we present multiclass extensions in the Experiments and in the Supplementary Material ( SM ) . We assume throughout that y = +1 is minority class . Fairness metrics . Given a training set we learn fw ∶ X ↦ Y parameterized by w ∈ Rp . For instance , linear models take the form fw = ⟨w , h ( x ) ⟩ for some feature representation h ∶ X ↦ Rp . Given a new sample x , we decide class membership ŷ = sign ( fw ( x ) ) . The ( standard ) risk or misclassification error is R ∶= P { ŷ ≠ y } . Let s = ( y , g ) define a subgroup for given values of y and g. We also define the class-conditional risks R± = P { ŷ ≠ y ∣y = ±1 } , and , the sub-group-conditional risks R± , j = P { ŷ ≠ y ∣y = ±1 , g = j } , j ∈ [ K ] . The balanced error averages the conditional risks of the two classes : Rbal ∶= ( R+ +R− ) /2 . Assuming K = 2 groups , Equal Opportunity requires R+,1 =R+,2 [ 15 ] . More generally , we consider the ( signed ) difference of equal opportunity ( DEO ) Rdeo ∶=R+,1 −R+,2 . In our experiments , we also measure the worst-case subgroup error max ( y∈±1 , g∈ [ K ] ) Ry , g . Terminal phase of training ( TPT ) . Motivated by modern training practice , we assume overparameterized fw so that Rtrain = 1n ∑i∈ [ n ] 1 [ sign ( fw ( xi ) ) ≠ yi ] can be driven to zero . Typically , training such large models continues well-beyond zero training error as the training loss is being pushed toward zero . As in [ 43 ] , we call this the terminal phase of training . 2.1 Algorithms . Cross-entropy adjustments.We introduce the vector-scaling ( VS ) loss , which combines both additive and multiplicative logit adjustments , previously suggested in the literature in isolation . The following is the binary VS-loss for labels y ∈ { ±1 } , weight parameters ω± > 0 , additive logit parameters ι± ∈ R , and multiplicative logit parameters ∆± > 0 : ` VS ( y , fw ( x ) ) = ωy ⋅ log ( 1 + eιy ⋅ e−∆yyfw ( x ) ) . ( 1 ) For imbalanced datasets with C > 2 classes , the VS-loss takes the following form : ` VS ( y , fw ( x ) ) = −ωy log ( e∆yfy ( x ) +ιy/ ∑ c∈ [ C ] e∆cfc ( x ) +ιc ) . ( 2 ) Here fw ∶ Rd → RC and fw ( x ) = [ f1 ( x ) , . . . , fC ( x ) ] is the vector of logits . The VS-loss ( Eqns . ( 1 ) , ( 2 ) ) captures existing techniques as special cases by tuning accordingly the additive/multiplicative hyperparameters . Specifically , we recover : ( i ) weighted CE ( wCE ) loss by ∆y = 1 , ιy = 0 , ωy = π−1y ; ( ii ) LA-loss by ∆y = 1 ; ( iii ) CDT-loss by ιy = 0 . With the goal of ( additionally ) ensuring fairness with respect to sensitive groups , we extend the VS-loss by introducing parameters ( ∆y , g , ιy , g , ωy , g ) that depend both on class and group membership ( specified by y and g , respectively ) . Our proposed group-sensitive VS-loss is as follows ( multiclass version can be defined accordingly ) : ` Group−VS ( y , g , fw ( x ) ) = ωy , g ⋅ log ( 1 + eιy , g ⋅ e−∆y , gyfw ( x ) ) . ( 3 ) CS-SVM . For linear classifiers fw ( x ) = ⟨w , h ( x ) ⟩ with h ∶ X → Rp , CS-SVM [ 31 ] solves min w ∥w∥2 sub . to { ⟨w , h ( xi ) ⟩ ≥ δ , yi = +1 ⟨w , h ( xi ) ⟩ ≤ −1 , yi = −1 , i ∈ [ n ] , ( 4 ) for hyper-parameter δ ∈ R+ representing the ratio of margins between classes . δ = 1 corresponds to ( standard ) SVM , while tuning δ > 1 ( resp . δ < 1 ) favors a larger margin δ/∥ŵδ∥2 for the minority vs 1/∥ŵδ∥2 for the majority classes . Thus , δ → +∞ ( resp . δ → 0 ) corresponds to the decision boundary starting right at the boundary of class y = −1 ( resp . y = +1 ) . Group-sensitive SVM . The group-sensitive version of CS-SVM ( GS-SVM ) , for K = 2 protected groups adjusts the constraints in ( 4 ) so that yi⟨w , h ( xi ) ⟩ ≥ δ ( or ≥ 1 ) , if gi = 1 ( or gi = 2 . ) δ > 1 , GS-SVM favors larger margin for the sensitive group g = 1 . Refined versions when classes are also imbalanced modify the constraints to yi⟨w , h ( xi ) ⟩ ≥ δyi , gi . Both CSSVM and GS-SVM are feasible iff data are linearly separable ( see SM ) . However , we caution that the GS-SVM hyper-parameters are in general harder to interpret as “ margin-ratios '' .

Summary: This paper studies the training loss used by neural networks for class-imbalanced learning/group-sensitive. It wants to investigate among two terms, multiplicative and additive, added to the weighted cross-entropy loss, how do they contribute to the class-imbalanced/group-sensitive learning and in what phase. By studying this problem, the paper did some experiments to get some insights and proved the equivalence of the solution achieved by the weighted cross-entropy loss with the additional terms, and the solution achieved by the cost-sensitive SVM on a linear model. The conclusion is drawn that the multiplicative term is more effective than the other one. On the other hand, it discusses that at the initial phase, the additive term may take effect. By additional distribution assumption of GMM model, the paper also derives generalization analysis and the trade-off between balanced error and imbalanced error. Finally, it shows some experimental results.  ============ Thank you for the rebuttal. I think the contribution of the paper is that it studies from the theoretical aspect a commonly met problem, and the insights drawn from theoretical studies can be used to design better loss functions.  In this way, I would keep my positive rate for this paper.