Label-Imbalanced and Group-Sensitive Classification under Overparameterization

1 Introduction . 1.1 Motivation and contributions . Equitable learning in the presence of data imbalances is a classical machine learning ( ML ) problem , but one with increasing importance as ML decisions are adapted in increasingly more complex applications directly involving people [ 3 ] . Two common types of imbalances are those appearing in label-imbalanced and group-sensitive classification . In the first type , examples from a target class are heavily outnumbered by examples from the rest of the classes . The standard metric of average misclassification error is insensitive to such imbalances and among several classical alternatives the balanced error is a widely used metric . In the second type , the broad goal is to ensure fairness with respect to a protected underrepresented group 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( e.g . gender , race ) . While acknowledging that there is no universal fairness metric [ 27 , 13 ] , several suggestions have been made in the literature including Equal Opportunity favoring same true positive rates across groups [ 15 ] . Methods for imbalanced data are broadly categorized into data- and algorithm- level ones . In the latter category , belong cost-sensitive methods and , specifically , those that modify the training loss to account for varying class/group penalties . Corresponding state-of-the-art ( SOTA ) research is motivated by observations that classical methods , such as weighted cross-entropy ( wCE ) fail when training overparameterized deep nets without regularization and with train-loss minimization continuing well beyond zero train-error , in the so-called terminal phase of training ( TPT ) ( [ 43 ] and references therein ) . Intuitively , failure of wCE when trained in TPT is attributed to the failure to appropriately adjust the relative margins between different classes/groups in a way that favors minorities . To overcome this challenge , recent works have proposed a so-called logit-adjusted ( LA ) loss that modifies the cross-entropy ( CE ) loss by including extra additive hyper-parameters acting on the logits [ 24 , 8 , 32 ] . Even more recently , [ 54 ] suggested yet another modification that introduces multiplicative hyperparameters on the logits leading to a class-dependent temperature ( CDT ) loss . Empirically , both adjustments show performance improvements over wCE . However , it remains unclear : Do both additive and multiplicative hyper-parameters lead to margin-adjustments favoring minority classes ? If so , what are the individual mechanisms that lead to this behavior ? How effective are different adjustments at each stage of training ? This paper answers the above questions . Specifically , we argue that multiplicative hyperparameters are most effective for margin adjustments in TPT , while additive parameters can be useful in the initial phase of training . Importantly , this intuition justifies our algorithmic contribution : we introduce the vector-scaling ( VS ) loss that combines both types of adjustments and attains improved performance on SOTA imbalanced datasets . Finally , using the same set of tools , we extend the VS-loss to instances of group-sensitive classification . We make multiple contributions as summarized below ; see also Figure 1 . ● Explaining the distinct roles of additive/multiplicative adjustments . We show that when optimizing in TPT multiplicative logit adjustments are critical . Specifically , we prove for linear models that multiplicative adjustments find classifiers that are solutions to cost-sensitive support-vector-machines ( CS-SVM ) , which by design create larger margins for minority classes . While effective in TPT , we also find that , at the start of training , the same adjustments can actually harm minorities . Instead , additive adjustments can speed up convergence by countering the initial negative effect of the multiplicative ones . The analytical findings are consistent with our experiments . ● An improved algorithm : VS-loss . Motivated by the unique roles of the two different types of adjustments , we propose the vector-scaling ( VS ) loss that combines the best of both worlds and outperforms existing techniques on benchmark datasets . ● Introducing logit-adjustments for groupimbalanced data.We introduce a version of VS-loss tailored to group-imbalanced datasets , thus treating , for the first time , loss-adjustments for label and group imbalances in a unifying way . For the latter , we propose a new algorithm combining our VS-loss with the previously proposed DRO-method to achieve state-of-the-art performance in terms of both Equal Opportunity and worst-subgroup error . ● Generalization analysis / fairness tradeoffs . We present a sharp generalization analysis of the VS-loss on binary overparameterized Gaussian mixtures . Our formulae are explicit in terms of data geometry , priors , parameterization ratio and hyperparameters ; thus , leading to tradeoffs between standard error and fairness measures . We find that VS-loss can improve both balanced and standard error over CE . Interestingly , the optimal hyperparameters that minimize balanced error also optimize Equal Opportunity . 1.2 Connections to related literature . CE adjustments . The use of wCE for imbalanced data is rather old [ 53 ] , but it becomes ineffective under overparameterization , e.g . [ 6 ] . This deficiency has led to the idea of additive label-based parameters ιy on the logits [ 24 , 8 , 50 , 32 , 52 ] . Specifically , [ 32 ] proved that setting ιy = log ( πy ) ( πy denotes the prior of class y ) leads to a Fisher consistent loss , termed LA-loss , which outperformed other heuristics ( e.g. , focal loss [ 28 ] ) on SOTA datasets . However , Fisher consistency is only relevant in the large sample size limit . Instead , we focus on overparameterized models . In a recent work , [ 54 ] proposed the CDT-loss , which instead uses multiplicative label-based parameters ∆y on the logits . The authors arrive at the CDT-loss as a heuristic means of compensating for the empirically observed phenomenon of that the last-layer minority features deviate between training and test instances [ 25 ] . Instead , we arrive at the CDT-loss via a different viewpoint : we show that the multiplicative weights are necessary to move decision boundaries towards majorities when training overparameterized linear models in TPT . Moreover , we argue that while additive weights are not so effective in the TPT , they can help in the initial phase of training . Our analysis sheds light on the individual roles of the two different modifications proposed in the literature and naturally motivates the VS-loss in ( 2 ) . Compared to the above works we also demonstrate the successful use of VS-loss in group-imbalanced setting and show its competitive performance over alternatives in [ 45 , 18 , 40 ] . Beyond CE adjustments there is active research on alternative methods to improve fairness metrics , e.g . [ 23 , 56 , 29 , 41 ] . These are orthogonal to CE adjustments and can potentially be used in conjunction . Relation to vector-scaling calibration . Our naming of the VS-loss is inspired by the vector scaling ( VS ) calibration [ 14 ] , a post-hoc procedure that modifies the logits v after training via v → ∆ ⊙ v + ι , where ⊙ is the Hadamard product . [ 55 ] shows that VS can improve calibration for imbalanced classes , but , in contrast to VS calibration , the multiplicative/additive scalings in our VS-loss are part of the loss and directly affect training . Blessings/curses of overparameterization . Overparameterization acts as a catalyst for deep neural networks [ 38 ] . In terms of optimization , [ 47 , 42 , 20 , 2 ] show that gradient-based algorithms are implicitly biased towards favorable min-norm solutions . Such solutions , are then analyzed in terms of generalization showing that they can in fact lead to benign overfitting e.g . [ 4 , 16 ] . While implicit bias is key to benign overfitting it may come with certain downsides . As a matter of fact , we show here that certain hyper-parameters ( e.g . additive ones ) can be ineffective in the interpolating regime in promoting fairness . Our argument essentially builds on characterizing the implicit bias of wCE/LA/CDT-losses . Related to this , [ 46 ] demonstrated the ineffectiveness of ωy in learning with groups . 2 Problem setup . Data . Let training set { ( xi , gi , yi ) } ni=1 consisting of n i.i.d . samples from a distribution D over X × G × Y ; X ⊆ Rd is the input space , Y = [ C ] ∶= { 1 , . . . , C } the set of C labels , and , G = [ K ] refers to group membership among K ≥ 1 groups . Group-assignments are known for training data , but unknown at test time . For concreteness , we focus here on the binary setting , i.e . C = 2 and Y = { −1 , +1 } ; we present multiclass extensions in the Experiments and in the Supplementary Material ( SM ) . We assume throughout that y = +1 is minority class . Fairness metrics . Given a training set we learn fw ∶ X ↦ Y parameterized by w ∈ Rp . For instance , linear models take the form fw = ⟨w , h ( x ) ⟩ for some feature representation h ∶ X ↦ Rp . Given a new sample x , we decide class membership ŷ = sign ( fw ( x ) ) . The ( standard ) risk or misclassification error is R ∶= P { ŷ ≠ y } . Let s = ( y , g ) define a subgroup for given values of y and g. We also define the class-conditional risks R± = P { ŷ ≠ y ∣y = ±1 } , and , the sub-group-conditional risks R± , j = P { ŷ ≠ y ∣y = ±1 , g = j } , j ∈ [ K ] . The balanced error averages the conditional risks of the two classes : Rbal ∶= ( R+ +R− ) /2 . Assuming K = 2 groups , Equal Opportunity requires R+,1 =R+,2 [ 15 ] . More generally , we consider the ( signed ) difference of equal opportunity ( DEO ) Rdeo ∶=R+,1 −R+,2 . In our experiments , we also measure the worst-case subgroup error max ( y∈±1 , g∈ [ K ] ) Ry , g . Terminal phase of training ( TPT ) . Motivated by modern training practice , we assume overparameterized fw so that Rtrain = 1n ∑i∈ [ n ] 1 [ sign ( fw ( xi ) ) ≠ yi ] can be driven to zero . Typically , training such large models continues well-beyond zero training error as the training loss is being pushed toward zero . As in [ 43 ] , we call this the terminal phase of training . 2.1 Algorithms . Cross-entropy adjustments.We introduce the vector-scaling ( VS ) loss , which combines both additive and multiplicative logit adjustments , previously suggested in the literature in isolation . The following is the binary VS-loss for labels y ∈ { ±1 } , weight parameters ω± > 0 , additive logit parameters ι± ∈ R , and multiplicative logit parameters ∆± > 0 : ` VS ( y , fw ( x ) ) = ωy ⋅ log ( 1 + eιy ⋅ e−∆yyfw ( x ) ) . ( 1 ) For imbalanced datasets with C > 2 classes , the VS-loss takes the following form : ` VS ( y , fw ( x ) ) = −ωy log ( e∆yfy ( x ) +ιy/ ∑ c∈ [ C ] e∆cfc ( x ) +ιc ) . ( 2 ) Here fw ∶ Rd → RC and fw ( x ) = [ f1 ( x ) , . . . , fC ( x ) ] is the vector of logits . The VS-loss ( Eqns . ( 1 ) , ( 2 ) ) captures existing techniques as special cases by tuning accordingly the additive/multiplicative hyperparameters . Specifically , we recover : ( i ) weighted CE ( wCE ) loss by ∆y = 1 , ιy = 0 , ωy = π−1y ; ( ii ) LA-loss by ∆y = 1 ; ( iii ) CDT-loss by ιy = 0 . With the goal of ( additionally ) ensuring fairness with respect to sensitive groups , we extend the VS-loss by introducing parameters ( ∆y , g , ιy , g , ωy , g ) that depend both on class and group membership ( specified by y and g , respectively ) . Our proposed group-sensitive VS-loss is as follows ( multiclass version can be defined accordingly ) : ` Group−VS ( y , g , fw ( x ) ) = ωy , g ⋅ log ( 1 + eιy , g ⋅ e−∆y , gyfw ( x ) ) . ( 3 ) CS-SVM . For linear classifiers fw ( x ) = ⟨w , h ( x ) ⟩ with h ∶ X → Rp , CS-SVM [ 31 ] solves min w ∥w∥2 sub . to { ⟨w , h ( xi ) ⟩ ≥ δ , yi = +1 ⟨w , h ( xi ) ⟩ ≤ −1 , yi = −1 , i ∈ [ n ] , ( 4 ) for hyper-parameter δ ∈ R+ representing the ratio of margins between classes . δ = 1 corresponds to ( standard ) SVM , while tuning δ > 1 ( resp . δ < 1 ) favors a larger margin δ/∥ŵδ∥2 for the minority vs 1/∥ŵδ∥2 for the majority classes . Thus , δ → +∞ ( resp . δ → 0 ) corresponds to the decision boundary starting right at the boundary of class y = −1 ( resp . y = +1 ) . Group-sensitive SVM . The group-sensitive version of CS-SVM ( GS-SVM ) , for K = 2 protected groups adjusts the constraints in ( 4 ) so that yi⟨w , h ( xi ) ⟩ ≥ δ ( or ≥ 1 ) , if gi = 1 ( or gi = 2 . ) δ > 1 , GS-SVM favors larger margin for the sensitive group g = 1 . Refined versions when classes are also imbalanced modify the constraints to yi⟨w , h ( xi ) ⟩ ≥ δyi , gi . Both CSSVM and GS-SVM are feasible iff data are linearly separable ( see SM ) . However , we caution that the GS-SVM hyper-parameters are in general harder to interpret as “ margin-ratios '' .

Summary: This paper investigate the prediction problem under the settings of label-imbalance and group-sensitive. Previous literature propose several methods to solve it, such as weighted Cross-Entropy, additive/multiplicative adjustments on logits. This paper explore the mechanism and effect of these methods, and analyze the relationship to SVM and CS-SVM. It propose to combine additive/multiplicative adjustments to enjoy the benefit of both methods. The experimental results demonstrate the effectiveness of VS-loss which combine additive/multiplicative adjustments.