Long-tail learning via logit adjustment

1 INTRODUCTION . Real-world classification problems typically exhibit a long-tailed label distribution , wherein most labels are associated with only a few samples ( Van Horn & Perona , 2017 ; Buda et al. , 2017 ; Liu et al. , 2019 ) . Owing to this paucity of samples , generalisation on such labels is challenging ; moreover , naïve learning on such data is susceptible to an undesirable bias towards dominant labels . This problem has been widely studied in the literature on learning under class imbalance ( Kubat et al. , 1997 ; Chawla et al. , 2002 ; He & Garcia , 2009 ) , and the related problem of cost-sensitive learning ( Elkan , 2001 ) . Recently , long-tail learning has received renewed interest in the context of neural networks . Two active strands of work involve post-hoc normalisation of the classification weights ( Zhang et al. , 2019 ; Kim & Kim , 2019 ; Kang et al. , 2020 ; Ye et al. , 2020 ) , and modification of the underlying loss to account for varying class penalties ( Zhang et al. , 2017 ; Cui et al. , 2019 ; Cao et al. , 2019 ; Tan et al. , 2020 ) . Each of these strands are intuitive , and have proven empirically successful . However , they are not without limitation : e.g. , weight normalisation crucially relies on the weight norms being smaller for rare classes ; however , this assumption is sensitive to the choice of optimiser ( see §2.1 ) . On the other hand , loss modification sacrifices the consistency that underpins the canonical softmax cross-entropy ( see §5.1 ) . Consequently , such techniques may prove suboptimal even in simple settings ( see §6.1 ) . In this paper , we establish a statistical framework for long-tail learning that offers a unified view of post-hoc normalisation and loss modification techniques , while overcoming their limitations . Our framework revisits the classic idea of logit adjustment based on label frequencies ( Provost , 2000 ; Zhou & Liu , 2006 ; Collell et al. , 2016 ) , which encourages a large relative margin between a pair of rare positive and dominant negative labels . Such adjustment can be achieved by shifting the learned logits post-hoc , or augmenting the softmax cross-entropy with a pairwise label margin ( cf . ( 11 ) ) . While similar in nature to recent techniques , our logit adjustment approaches additionally have a firm statistical grounding : they are Fisher consistent for minimising the balanced error ( cf . ( 2 ) ) , a common metric in long-tail settings which averages the per-class errors . This statistical grounding translates into strong empirical performance on four real-world datasets with long-tailed label distributions . In summary , our contributions are : ( i ) we establish a statistical framework for long-tail learning ( §3 ) based on logit adjustment that provides a unified view of post-hoc correction and loss modification 2 PROBLEM SETUP AND RELATED WORK . Consider a multiclass classification problem with instances X and labels Y = [ L ] .= { 1 , 2 , . . . , L } . Given a sample S = { ( xn , yn ) } Nn=1 ∼ PN for unknown distribution P over X × Y , our goal is to learn a scorer f : X→ RL that minimises the misclassification error Px , y ( y /∈ argmaxy′∈Y fy′ ( x ) ) . Typically , one minimises a surrogate loss ` : Y× RL → R such as the softmax cross-entropy , ` ( y , f ( x ) ) = log [ ∑ y′∈ [ L ] efy′ ( x ) ] − fy ( x ) = log [ 1 + ∑ y′ 6=y efy′ ( x ) −fy ( x ) ] . ( 1 ) We may view the resulting softmax probabilities py ( x ) ∝ efy ( x ) as estimates of P ( y | x ) . The setting of learning under class imbalance or long-tail learning is where the distribution P ( y ) is highly skewed , so that many rare ( or “ tail ” ) labels have a low probability of occurrence . Here , the misclassification error is not a suitable measure of performance : a trivial predictor which classifies every instance to the majority label will attain a low misclassification error . To cope with this , a natural alternative is the balanced error ( Chan & Stolfo , 1998 ; Brodersen et al. , 2010 ; Menon et al. , 2013 ) , which averages each of the per-class error rates : under a uniform label distribution BER ( f ) . = 1 L ∑ y∈ [ L ] Px|y ( y /∈ argmaxy′∈Y fy′ ( x ) ) . ( 2 ) This can be seen as implicitly using a balanced class-probability function Pbal ( y | x ) ∝ 1L ·P ( x | y ) , as opposed to the native P ( y | x ) ∝ P ( y ) · P ( x | y ) that is employed in the misclassification error.1 Broadly , extant approaches to coping with class imbalance modify : ( i ) the inputs to a model , for example by over- or under-sampling ( Kubat & Matwin , 1997 ; Chawla et al. , 2002 ; Wallace et al. , 2011 ; Mikolov et al. , 2013 ; Mahajan et al. , 2018 ; Yin et al. , 2018 ) ; ( ii ) the outputs of a model , for example by post-hoc correction of the decision threshold ( Fawcett & Provost , 1996 ; Collell et al. , 2016 ) or weights ( Kim & Kim , 2019 ; Kang et al. , 2020 ) ; or ( iii ) the training procedure of a model , for example by modifying the loss function ( Zhang et al. , 2017 ; Cui et al. , 2019 ; Cao et al. , 2019 ; Tan et al. , 2020 ; Jamal et al. , 2020 ) . One may easily combine approaches from the first stream with those from the latter two . Consequently , we focus on the latter two in this work , and describe some representative recent examples from each . Post-hoc weight normalisation . Suppose fy ( x ) = w > y Φ ( x ) for classification weights wy ∈ RD and representations Φ : X→ RD , as learned by a neural network . ( We may add per-label bias terms 1Both the misclassification and balanced error compare the top-1 predicted versus true label . One may analogously define a balanced top-k error ( Lapin et al. , 2018 ) , which may be useful in retrieval settings . to fy by adding a constant feature to Φ . ) A fruitful avenue of exploration involves decoupling of representation and classifier learning ( Zhang et al. , 2019 ) . Concretely , we first learn { wy , Φ } via standard training on the long-tailed training sample S , and then for x ∈ X predict the label argmaxy∈ [ L ] w > y Φ ( x ) /ν τ y = argmaxy∈ [ L ] fy ( x ) /ν τ y , ( 3 ) for τ > 0 , where νy = P ( y ) in Kim & Kim ( 2019 ) ; Ye et al . ( 2020 ) and νy = ‖wy‖2 in Kang et al . ( 2020 ) . Intuitively , either choice of νy upweights the contribution of rare labels through weight normalisation . The choice νy = ‖wy‖2 is motivated by the observations that ‖wy‖2 tends to correlate with P ( y ) . Further to the above , one may enforce ‖wy‖2 = 1 during training ( Kim & Kim , 2019 ) . Loss modification . A classic means of coping with class imbalance is to balance the loss , wherein ` ( y , f ( x ) ) is weighted by P ( y ) −1 ( Xie & Manski , 1989 ; Morik et al. , 1999 ) : e.g. , applied to ( 1 ) , ` ( y , f ( x ) ) = 1 P ( y ) · log [ 1 + ∑ y′ 6=y efy′ ( x ) −fy ( x ) ] . ( 4 ) While intuitive , balancing has minimal effect in separable settings : solutions that achieve zero training loss will necessarily remain optimal even under weighting ( Byrd & Lipton , 2019 ) . Intuitively , one would instead like to shift the separator closer to a dominant class . Li et al . ( 2002 ) ; Wu et al . ( 2008 ) ; Masnadi-Shirazi & Vasconcelos ( 2010 ) thus proposed to add per-class margins into the hinge loss . Cao et al . ( 2019 ) similarly proposed to add a per-class margin into the softmax cross-entropy : ` ( y , f ( x ) ) = log [ 1 + ∑ y′ 6=y eδy · efy′ ( x ) −fy ( x ) ] , ( 5 ) where δy ∝ P ( y ) −1/4 . This upweights rare “ positive ” y to encourage a larger gap fy ( x ) − fy′ ( x ) , i.e. , the margin between y and any “ negative ” y′ 6= y . Separately , Tan et al . ( 2020 ) proposed ` ( y , f ( x ) ) = log [ 1 + ∑ y′ 6=y eδy′ · efy′ ( x ) −fy ( x ) ] , ( 6 ) where δy′ ≤ 0 is a non-decreasing transform of P ( y′ ) . Note that in the original softmax cross-entropy with δy′ = 0 , a rare label often receives a strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels . This can be modulated by letting δy′ 0 . 2.1 LIMITATIONS OF EXISTING APPROACHES . Each of the above methods are intuitive , and have shown strong empirical performance . However , a closer analysis identifies some subtle limitations . Limitations of weight normalisation . Post-hoc weight normalisation with νy = ‖wy‖2 per Kang et al . ( 2020 ) is motivated by the observation that the weight norm ‖wy‖2 tends to correlate with P ( y ) . However , this assumption is highly dependent on the choice of optimiser , as Figure 1 illustrates : for ResNet-32 models trained on long-tailed versions of CIFAR-10 and CIFAR-100 , when using the Adam optimiser , the norms are either anti-correlated or independent of P ( y ) . Weight normalisation thus can not achieve the desired effect of boosting rare labels ’ scores . One may hope to side-step this by simply using νy = P ( y ) ; unfortunately , this choice has more subtle limitations ( see §4.2 ) . Limitations of loss modification . Enforcing a per-label margin per ( 5 ) and ( 6 ) is intuitive , as it allows for shifting the decision boundary away from rare classes . However , when doing so , it is important to ensure Fisher consistency ( Lin , 2004 ) ( or classification calibration ( Bartlett et al. , 2006 ) ) of the resulting loss for the balanced error . That is , the minimiser of the expected loss ( equally , the empirical risk in the infinite sample limit ) should result in a minimal balanced error . Unfortunately , both ( 5 ) and ( 6 ) are not consistent in this sense , even for binary problems ; see §5.1 , §6.1 for details .

Summary: This paper provides a statistical framework for long-tail learning by revisiting the idea of logit adjustment based on the label frequencies. The proposed framework then yields two variant techniques that follow the paradigm of weight normalization or loss modification. Compared with the existing methods, the proposed methods are generalized and Fisher consistent for minimizing the balanced error. Finally, empirical results on four real-world datasets validate the effectiveness and statistical grounding of the proposed methods.