Long-tail learning via logit adjustment
Authors: Aditya Krishna Menon
Sadeep Jayasumana
Ankit Singh Rawat
Himanshu Jain
Andreas Veit
Sanjiv Kumar
Venue: ICLR 2021 Spotlight (2021)
Keywords: long-tail learning
class imbalance

Abstract: Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels have only a few associated samples. This poses a challenge for generalisation on such labels, and also  makes naive learning biased towards dominant labels. In this paper,  we present a statistical framework that unifies and generalises several recent proposals to cope with these challenges. Our framework revisits the classic idea of logit adjustment based on the label frequencies, which encourages a large relative margin between logits of rare positive versus dominant negative labels. This yields two techniques  for long-tail learning, where such adjustment is either applied post-hoc to a trained model, or enforced in the loss during training. These techniques are statistically grounded, and practically effective on four real-world datasets with long-tailed label distributions. 

Published as a conference paper at ICLR 2021
LONG-TAIL LEARNING VIA LOGIT ADJUSTMENT
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain Andreas Veit & Sanjiy Kumar
Google Research
New York, NY
(adi tyakmenon, saniee,, aadeep,,nkitssrawee himj awwiit,snjivkj@goggee..om
ABSTRACT
Real-world classification problems typically exhibit an imbalanced or long-tailed lahel distribution, wherein many labels have only few associated samples. This poses a challenge for generalisation on such labels, and also makes naive learning biased towards dominant labels. In this paper, we present a statistical framework that unifies and generalises several recent proposals to cope with these challenges. Our framework revisits the classic idea of logit adjustment based on the label frequencies, which encourages large relative margin between logits of rare positive versus dominant negative labels. This yields two techniques for long-tail learning, where such adjustment is either applied post-hoc to a trained model, or enforced in the loss during training. These techniques are statistically grounded, and practically effective on four real-world datasets with long-tailed label distributions.
INTRODUCTION
Real-world classification problems typically exhibit a long-tailed label distribution, wherein most labels are associated with only few samples (Van Horn & Perona, 2017; Buda et al., 2017; Liu et al., 2019). Owing to this paucity of samples, generalisation on such labels is challenging; moreover, naive learning on such data is susceptible to undesirable bias towards dominant labels. This problem has been widely studied in the literature on learning under class imbalance (Kubat et al., 1997: Chawla et al., 2002; He & Garcia, 2009), and the related problem of cost-sensitive learning (Elkan, 2001),
Recently, long-tail learning has received renewed interest in the context of neural networks. Two active strands of work involve post-hoc normalisation of the classification weights (Zhang et al., 2019; Kim & Kim, 2019; Kang et al., 2020; Ye et al., 2020), and modification of the underlying loss account for varying class penalties (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020). Each these strands are intuitive, and have proven empirically successful. However, they are not without limitation: e.g., weight normalisation crucially relies on the weight norms being smaller for rare classes; however, this assumption is sensitive to the choice of optimiser (see $2.1). On the other hand, loss modification sacrifices the consistency that underpins the canonical softmax cross-entropy (see $5.1). Consequently, such techniques may prove suboptimal even in simple settings (see $6.1).
In this paper, we establish statistical framework for long-tail learning that offers unified view of post-hoc normalisation and loss modification techniques, while overcoming their limitations. Our framework revisits the classic idea of logit adjustment based on label frequencies (Provost, 2000; Zhou & Liu, 2006; Collell et al., 2016), which encourages a large relative margin between a pair of rare positive and dominant negative labels. Such adjustment can be achieved by shifting the learned logits post-hoc, or augmenting the softmax cross-entropy with a pairwise label margin (cf. (11)). While similar in nature recent techniques, our logit adjustment approaches additionally have firm statistical grounding: they are Fisher consistent for minimising the balanced error (cf. (2)), a common metric in long-tail settings which averages the per-class errors. This statistical grounding translates into strong empirical performance on four real-world datasets with long-tailed label distributions.
In summary, contributions
we establish statistical framework for long-tail tail learning ($3) based on logit adjustment that provides unified view of post-hoe correction and loss modification

---

Published as a conference paper at ICLR 2021
Table Comparison of approaches to long-tail learning. Weight normalisation re-scales the classifi- cation weights; by contrast, we add per-label offsets to the logits. Margin approaches uniformly increase the margin between a rare positive and all negatives (Cao al., 2019), or decrease the margin between all positives and rare negative (Tan et al., 2020) to prevent rare labels' gradient suppression. By contrast, we increase the margin between rare positive and a dominant negative.
Method
Procedure
Consistent?
Reference
Weight normalisation
Post-hoc weight scaling
(Kang et al., 2020)
Adaptive margin
Softmax with rare +ve upweighting
(Cao al., 2019)
Equalised margin
Softmax with rare -ve downweighting
(Tan et al., 2020)
Logit-adjusted threshold
Post-hoc logit translation
This paper (cf. (9))
Logit- logit-adjusted loss
Softmax with logit translation
This paper (cf. (10))
(ii) we present two realisations of logit adjustment, applied either post-hoc ($4.1) or during training ($5.1), I); unlike recent proposals (Table 1), these are consistent for minimising the balanced error (iii) we confirm the efficacy of the proposed logit adjustment techniques compared to several baselines on four real-world datasets with long-tailed label distributions ($6).
PROBLEM SETUP AND RELATED WORK
Consider a multiclass classification problem with instances X and labels y [L] = 21. L). Given a sample $ = {(In, } ~ Aa for unknown distribution over X x y, our goal is to learn a scorer f: X + R² that minimises the misclassification error (x,v(y (y & argmax, fo (x)).
Typically, one minimises a surrogate loss l: y x R + R such as the softmax cross-entropy,
E(y, f())) log
Eyret e - fx(x) -log log 1+ [1
(1)
y'ElL
We may view the resulting softmax probabilities (x(x) x (fa( as estimates of P(y z))
The setting of learning under class imbalance or long-tail learning is where the distribution (y() is highly skewed, so that many rare (or "tail") labels have a low probability of occurrence. Here, the misclassification error is not suitable measure of performance: trivial predictor which classifies every instance the majority label will attain low misclassification error. To cope with this, natural alternative is the balanced error (Chan & Stolfo, 1998; Brodersen et al., 2010; Menon et al., 2013), which averages each of the per-class error rates: under a uniform label distribution
BER(F) : Σ DEIL]
P Poty argmax, (x)).
This can be seen as implicitly using a balanced class-probabillty function phal (y/ [x) x 1 IP(x y), as opposed to the native Ppy| æ) x PPy) Prr y) that is employed in the misclassification error.'
Broadly, extant approaches coping with class imbalance modify:
ii) the inputs to a model, for example by over- or under-sampling (Kubat & Matwin, 1997; Chawla et al., 2002; Wallace et al., 2011; Mikolov et al., 2013; Mahajan et al., 2018; Yin et al., 2018);
(ii) the outputs of model, for example by post-hoc correction of the decision threshold (Fawcett & Provost, 1996; Collell et al., 2016) or weights (Kim & Kim, 2019; Kang et al., 2020); or (iii) the training procedure of model, for example by modifying the loss function (Zhang et al., 2017; Cui al., 2019; Cao al., 2019; Tan et al., 2020; Jamal et al., 2020).
One may easily combine approaches from the first stream with those from the latter two. Consequently, we focus on the latter two this work, and describe some representative recent examples from each.
Post-hoc weight normalisation. Suppose fy(2) (z) = I (xx) for classification weights wy E R and representations +; X D, learned by neural network. (We may add per-label bias terms
Both the misclassification and halanced error compare the top-1 predicted versus true label. One may analogously define balanced top-k error (Lapin al., 2018), which may be useful in retrieval settings.
2

---

Published as a conference paper at ICLR 2021
13
Homantum
Homontum
12
10
100
Class y
Class
(a) CIFAR-IO-LT.
(b) 100AR-I00-LT.
Figure 1: Mean and standard deviation of per-class weight norms || over runs for ResNet-32 under momentum and Adam optimisers. We use long-tailed ("L.I") versions of CIFAR-10 and CIFAR-100, and sort classes in descending order of frequency; the first class is 100x more likely to appear than the last class (see $6.2). Both optimisers yield comparable balanced error. However, the weight norms have incompatible trends: under momentum, the norms are strongly correlated with class frequency, while with Adam, the norms are anti cotticorrellted or independent of the class frequency. Consequently, weight normalisation under Adam ineffective for combatting class imbalance.
to Jy by adding a constant feature to Φ.) A fruitful avenue of exploration involves decoupling of representation and classifier learning (Zhang al., 2019). Concretely, we first learn (wy, b} via standard training on the long-tailed training sample S, and then for x E X predict the label
argmax wy T + (x) = argmax (2)///)
(3)
for > O, where V, -P(y) in Kim & Kim (2019); Ye et al. (2020) and My w ll in Kang et al. (2020). Intuitively, either choice of vy upweights the contribution of rare labels through weight normalisation. The choice V ||w 2 is motivated by the observations that || ll2 tends to correlate with Ply). Further to the above, one may enforce ||wy llz = during training (Kim & Kim, 2019).
Loss modification. A classic means of coping with class imbalance is to balance the loss, wherein f(y, f(x)) is weighted by IP(y) (Xie & Manski, 1989; Morik et al., 1999): e.g., applied to (1),
1 Ely, f()) Ply)
log 1+ Σ e (0(()))))))))))
While intuitive, balancing has minimal effect in separable settings: solutions that achieve zero training loss will necessarily remain optimal even under weighting (Byrd & Lipton, 2019). Intuitively, one would instead like to shift the separator closer to a dominant class. Li et al. (2002); Wu et al. (2008); Hasnadi-Shirazz & Vasconcelos (2010) thus proposed to add per-class margins into the hinge loss. Cao et al. (2019) similarly proposed to add a per-class margin into the softmax cross-entropy:
Ely, fx)) = log I Σ
Φdv .l..elllee (x) (
(5)
where 5y (P(v)) This upweights rare "positive" encourage larger gap Sy (x) ie., the margin between y and any "negative" v' + y. Separately, Tan et al. (2020) proposed
Sy (x),
fly, f()) log Ev evv (efef(t)))-()()))
where b. < is a non-decreaiing transform of IP(y'). Note that in the original softmax cross-entropy with by = 0, rare label often receives strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels. This can be modulated by letting by < 0.
2.1 LIMITATIONS OF EXISTING APPROACHES
Each of the above methods are intuitive, and have shown strong empirical performance. However, closer analysis identifies some subtle limitations.
Limitations of weight normalisation. Post-hoc weight normalisation with by lwyll2 per Kang et al. (2020) is motivated by the observation that the weight norm |u,]l2 tends to correlate with P(y). However, this assumption highly dependent the choice of optimiser, Figure illustrates: for
3

---

Published as a conference paper at ICLR 2021
ResNet-32 models trained long-tailed versions of CIFAR- 10 and 1IFAR-100, when using the Adam optimiser, the norms are either anti-correlated or independent of P(y). Weight normalisation thus cannot achieve the desired effect of boosting rare labels' scores. One may hope to side-step this by simply using Vy = P(y); unfortunately, this choice has more subtle limitations (see $4.2).
Limitations of loss modification. Enforcing per-label margin per (5) and (6) is intuitive, for shifting the decision boundary away from rare classes. However, when doing so, it is important ensure Fisher consistency (Lin, 2004) (or classification calibration (Bartlett al., 2006)) of the resulting loss for the balanced error. That is, the minimiser of the expected loss (equally, the empirical risk in the infinite sample limit) should result in minimal balanced error. Unfortunately, both (5) and (6) are not consistent in this sense, even for binary problems; see $5.1, $6.1 for details.
allows
3 LOGIT ADJUSTMENT FOR LONG-TAIL LEARNING: A STATISTICAL VIEW
The above suggests that there is scope for improving performance on long-tail problems, both in terms of post-hoc correction and loss modification. We now show how a statistical perspective suggests simple procedures of each type, both of which overcome the limitations discussed above.
Recall that our goal is to minimise the balanced error (2). A classical result is that the best pos- sible or Bayes-optimal scorer for this problem, i.e., f* argmin, X- BER(f), satisfies the following (Menon et al., 2013), (Koyejo et al., 2014, Corollary 4), (Collell et al., 2016, Theorem I):
arginax L] 5g (x) argmaxyyej) pbal(y) (1/ x)) argmax P(ar v))
where pball is the balanced class- paas-prroaability as per $2 In words, the Bayes-optimal prediction is the most likely. Consequently, for fixed conditonnss P(x y), varying the class priors Py() arbitrarily will not affect the optimal scorers. This is intuitively desirable: the balanced error is agnostic to the level of imbalance in the label distribution.
label under which the given instance
To further probe (7), suppose the underlying class-probabilities Pyy x) x (x)( for scorer s* X + R Since by definition phal(y x)) Pyy x)/P(y), (7) becomes
argmax yE[L] Iphal (y a)) = argmaxx exp(s (x))/P(v) = argmax 8 (x) - P()), (8)
i.e., we translate the optimal logits based on the class priors. Equation 8 provides the ideal predictions for optimising the balanced error, which necessarily rely on unknown quantities (z), IP(y) that
depend on the underlying distribution. Nonetheless, we may seek to approximate these quantities based our training sample. Concretely, (8) suggests two means of optimising for the balanced error:
train a model to estimate the standard Py( x)) (c.g., by minimising the standard softmax-cross entropy on the long-tailed data), and then explicitly modify its logits post-hoc as per (8)
(ii) train a model to estimate the balanced phaal I), whose logits are implicitly modified as per (8).
Such logit adjustment techniques which have been a classic approach class-imbllance (Provost, 2000) neatly align with the post-hoc and loss modification streams discussed in $2. However, unlike most previous techniques from these streams, logit adjustment is endowed with a clear statistical grounding: by construction, the optimal solution under such adjustment coincides with the Bayes-optimal solution (7) for the balanced error, i.e., it is Fisher consistent for minimising the balanced error. We now study each of the techniques (i) and (ii) in turn.
POST-HOC LOGIT ADJUSTMENT
We now propose post-hoc logit adjustment scheme for classifier trained long-tailed data. We further show this has subtle advantages over recent weight normalisation schemes.
4.1 THE POST-HOC LOGIT ADJUSTMENT PROCEDURE
When employing the softmax cross cross-entropy to train a neural network, we aim to approximate the underlying (P(y æ)) with (I) (x)(f((z)) for logits fy( wy (xr). Given learned {w, $).
4

---

Published as a conference paper at ICLR 2021
one typically predicts the label argmaxye(4) fy(#), i.e., the most likely label under the model's Pyy x). In post-hoc logit adjustment, propose instead predict, for suitable 0:
(r( exp(w, ( - argmaxye(1] [
log ng.
where π E AL (for simplex A) are estimates of the class priors P(y). e.g., the empirical frequencies on the training sample Effectively, (9) adds label-dependent offset each of the logits.
When 7 1, this can be seen as applying (8) with plugin estimate of P(v 2). i.e., Py(x) x exp(u, (())). When T * L, this can be seen as applying (8) to temperature scaled estimates (z) exp(r-!. wy ((x)). To unpack this, recall that (8) justifies post-hoe logit thresholding given access to the true probabilities Py x²). In practice, high-cupacity neural networks often produce uncalibrated estimates of these probabilities (Guo et al., 2017). Temperature scaling is a means to calibrate the estimates, and is routinely employed for distillation (Hinton al., 2015).
One may treat 7 as a tuning parameter to be chosen based on holdout calibration, e.g.. the expected calibration error (Murphy & Winkler, 1987; Guo et al., 2017), probabilistic sharpness (Gneiting et al., 2007; Kuleshov et al., 2018), or proper scoring rule such as the log-loss or squared error (Gneiting & Raftery, 2007). One may alternately fix e.g., via label smoothing (Szegedy et al., 2016; Müller et al., 2019).
and aim learn inherently calibrated probabilities,
Post-hoe logit adjustment with T is not a new idea in the classical label imbalance literature (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003; Zhou & Liu, 2006; Collell et al., 2016); however, it has had limited exploration the recent long-tail learning literature. Further, the case * 1 is important in practical usage of neural networks, owing to their typical lack of probabilistic calibration (Guo et al., 2017). Interestingly, post-hoc logit adjustment also has an important advantage over recently proposed weight normalisation techniques, we now discuss.
4.2 COMPARISON TO POST-HIOC WEIGIIT NORMALISAITON
Recall that weight normalisation involves learning logits fy(x) = /. ((x), and then post-hoc normalising the weights via wy/v, b for r > 0. We demonstrated $2 that using Vy = ||w,|la may be ineffective when using adaptive optimisers. However, even with V/ Ty. there is subtle contrast to post-hoc logit adjustment: while the former performs a multiplicative update to the logits, the latter performs an additive update. The two techniques may thus yield different orderings labels, since
wI (x)/TT w. (T2 क exp(w] (x))\</ < exp(w, ((x))/TT.
Observe that if rare label has negative score w, ((x) 0, and there is another label with positive score, then it is impossible for the weight normalisation to give y the highest score. By contrast, under logit adjustment, w. ((x)) In Ty will be lower for dominant classes, regardless of the original sign. Weight normalisation is thus not consistent for the balanced error, unlike logit adjustment.
THE LOGIT ADJUSTED SOFTMAX CROSS-ENTROPY
We now show how directly encode logit adjustment into the softmax cross-entropy. The resulting approach has an intuitive relation to existing loss modification techniques.
5.1 THE LOGIT ADJUSTED LOSS
From $3, the second approach to optimising for the balanced error is directly model phal (y 22) x P(y| =))P(y). To do so, consider the following logit adjusted softmax cross- cross-entropy loss for 7 > 0:
E(y, f())) = log
Eyeely (fv(())t = log 1
Σ
Thy Fty
efv(t)- (10)
Given scorer that minimises the above, now predict argmax,e(r)]fa fy(r) usual.
Compared to the standard softmax cross-entropy (1), the above applies label- dependde offset to each logit. Compared to (9), we directly enforce the class prior offset while learning the logits,
5

---

Published as a conference paper at ICLR 2021
rather than doing this post-hoc. The two approaches have deeper connection: observe that (10) is equivalent to using a scorer of the form 9y (x( = fz( + T log Ty. with yel) = argmax (e[])))) - r log Ty. Consequently, one can equivalently view learning with this loss as learning a standard scorer g(æ),, and post-hoc adjusting its logits. For non-convex objectives, as encountered in neural networks, the bias endowed by adding 7- log Ty to the logits is likely to result in different local minima, typically with improved performance.
5.2 COMPARISON TO LOSS MODIFICATION TECHNIQUES
For more insight into the logit adjusted loss, consider the following pairwise margin loss
f()) ag log I1+E, Av -(
(11)
for label weights Cty > 0, and pairwise label margins Agy representing the desired gap between scores for y and y'. For T 1. our logit adjusted loss (I0) corresponds (11) with ag 1 and
Ayy' = llog Tyf This demands a larger margin between rare positive (Ty ~ 0) and dominant negative (Tyy ~ 1) labels, so that scores for dominant classes do not overwhelm those for rare ones.
Existing loss modification techniques can be viewed as special cases of (1I). For example, 0g 1/ and A yy O yields the balanced loss (4). When Oty = 1, the choice Agy Ty 114 yields (5). Finally, A: = log F(ay') yields (6), where F: [0, 1] - (0, 1] is some non-decreasing function. These losses thus either consider the frequency of the positive or negative y', but not both. Remarkably, the specific choice that leads to our loss in (10) has a firm statistical grounding: ensures Fisher consistency (in the sense of, e.g., Bartlett et al. (2006)) for the balanced error. Proofs for all results are in the supplementary.
Theorem 1. For any 8 E IR4, the pairwise loss in (11) is Fisher consistent with weights and margins
0g = 5y/P(y))
Agy = log (6g /5y)-
Letting dy Ty. we immediately deduce that the logit-adjusted loss of (10) is consistent, provided our Ty is a consistent estimate of PPy).. Similarly, 6y = 1 recovers the classic result that the balanced loss is consistent. While Theorem only provides sufficient condition in multi-class setting, one can provide a necessary and sufficient condition that rules out other choices of A in the binary case.
Theorem 2. Suppose y = {1,, Let ōy = -y.- y and (2) = (1 + exp(=))-1, Then, the pairwise margin loss in (11) is Fisher consistent for the balanced error iff
a+1 o(+11) 1- = P(y=+1)
a-1 a(01)) P(y = +1)
5.3 DISCUSSION AND EXTENSIONS
Our pairwise margin loss in (11) subsumes several existing loss-correction approaches in the literature. Further, it suggests the exploration of new choices of A. For example, we shall see the efficacy of combining the implicit in the adaptive loss of Cao et al. (2019) with our proposed A. One may also generalise the formulation in Theorem 1, and employ Awy' 71 log T. 72 log Ty,, where TI, 72 > 0. This interpolates between our loss (T) = T2) and a version of the equalised loss (7I = 0).
For T = 11 a similar loss to (10) has been considered in the context of negative sampling for scalability (Yi et al., 2019): here, one samples a subset of negatives based on æ, and corrects the logits to obtain an unbiased estimate of the loss based on all negatives (Bengio & Senecal, 2008). Losses of the general form (11) have also been explored for structured prediction (Pletscher et al., 2010).
Cao et al. (2019, Theorem 2) provides rigorous generalisation bound for the adaptive margin loss under the assumption of separable data with binary labels. The inconsistency of the loss with respect to the balanced error concerns the more general scenario of non-separable multiclass data, which may occur, e.g., owing to label noise or limited model capacity. We shall subsequently demonstrate that encouraging consistency is not merely of theoretical interest, and can lead gains in practice.

---

Published as a conference paper at ICLR 2021
22.5
BIM
50
Weight romalisaton
20.0
Adaptive
Equalsed
error
40
Logt adjustment
17.5
Lug
sojusted
Belanced
Bayes
12.5
20
10.0
10
BRM
Adaplive
Equalieed
Logie
adpazzd Buyees
Scaling
Figure Results on synthetic binary classification problem. Our logit adjusted loss tracks the Bayes-optimal solution and separator (left & middle panel). Post-hoc logit adjustment matches the Bayes performance with suitable scaling (right panel); however, any weight normalisation fails.
EXPERIMENTAL RESULTS
We now present experiments confirming our main claims: (i) on simple binary problems, existing weight normalisation and loss modification techniques may not converge to the optimal solution 1$6.1); (ii) on real-world datasets, our post-hoc logit adjustment generally outperforms weight normalisation, and one can obtain further gains via our logit adjusted softmax cross-entropy ($6.2).
6.1 RESULTS SYNTHETIC DATASET
We begin with binary classification task, wherein samples from class E {+1} are drawn from 2D Gaussian with isotropic covariance and means My = y (+1,+1). We introduce class imbalance
by setting f +1) 5%. The Bayes-optimal classifier for the balanced error is (see Appendix F)
(xx)= =
P(r y = +1) P(x y = 1) = - (-1) x O,
(12)
i... it a linear separator passing through the origin. We compare this separator against those found by several margin losses based on (11): standard ERM (Agy' 0), the adaptive loss (Cao et al., 2019) (Awy' = xy -,11), an instantiation of the equalised loss (Tan et al., 2020) (A = log ) and our logit adjusted loss (Ayy' -log Tuf ). For each loss, we train an affine classifier on sample of 10, 000 instances, and evaluate the balanced error on test set of 10, 000 samples over 100 independent trials.
Figure 2 confirms that the logit adjusted margin loss attains a balanced error close to that of the Bayes-optimal, which is visually reflected by its learned separator closely matching that in (12). This in line with our claim of the logit adjusted margin loss being consistent for the balanced error, unlike other approaches. Figure also compares post-hoc weight normalisation and logit adjustment for varying scaling parameter 7 (c.f. (3), (9). Logit adjustment is seen to approach the performance of the Bayes predictor; any weight normalisation is however seen hamper performance. This verifies the consistency of logit adjustment, and inconsistency of weight normalisation ($4.2).
6.2 RESULTS ON REAL-WORLD DATASETS
We present results on the CIFAR-IO, CIFAR-100, ImageNet and iNaturalist 2018 datasets. Following prior work, we create "long-tailed versions" of the CIFAR datasets by suitably downsampling examples per label following the ExP profile of Cui et al. (2019); Cao et al. (2019) with imbalance ratio = max, (gy) ( P 100. Similarly, we use the long- version of ImageNet produced by Liu et al. (2019). We employ ResNet-32 for CIFAR, and ResNet-50 for ImageNet and iNaturalist. All models are trained using SGD with momentum; see Appendix for more details. See also Appendix D.3 for results on CIFAR under the STEP profile also considered in the literature.
Baselines. We consider several representative baselines: (i) empirical risk minimisation (ERM) on the long-tailed data, (ii) post-hoc weight normalisation (Kang al., 2020) per (3) (using My = ||w,, 2 applied to ERM, (iii) the balanced loss of Cui et al. (2019), (iv) the adaptive margin loss (Cao et al., 2019) per (5), including with the "deferred reweighting" (DRW) training scheme, and (v) the equalised loss (Tan et al., 2020) per (6), with o, = F(T,)) for the threshold- based F of Tan et al. (2020). Where possible, we report numbers for the baselines from the respective papers.
7

---

Published as a conference paper at ICLR 2021
Table 2:
Test set balanced error (averaged over 5 trials) on real-world datasets. Here, 1, 1,*, are numbers for "LDAM SGD" and "LDAM+ DRW" from Cao et al. (2019, Table 2, 3); "r-normalised" from Kang et al. (2020, Table 3, 7); and Baass--Blaaned" from Cui et al. (2019, Table 2, 3). Here, 7" refers using the best possible tuning parameter т; see Figure 3 for results on various Highlighted cells denote the best performing method for given dataset.
Method
CIFAR-IO-LT
CIFAR-100-LT ImageNet-LT
INaturalist
ERM
27.16
61.64
53.11
38.66
Weight normalisation (r 1) (Kang et al., 2020)
24.02
58.89
52.00
48.05
Weight normalisation (r = r²) (Kang et al., 2020)
21.50
58.76
49.37
34.40*
Class-balanced (Cui et al., 2019)
25.43
60.40
53.21
35.84
Adaptive (Cao et al., 2019)
26.65'
60.40
52.15
33.31
Adaptive + DRW (Cao et al., 2019)
22.971
57.967
49.85
32.00
Equalised (Tan al., 2020)
26.02
57.26
54.02
38.37
Logit adjustment post-hoc (r 1)
22.60
58.24
49.66
33.98
Logit adjustment loss (r 1)
22.33
56.11
48.89
33.64
Logit adjustment plus adaptive (r 1)
22.42
55.92
51.25
31.56
Legkt sduummen Weaght bation
uommert Worght memmaaattoon
Megthnammeboda
Sa aner
3.
11
14
Scaling pa rometteri in)
2.
Staligg paramtto
Sca paramater
Scalin parametert
(a) CIFAR-IO-LT.
(b) I00-L-
(c) ImageNet-LT.
(d) iNaturulist.
Figure 3: Comparison of balanced error for post-hoc correction techniques when varying scaling parameter T (c.f. (3), (9)). Post-hoc logit adjustment consistently outperforms weight normalisation.
We compare the above methods against our proposed post-hoc logit adjustment (9), and logit adjusted loss (10). For post-hoc logit adjustment, we fix the scalar r 1 for basic results; analyse the effect of tuning this Figure 3. We additionally evaluate a combination of our logit adjusted softmax cross-entropy with the adaptive margin of Cao al. (2019); this uses (11) with
Ayy = log((TT) / ) + Ty 114 We do not perform any further tuning of our techniques. For all methods, report the balanced error the test set. (Note that since the test sets are all balanced for these benchmarks, the balanced error is equivalent to the misclassification error.) For all methods, we pre-compute the empirical label frequency on the entire training set.
Results and analysis. Table 2 demonstrates that our proposed logit adjustment techniques consis- tently outperform existing methods. Indeed, weight normalisation with 1 is generally improved significantly by post-hoc logit adjustment (c.g., 8% relative reduction on CIFAR-10). Similarly, loss correction techniques are generally outperformed by our logit adjusted softmax cross-entropy (e.g., 6% relative reduction on iNaturalist). Cao et al. (2019) observed that their loss benefits from a deferred reweighting scheme (DRW), wherein class- dass-weighting is applied after a fixed number of epochs. Table 2 indicates this is consistently outperformed by suitable variants of logit adjustment.
Table only reports the results for logit adjustment with scalar T 1. In practice, tuming т can significantly improve performance further. Figure 3 studies the effect of tuning for post-hoc weight normalisation (using Vy = ||W,yla) and post-hoe logit adjustment. Even without any sealing, post-hoc logit adjustment generally offers superior performance to the best result from weight normalisation (cf. Table 2); with scaling, this further improved. In practice, one may choose 7 via cross-validation estimate 7" 2.6
against the balanced error on the training set. For example, CIFAR-IO-LT,, for hoc logit adjustment, for which the resulting balanced test error of 18. 73% is superior to that of weight normalisation for any T.
To better understand the gains, Figure 4 reports errors on a per-group basis, where following Kang et al. (2020) we construct three groups of classes "Many", "Medium", and "Few" comprising
8

---

Published as a conference paper at ICLR 2021
LOW
Ma
Mtayy
(0) CIFAR-IO-LT
(b) CIFAR-I8O-LT:
(c) ImageenetLLT
(d) INaturalist.
Figure 4: Comparison of per-group emror for loss modification techniques. We construct three groups of classes: "Many", comprising those with least 100 training examples; "Medium". comprising those with at least 20 and at most 100 training examples; and "Few", comprising those with at most 20 training examples.
those with 2 100, between (20, 100), and I 20 training examples respectively. Logit adjustment shows consistent gains over the "Medium" and "Few" groups, albeit at some expense in "Many" group performance. See Appendix D.2 for a finer-grained breakdown.
While both logit adjustment techniques perform similarly, there is slight advantage to the loss function version. Nonetheless, the strong performance of post-hoc logit adjustment corrohorates the ability to decouple representation and classifier learning in long-tail settings (Zhang et al., 2019). A reference implementation of our methods is planned for release at:
https://githhbb
DISCUSSION AND FUTURE WORK
Table 2 shows the advantage of logit adjustment over recent proposals, under standard setups from the literature. Further improvements are possible by fusing complementary ideas, and we remark few such options. First, one may use a more complex base architecture; e.g., Kang et al. (2020) found gains by employing ResNet-152 and training for 200 epochs. Table 3 (Appendix) confirms that logit adjustment similarly benefits from this choice, achieving a balanced error of 30.12% on iNaturalist, and 28. 08.02% when combined with the adaptive margin. Second, the DRW training scheme (which applies to any loss) may result in further gains for our techniques. Third, incorporating developments in meta-learning (Wang et al., 2017; Jamal et al., 2020) is also a promising avenue. While further exploring such variants are of empirical interest, we hope to have illustrated the conceptual and empirical value of logit adjustment, and leave this for future work.
REFERENCES
Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 2006.
Y. Bengio and J. S. Senecal. Adaptive importance sampling to accelerate training of a neural probabilistic language model. Trans. Neur. Netw,, 19(4):713-72) April 2008. ISSN 1045-9227.
Kay H. Brodersen, Cheng Soon Ong, Klaas E. Stephan, and Joachim M. Buhmann. The balanced accuracy and its posterior distribution. In Proceedings of the International Conference on Pattern Recognition (ICPR), pp. 3121-3124, Aug 2010.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. arXiv: [es, stat], October 2017.
Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 872-881, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution aware margin loss. In Advances in Neural Information Processing Systems, 2019.

---

Published as a conference paper at ICLR 2021
Philip K. Chan and Salvatore J. Stolfo. Learning with non-uniform class and cost distributions: Effects and a distributed multi-classifier approach. In KDD-98 Workshop on Distributed Data Mining, 1998.
Nitesh Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research (JAIR), 16: 321-357, 2002.
Guillem Collell, Drazen Prelec, and Kaustubh R. Patil. Reviving threshold-moving: simple plug-in bagging ensemble for binary and multiclass imbalanced data. CoRR,
2016.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song. and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
Charles Elkan. The foundations cost-sensitive learning. Proceedings of the International Joint Conference Artificial Intelligence (IJCAT), 2001.
Tom Fawcett and Foster Provost. Combining data mining and machine learning for effective user profiling. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 13. AAAI Press, 1996.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-37, 2007.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): 243-268, 2007.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.0222 02677, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1321-1330, 2017.
Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263-1288 2009.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Geoffrey E. Hinton. Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1500.00231, 2015.
Muhammad Abdullah Jamal, Matthew Brown, Ming- Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Byungju Kim and Jummo Kim. Adjusting decision boundary for class imbalanced learning, 2019.
Vladimir Koltchinskii, Dmitriy Panchenko, and Fernando Lozano. Some new bounds on the general- ization error of combined classifiers. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 245-251. MIT Press, 2001.
Oluwasammi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, and Inderjit S. Dhillon. Consistent binary classification with generalized performance metrics. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. Cambridge, MA, USA, 2014. MIT Press.
10

---

Published as a conference paper at ICLR 2021
Miroslav Kubat and Stan Matwin. Addressing the curse of imbalanced training sets: One-sided selection. In Proceedings of the International Conference on Machine Learning (ICML), 1997.
Miroslav Kubat, Robert Holte, and Stan Matwin, Learning when negative examples abound. In Maarten van Someren and Gerhard Widmer (eds.), Proceedings of the European Conference Machine Learning (ECML), volume 1224 of Lecture Notes in Computer Science, pp. 146-153. Springer Berlin Heidelberg, 1997. ISBN 978-3-540-62858-3.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2796-2804, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
M. Lapin, M. Hein, and B. Schiele. Analysis and optimization of loss functions for multiclass, top-k. and multilabel classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 10:1533-1554, July 2018.
Yaoyong Li, Hugo Zaragoza, Ralf Herbrich, John Shawe-Taylor, and Jaz S. Kandola. The perceptron algorithm with uneven margins. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp. 379-386, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737.
Yi Lin. A note on margin- based loss functions in classification. Statistics & Probability Letters, 68 (1):73-822 2004. ISSN 0167-7152.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546. Computer Vision Foundation IEEE, 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 185-201, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01216-8.
Marcus A. Maloof. Learning when data sets are imbalanced and when costs are unequal and unknown. In ICML 2003 Workshop on Learning from Imbalanced Datasets, 2003.
Hamed Masnadi- -hirrazi and Nuno Vasconcelos. Risk minimization, probability elicitation, and cost- sensitive SVMs. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp. 759-766, Madison, WI, USA, 2010. Omnipress. ISBN
9781605589077.
Aditya Krishna Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In Proceedings of the 30th International Conference on Machine Learning. ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 603-611, 2013.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS' 13, pp. 3111-3119, Red Hook, NY, USA, 2013, Curran Associates Inc
Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with knowledge- approach a case study intensive care monitoring. Proceedings of the Sixteenth International Conference on Machine Learning (ICML), pp. 268-277 San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver. BC, Canada, pp- 4696-4705, 2019.
11

---

Published as a conference paper at ICLR 2021
Allan H. Murphy and Robert L. Winkler. A general framework for forecast verification. Monthly Weather Review, 115(7):130-1133, 1987.
Patrick Pletscher, Cheng Soon Ong, and Joachim M. Buhmann. Entropy and margin maximization for structured output learning. In José Luis Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag (eds.). Machine Learning and Knowledge Discovery in Databuses, pp. 83-98, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
Foster Provost. Machine learning from imbalanced data sets 101. In Proceedings of the AAA1-2000 Workshop Imbalanced Data Sets, 2000.
Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research, 1 11:2387-2422, 2010.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826, 2016.
J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and Yan. Equalization loss for long-tailed object recognition. In 2020 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11659-11668, 2020.
Keiji Tatsumi and Tetsuzo Tanino, Support vector machines maximizing geometric margins for multi-class classification. TOP, 22(3):815-840, 2014.
Keiji Tatsumi, Masashi Akao, Ryo Kawachi, and Tetsuzo Tanino. Performance evaluation of multiobjective multiclass support vector machines maximizing geometric margins. Numerical Algebra, Control Optimization, 1:151, 2011. ISSN 2155-3289.
Grant Van Horn and Pietro Perona, The devil is in the tails: Fine- grained classification in the wild. arXiv preprint arXiv: 1rX.v:170.0015500 2017.
B.C. Wallace, K.Small, C.E. Brodley, and T.A. Trikalinos. Class imbalance, redux. In Proc. ICDM, 2011.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning model the tail. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 7029-7039. Curran Associates, Inc., 2017.
Shan-Hung Wu, Keng-Pei Lin, Chung-Min Chen, and Ming-Syan Chen. Asymmetric support vector machines: Low false-positive learning under the user tolerance. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, pp. 749-757, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 97816055819 1334.
Yu Xie and Charles F. Manski. The logit model and response-based samples. Sociological Methods & Research, 17(3):283-302, 1989.
Han-Jia Ye, Hong-Yow Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning, 2020.
Xinyang Yi, Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias-coreess bios-coreeceedd neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys '19, pp. 269-277, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362436.
Xi Yin, Xiang Yu. Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for deep face recognition with long-tail data. CoRR, abs/1803.0900) 2018.
Junjie Zhang, Lingqiao Liu, Peng Wang, and Chunhua Shen. To balance or not to balance: A simple-yet-effective approach for learning with long-tailed distributions, 2019.
12

---

Published as a conference paper at ICLR 2021
X. Zhang. Z. Fang, Y. Wen, Z. Li, and Y. Qiao. Range loss for deep face recognition with long-tailed training data. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5419-5428, 2017.
Zhi-Hua Zhou and Xu- Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trunsactions on Knowledge and Data Engineering (TKDE), 18(1) 2006.
13

---

Published as a conference paper at ICLR 2021
Supplementary material for "Long tail learning via logit adjustment
A PROOFS OF RESULTS IN BODY
Proof of Theorem I. Denote Tly (x) = PP(y | x). Suppose we employ a margin Awy log 2/ Then,
the loss is
f(y.f(x) log 8y cfo(x)
cfuln)+oon
Eve(L) by efv(x)) log
Eve(r elw(
Consequently, under constant weights og 1, the Bayes-optimal score will satisfy fy (x) log by log (2)), or (z) log Now suppose generic weights Rt. The risk under this loss is
(x) 6%
Ex,y (v. f(x)){
Σ VEIL
Ty
Ex| f(x))
Σ /E[L]
Ty
Ex|s=y Ka f()))
> Ty - ay Er (, f(, f()))
ME(L]]
Σ DE[L]
Ty. E Ex 10 f(x))),
where Ty OC Ty. ay. Consequently, learning with the weighted loss is equivalent to learning with the original loss, a distribution with modified base-rates म. Under such distribution, we have class-conditional distribution
(x) = Ply x)) P(x | y) - ny
Ty P(x) Ty P(x)
-my(x)-
o (1())) ay.
P(x)
Consequently, suppose Oy 6p Then, (x( = log Fy(x) = log (m(
C(x), where ((x) does not which is the Bayes-optimal
8,
depend y. Consequently, argmax fū argmax 4y prediction for the balanced error.
(z)
In sum, consistent family can be obtained by choosing any set of constants by > and setting
dy
My
log 6y 8,
Ayy
Proof of Theorem 2. We establish general result in Lemma 3 of the next section, which allows for a temperature parameter the loss. This allows for interpolating between the standard softmax cross-entropy and margin based losses.
14

---

Published as a conference paper at ICLR 2021
B ON THE CONSISTENCY OF BINARY MARGIN-BASEI LOSSES
It is instructive to study the pairwise margin loss (11) in the binary case. Endowing the loss with temperature parameter 7 > get"
f)+1,f) W+1 log(1 e7 --71)
(13)
f(-1,f) = w-1 log(1 + e -77-))
7
constants W > 0 and d=1 E Here, we have used 6+1 = 1+1-1 and 8-1 A-1.+1 for simplicity. The choice we1 1, 0+1 0 recovers the temperature scaled binary logistic loss. Evidently, as "y + +00, these converge to weighted hinge losses with variable margins, i.c.,
f(+1,)) W+1 [641 fl+ {(-1,))= - 5__1 + f]+-
We study two properties of this family losses. First, under what conditions are the losses Fisher consistent for the balanced error? We shall show that in fact there is simple condition characterising this. Second, do the losses preserve properness of the original binary logistic loss? We shall show that this always the case, but that the losses involve fundamentally different approximations.
B.1 CONSISTENCY OF THE BINARY PAIRWISE MARGIN LOSS
Given a loss É, its Bayes optimal solution is f argmin x-R E [e(y, f(r))) For consistency with respect to the balanced error in the binary case, we require this optimal solution f to satisfy f (x) = ry(x) т, where (2x) = Pyy 1 x) and P(y 1) (Menon al., 2013). This is equivalent to a simple condition on the weights w and margins of the pairwise margin loss.
Lemma 3. The losses in (13) are consistent for the balanced error iff
W+1 2(7-84)) 1
W-1 007.801)
where (z) = (1 + exp(x))-
Proof of Lemma 3. Denote 772) = Ply = +1 x), and i = Ply +1). From Lemma below,
the pairwise margin loss is proper composite with invertible link function v: [0, 1] + R U (000). Consequently, since by definition the Bayes-optimal score for a proper composite loss is (*(x) = V (n((z)) (Reid& & Williamson, 2010), to have consistency for the balanced error, from (14), (15), require
T 0-'(0) = n
((+1,))
71-1,00
(++1,0)
1
((-1,0)
((+1,0)
(0-1,
W+1 0( 541)
w-1 a(7 8-1)
From the above, admissible parameter choices include:
W41 1. - 8+1 1; i.c. the standard weighted loss with constant margin
Compared to the multiclass case, assume here scalar score E R. This is equivalent to constraining that Σ, Evere fy for the multiclass case.
15

---

Published as a conference paper at ICLR 2021
W+1 6,1 = log 1- 1, 6-1
log ; i.e., the unweighted loss with margin biased towards the rare class, per our logit adjustment procedure
The second example above is unusual in that it requires scaling the margin with the temperature; consequently, the margin disappears as y + +00. Other combinations are of course possible, but note that one cannot arbitrarily choose parameters and hope for consistency in general. Indeed, some inadmissible choices are naïve applications of the margin modification or weighting, e.g.,
8-1 = 7 log -xx i.e. combining both w+1 x. w-1 - 6+1 1 log 1-x m weighting and margin modification
7
w=1 L, 6+1 1 (1 x). 8-1 = æ, i.e. specific margin modification
Note further that the choices Cao et al. (2019); Tan et al. (2020) do not meet the requirements of Lemma 3.
We make two final remarks. First, the above only considers consistency of the result of loss minimisation. For any choice of weights and margins, may apply suitable post-hoe correction to the predictions to account for any bias in the optimal scores. Second, + +o0, any constant margins 0=1 0 will have no effect on the consisteney condition, since ( 8+1) - 1. The condition will be wholly determined by the weights wil For example, we may choose W+1 = +
w-1 T- 5+1 1, and 0_1 - the resulting loss will not be consistent for finite but will become so in the limit + +00.
B.2 PROPERNESS OF THE PAIRWISE MARGIN LOSS
In the above, we appealed to the pairwise margin loss being proper composite, in the sense of Reid Williamson (2010). Intuitively, this specifies that the loss has Bayes-optimal score of the form (z) = (n(x)), where y is some invertible function, and nœx) = Py = =1 |x). We have the following general result about propernsss of any member of the pairwise margin family.
Lemma 4. The losses in (13) are proper composite, with link function
1 log
(a.b.
-c)
a-
(p))
+
+ 4
log 2,
7
q
where W12 7d41 w-1 1 b ero-,, c eb, and q
1-2
Proof of Lemma 4. The above family of losses is proper composite iff the function
(14)
((+10) F(-IJ)
invertible (Reid & Williamson, 2010, Corollary 12); the resulting inverse is the link function V We have
erb+1.e-11
f'(+1,f) W-11
1+er e- gTō-1. .7.
(15)
((-1,)) +W-1 1 erd11.e7
The invertibility of (14) is immediate. To compute the link function V. note that
1
1 =1 (((+1,))
p 1
c(+1f) P(-1.))
p (((1,f)
((+1,f) 1-p
1(-1,f) p
W+1 6 -r-
+ 4 1.67
1 4 .aaiiee-h --
W-1
16

---

Published as a conference paper at ICLR 2021
W+1 e
+ 170-111
1-22
6 er +er0+1
ear
1 b.
a g² + c
4:
where =ii 7d --
b : r 9 e. and q
1-2 Thus,
--1
p
1+ b.g. g
g + c g
a
q
8 + c.g
+ b g
q
b
- g²
9
0
g
q
2
a.b
+
n-b
+
4
9
9
2
As sanity check, suppose b C 7 1.
This corresponds to the standard logistic loss. Then,
+
-11
+ 4
(p) log
log
P
2
which is the standard logit function.
Figure and compares the link functions for a few different settings:
the balanced loss, where w41 1 w-1
and 6
unequal margin loss, where wil 8+1 7 llog 1-T and 5_1 log 1-
x
balanced + margin loss, where w41 1. w-1 1- 8+1 1, and 8-1 1-x
The property Vi (0) = æ for - = PPy = holds for the first two choices with any >> 0, and the third choice - +00. This indicates the Fisher consistency of these losses for the balanced error. However, the precise way this is achieved is strikingly different in each case. In particular, each loss implicitly involves a fundamentally different link function.
To better understand the effect of parameter choices, Figure illustrates the conditional Bayes risk curves, i.c.,
L(p) = ((-1, (p)) (1- ((t,,
We remark here that for the balanced error, this function takes the form L(p) = p Ip < m] + (1 p) [p m]] i.e., it is "tent shaped" concave function with maximum at p n.
For ease of comparison, we normalise this curves to have a maximum of 1. Figure shows that simply applying unequal margins does not affect the underlying conditional Bayes risk compared to the standard log-loss; thus, the change here is purely in terms of the link function. By contrast, either balaneing the loss or applying combination of weighting and margin modification results closer approximation to the conditional Bayes risk curve for the cost-sensitive loss with cost π.
17

---

Published as a conference paper at ICLR 2021
1.5
u(
0.0
-2
Balanced
-0.5
Balanced
Unequal margin
-1.0
Unequal margin
-4
Balanced margin
Balanced + margin
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
P
Figure 5: Comparison of link functions for various losses assuming 0.2, with 1 (left) and
= (right). The balanced loss uses wy = "x The unequal margin loss uses by y loog 1- The
balanced margin loss uses 8_1 -- 8+1 1, 4/41 =
1.0
Balanced
Balanced
Unequal margin
Unequal margin
0.8
Balanced margin
Balanced margin
0.6
0.6
0.4
0.2
0.2
0.0
0.0
-4
-2
2
-2
0
2
Figure 6: Comparison of link functions for various losses assuming π = 0.2, with 1 (left) and
(right). The balanced loss uses wy Ty. The balanced + margin loss uses 6_1 =
1. The unequal margin loss uses by 1 -log 1-
x 1-w 541 1. 1,441 4'
1.0
Balanced
1.0
Balanced Unequal margin Balanced margin
Unequal margin Ba Salanced + margin
L(cb
0.6
0.4
10.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
Figure 7: Comparison of conditional Bayes risk functions for various losses assuming = 0.2, with
7 = 1 (left) and 7 = (right). The balanced loss uses wl 1. The unequal margin loss uses
n.. 5, - log 1_ The first balanced + margin loss uses 8_1 ग. 8+1 = 1, 1,441 = 1. The second
balanced margin loss uses 8_1 -= 8+1 1, w411 $
18

---

Published as a conference paper at ICLR 2021
C EXPERIMENTAL SETUP
Intending a fair comparison, we use the same setup for all the methods for each dataset. Unless otherwise specified, all networks are trained with SGD with momentum value of 0.9, a linear learning rate warm-up in the first 5 epochs to reach the base learning rate, and a weight decay of 10 10-4. Other dataset specific details are given below.
CIFAR-10 and CIFAR-100: We use CIFAR ResNet-32 model trained for 120K steps with a batch size of 128. The base learning rate is 0.1, with linear warmup for the first 2000 steps, and decay of 0.1 at 60K, 90K, and 110K steps.' We also use the standard CIFAR data augmentation procedure used in previous works such as Cao al. (2019); He et (2016), where 4 pixels are padded on each size and random 32 x 32 crop is taken. Images are horizontally flipped with a probability of 0.5.
ImageNet: We use a ResNet-50 model trained for 90 epochs with batch size 512. The base learning rate is 0.4, with cosine learning rate decay and Nesterov momentum, We use weight decay 5 x 10-4 following Kang et al. (2020). Additionally, we apply standard data augmentation comprising of random cropping and flipping described in Goyal et al. (2017).
iNaturalist:
We use a ResNet-50 trained for 90 epochs with a batch size of 1024. (Note however that
for the ResNet-152 experiments in Appendix D.I, we use a batch size of 512.) The base learning rate is 0.4, with cosine learning rate decay. The data augmentation procedure the same as used in ImageNet.
'Qualitatively similar results may be achieved with other schedules, which involve training for fewer steps. For example, one may train for 256 epochs with base learning rate of 0.4, linear warmup for the first 15 epachs, and decay of 0.1 at the 96th, 192nd, and 224th epoch.
19

---

Published as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTS
We present here additional experiments:
present detailed table of results with more complex base architecture and number of training epochs for ImageNet-LT and iNaturalist;
present results for CIFAR-10 and CIFAR-100 the STEP profile (Cao et al., 2019) with 100.
(iii) we present results on synthetic data with varying imbalance ratios.
D.1
RESULTS WITH MORE COMPLEX BASE ARCHITECTURE
Table 3 presents results when using ResNet- 152, trained for either 90 or 200 epochs, on the larger ImageNet-LT and iNaturalist datasets. Consistent with the findings in Kang et al. (2020), training with a more complex architecture for longer generally yields significant gains. Logit adjustment, potentially when combined with the adaptive margin, is generally superior baselines with the sole exception of results for ResNet-152 with 200 epochs on iNaturalist.
Table 3: Test set balanced error (averaged over 5 trials) on real-world datasets with more complex base architectures. Employing ResNet- 152 systematically improves all methods' performance, with logit adjustment remaining superior to existing approaches. The final row reports the results of combining logit adjustment with the adaptive margin loss of Cao et (2019), which yields further gains iNaturalist.
ImageNet-LT
iNaturalist
Method
ResNet-50
ResNet-152
ResNet-50 90 epochs
ResNet-152
ResNet-152
90 epochs
200 epochs
ERM
53.11
53.30
38.66
35.88
34.38
Weight normalisation 1) (Kang et al., 2020)
52.00
51.49
48.05
45.17
45.33
Weight normalisation r*) (Kang et al., 2020)
49.37
48.97
34.10
31.85
30.34
Learnable weight scaling (LWS) (Kang et al., 2020)
52.30
49.50
34.10
30.90
27.90
Classifier retraining (cRT) (Kang et al., 2020)
52.70
49,90
34.80
31.50
28. 28.80
Adaptive (Cao al., 2019)
52.15
53.34
31.18
Equalised (Tan et al., 2020)
54,02
51.38
38.37
35.86
34,53
Logit adjustment post-hoc (T 1)
49.66
49.25
33.98
31,46
30.15
Logit udjustment post-huc (7
49.56
49.15
33.80
31.08
29.74
Logit udjustment loss 1))
48.89
47.86
33.64
31.15
30.12
Logit adjustment plus adaptive loss
51.25
50.46
31.56
29.22
28.02
D.2 PER-CLASS ERROR RATES
Figure 8 breaks down the per-class accuracies on CIFAR-10, CIFAR-100, and iNaturalist. On the latter two datasets, for ease of visualisation, we aggregate the classes into ten groups based on their frequency-sorted order (so that, e.g., group 0 comprises the top 10 most frequent classes). As expected, dominant classes generally see lower error rate with all methods. However, the logit adjusted loss is seen to systematically improve performance over ERM, particularly on rare classes.
L
D.3 RESULTS ON CIFAR-LT WITH STEP-100 PROFILE
Table 4 summarises results on the STEP-100 profile. Here, with = 1, weight normalisation slightly outperforms logit adjustment. However, with 1, logit adjustment is again found to be superior (54.80); see Figure 9.
D.4 RESULTS ON SYNTHETIC DATA WITH VARYING IMBALANCE RATIO
Figure 10 shows results on the synthetic data of $6.1 for varying choice of P(y = +1). As expected, that Ply +1) increases, all methods become equitable in terms of performance. We
20

---

Published as a conference paper at ICLR 2021
Adaptive
Adaptive
Equalised
Equalised
Logit adjusted
Logil adjusted
0.3
on
0.4
0.0
9
9
Class index
Class group
(a) CIFAR-IO-LII 10-LT.
(b) CIFAR-100-LT.
ERM
0.8
Adapitive
0.5
Equailised
0.6
udjisttddd
Loga
D.3
ERM
0.4
Error
Adaptive
Equalised
0.1
adjusted
0
2
4
6
8
10
Class group
Class group
(c) ImageNet-LT.
(d) iNaturalist.
Figure 8: Per-class error rates of loss modification techniques. For (b) and (c), we aggregate the classes into 10 groups. ERM displays a strong bias towards dominant classes (lower indices). Our proposed logit adjusted softmax loss achieves significant gains rare classes (higher indices).
Table Test set balanced error (averaged over 5 trials) on CIFAR-IO-LT and CIFAR- under the STEP-100 profile; lower is better. On CIFAR-I00-LT, weight normalisation edges out logit adjustment. See Figure for a demonstrated that tuned versions the same outperfom weight
normalisation.
Method
CIFAR-10-LT
CIFAR-100-LT
ERM
36.54
60.23
Weight normalisation (r = 1)
30.86
55.19
Adaptive
34.61
58.86
Equalised
31.42
57.82
Logit adjustment post-hoe (r = 1)
28.66
55.82
Logit adjustment (loss)
27.57
55.52
generally find a consistent trend in the relative performance of the various methods, which matches the results the body.
21

---

Published as a conference paper at ICLR 2021
50
Logit adjustment
75
Logit adjustment Weight normalisation
45
Welght Weight norma lisation
Balanced error
40
Balanced error
35
65
30
25
60
20
0
1
2
3
0.0
0.5
1.0
1.5
2.0
Scaling parameter (x)
Scaling parameter (x)
(a) CIFAR-1O-LT.
(b) CIFAR-100-ITT.
Figure 9: Post-hoc adjustment STEP-100 profile, CIFAR- and CIFAR-100-LT Logit adjust- ment outperforms weight normalisation with suitable tuning.
22.5
20.0
17.5
Balanced
13.0
10.0
CRM
Adaptive
Equalied
adjustes Bayes
ERM
Adaptive
Eqpakaad
vdlecttdd Buyas
ERM
Acaptive
Equalaad
adecided Bloyes
(a) Ply +1) 1%.
(b) P(y +1) 2%
(c) (
+1) 5%
5.75
Balanced umat
Relanadd aner
0.25
8..0
Adopiive
Equatied
Logg injurte Bay
Adaptoe
Espatiedd
Lagd
Bayas
Acapllvee
Equalied
Lagil
Baytt
(d) (
10%.
(e) IP
+1)
20%.
P(y
+1) 40%.
Figure 10: Results on synthetic data with varying imbalance ratio.
22

---

Published as a conference paper at ICLR 2021
E DOES WEIGHT NORMALISATION INCREASE MARGINS?
Suppose that one uses SGD with a momentum, and finds solutions where ||ıcyl2 tracks the class priors. One intuition behind normalisation of weights is that, drawing inspiration from the binary case, this ought to increase the classification margins for tail classes. Unfortunately, as discussed below, this intuition is not necessarily borne out.
Consider scorer fx) = w. ((=),, where wy E RR and o: X - Rd The functional margin for an example (æ, y) is (Koltchinskii et al., 2001)
(z, 3) = w. (z)) max w y (x).
(16)
8'
This generalises the classical binary margin, wherein by convention y = {+11. w-1 W11, and
((, y) y wi (x) = 2 )()])(x>>>)))))))))())) w w. (x))
(17)
which agrees with (16) upto scaling. One may also define the geometric margin in the binary case to be the distance of (z, y) from its classifier:
[0g ((x)
(x)
(18)
|wil 2|2
Clearly, (z) = (x)(, and so for fixed functional margin, one may increase the geometric walla
margin by minimising ||w]]] |12. However, the same is not necessarily true in the multiclass setting, since here the functional and geometric margins do not generally align (Tatsumi et al., 2011; Tatsumi & Tanino, 2014). In particular, controlling each ||w, ll does not necessarily control the geometric margin.
23

---

Published as a conference paper at ICLR 2021
F BAYES-OPTIMAL CLASSIFIER UNDER GAUSSIAN CLASS-CONDITTONALS
Derivation of (12). Suppose
P(x v) 1
(- || - Myll
exp
V2TO
20
for suitable By and o. Then,
| - 141112
|| - p 11
F(r y = > P(x y = = exp
> exp
20²
20²
|
M41l1 11
1 1-1
20
20²
|| p+1ll 11?
< || M-1 112
2. (M+1 1-1)
T x > | 11
Now use the fact that in our setting. || 2|2 = |
We now explicate that the class- probabiiity function for the synthetic dataset in $6.1 exactly in the family assumed by the logistic regression. This implies that logistic regression is well-specified for this problem, and thus can perfectly model Pyy +1 | x)) in the infinite sample limit. Note that
P( y +1) P(y +1)
Ply +1 x)
P(x))
P( y +1) Pp +1)
Ey P( 3/ Ply))
1
1
P(aly - 1) P(y=-1)
P(aly - + 1). Ply=
Now,
P( y 1)
I |1 1| || 14 1111 113
P(r y = +1) exp
20
|| || || 1 112 2. (141 - p-1) x
exp
202
- (141 - -1-11) Tx
exp
σ²
Thus,
P(y = +1 | x) =
1 + exp( w x + b.)
where w. = 1 - p_1, and b, = log P(y=-1) This implies that a sigmoid model for
Fly=-1)
Fly +1 x). as employed by logistic regression, is well-specified for the problem. Further, the bias term b, is seen to take the form of the log-odds of the class-priors per (8), as expected.
24


Reviews:
[{"review_id": "xxhEHRAsbc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis well-written paper re-visits the idea of logit adjustment to tackle long-tailed problems. The paper begins by setting up a statistical framework and use that to deliver two ways of realizing the logits adjustment effectively. They further prove the potential of such an approach by benchmarking it with several related baselines on both synthetic and natural long-tailed datasets.\n\n###########################################################\n+ves:\n+ The paper motivates the proposed method very well, by exposing the cases of failures of certain existing approaches and addressing those shortcomings in the proposed method.\n+ The explanation about how the proposed methods standout - that is post-hoc logit adjustment with respect to the weight normalization and logit adjusted loss with respect to the LDAM loss is clear and well written.\n+ The proposed methods are very versatile as it can both be incorporated into the training, used after the training, and in combination with each other.\n+ The experiments and the analysis are both comprehensive, and the paper has a nice technical depth to it.\n+ The paper is very well-written.\n\n###########################################################\nConcerns/Potential Improvements:\n- There is no justification as to why results on synthetic datasets are provided only for the imbalance ratio of 100 while it is a community norm to benchmark on a range of imbalance ratios (typically 100, 50, 20, 10, 1)\n- Weight normalization has a term multiplied to the logits and the proposed post-hoc logit adjustment has a term added to the argmax of the logits. Therefore both the terms are independent of each other. It would be interesting to see how both of these methods work in conjunction. Does it improve the results or the cons of one just penalizes the pros of another?\n\n###########################################################\nMinor editorial/typo issues:\n- For post-hoc logit adjustment, the paper provides a separate subsection named “COMPARISON TO POST-HOC WEIGHT NORMALISATON”. It would have been nice to see something similar for logit adjusted loss. The content is already there in the paper, and just has to be modified a bit to stand out separately.\n- Near equation 9, the paper mentions that “(8) immediately suggests two means of optimizing for the balanced error” and goes on to provide the two methods. This was not very evident from (8). Some description to link would have been nice. \n\nPOST-REBUTTAL:\n\nI thank the authors for their response. I am happy with the responses to my concerns/questions, and retain my decision of Accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G5aAtvF1LyH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers presents a general loss function for long-tail classification with several previous work as its special cases.\n\nThis is a well-written paper, and the results are impressive. The approach builds upon prior work and a general framework is presented. The proposed approaches are eveluated on several commonly used datasets and show some improvements. \n\nMy one major technical concern are as follows:\n1. The originality of this paper is not very high since the proposed framework and its components are not novel (there might be some minor novelty such as the fisher consistency property of the objective);\n2. Regarding the post-hoc logit adjustment, I am supposing it is sensitive to $\\pi_y^\\tau$, which is not very much similar with weight normalization;\n3. For the balanced error, I am interested in why it is supposed to a better performance measure, given that the test data distribution is uniform (as per datasets used in experiments);\n4. In the experiments, e.g., Table 2, in my humble opinion, some of resfults for comparison methods are incorrect. Since prior work (including Weight normalisation, Adaptive, Equalised) report Top-1 classification error, instead of the balanced error. Hence, I guess that the comparison is not fair at all.\n\n========== after reading the authors feedback =========\n\nThanks the authors for addressing my concerns and I am convinced that this work is very much different from prior literature. In addition, the evaluation metrics are correct in the studied problem setup. Based on that, I would like to raise my score from 5 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dq_xZ2DZECO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis paper proposes an unifying statistical framework for imbalanced or long-tailed data, where the number of samples for some of the classes may be extremely small compared with other classes. Previous methods work empirically well, but was not consistent, meaning that even in the infinite sample limit, the minimiser doesn't result in a minimal balanced error. The paper proposes a framework based on logit adjustment in two ways: a post-hoc logit adjustment way and another way that injects the logit adjustment to the softmax cross-entropy loss function directly. The paper shows that they are both consistent. Experiments show that the proposed methods work better than previous methods.\n\nPros:\nI feel the paper is very well organized and the story of the paper is clear.  The paper explains the issues of recent methods such as weight normalization and loss modification methods in long-tail learning, and propose a simple statistical formulation with consistency guarantee. It gives further theoretical insights for the binary classification case.\n\nThe synthetic data experiments are well designed, and it gives the reader a better understanding of the behavior of the proposed method. The proposed method matches the Bayes optimal decision boundary, while previous methods and naive ERM seem to be worse, demonstrating the characteristics of proposed and previous methods. The ablation study of the scaling parameter shows how a default value of 1 is already good enough in most cases, but can be tuned for further performance gains.\n\nCons:\nThe ImageNet-LT results were shown in the Appendix, but the proposed Post-hoc correction seems to be slightly worse than weight normalization. I feel readers will be more comfortable to have this in the main paper.\n\nOther comments:\nWhat does the gray-colored highlights in Table 2 mean? If it means the best performing method, then for CIFAR-10-LT, weight normalisation should be highlighted instead of logit adjustment loss.\n\nIt would be nicer to explain that \"the proofs for the two theorems are in the supplementary\" in page 6. Since the supplementary was in a separate PDF file, I didn't notice this when I first read this paper.\n\nOn page 8, it says \"See Appendix D.4 for a plot on ImageNet-LT\", but this should be D.2 instead.\n\n*******************\nAfter rebuttal period: thank you for answering my questions and for updating the paper. I still think it is a good paper and would like to keep my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bx7-7_fGApb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nThis paper provides a statistical framework for long-tail learning by revisiting the idea of logit adjustment based on the label frequencies. The proposed framework then yields two variant techniques that follow the paradigm of weight normalization or loss modification. Compared with the existing methods, the proposed methods are generalized and Fisher consistent for minimizing the balanced error. Finally, empirical results on four real-world datasets validate the effectiveness and statistical grounding of the proposed methods.\n\n[Pros]\n-\tThe idea of this paper is novel and interesting. The proposed framework is proved to be Fisher consistent for minimizing the balanced error and generalizes several recent methods for long-tail learning. Meanwhile, some insights about the logit adjustment technique are also revealed to help understanding.\n-\tThe experiments are sufficient and supportive to validate the effectiveness and statistical grounding of the proposed methods.\n-\tThe paper is well organized, which makes it easy to grasp the core idea.\n\n[Cons]\n-\tFrom the left panel of Fig.2 it seems that the error bar of logit adjusted is even thinner than that of the Bayes predictor. It would be better to provide some explanation.\n-\tIt seems that the proposed framework could work well with linear classifiers. Does it also applies to other classifiers such as cosine classifier?\n-\tCaptions of tables should be put above the table contents.\n-\tBesides, there are some grammatical errors and typos to be corrected. Some are lists as follows:\n  1)\tPage 3, ‘an non-decreasing transform’ should be ‘a non-decreasing transform’;\n  2)\tPage 4, ‘which overcome’ should be ‘which overcomes’;\n  3)\tPage 5, ‘has negative score’ should be ‘has a negative score’.\n  4)\tPage 5, ‘another label with positive score’ should be ‘another label with a positive score’.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper got 3 acceptance and 1 marginally below the threshold. After the rebuttal, the rating was raised to above the threshold. All the reviewers are positive about this submission. They agree that the method proposed in the submission is novel, the experiments are comprehensive and convincing. AC agrees and recommend acceptance. 

Decision: Accept (Spotlight)