Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
Year: 2021

Abstract:
Real-world data universally confronts a severe class-imbalance problem and exhibits a long-tailed distribution, i.e., most labels are associated with limited instances. The naïve models supervised by such datasets would prefer dominant labels, encounter a serious generalization challenge and become poorly calibrated. We propose two novel methods from the prior perspective to alleviate this dilemma. First, we deduce a balance-oriented data augmentation named Uniform Mixup (UniMix) to promote mixup in long-tailed scenarios, which adopts advanced mixing factor and sampler in favor of the minority. Second, motivated by the Bayesian theory, we figure out the Bayes Bias (Bayias), an inherent bias caused by the inconsistency of prior, and compensate it as a modification on standard cross-entropy loss. We further prove that both the proposed methods ensure the classification calibration theoretically and empirically. Extensive experiments verify that our strategies contribute to a better-calibrated model and their combination achieves state-of-the-art performance on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.

heading: 1 Introduction
text: Balanced and large-scaled datasets [49, 39] have promoted deep neural networks to achieve remarkable success in many visual tasks [23, 48, 21]. However, real-world data typically exhibits a long-tailed (LT) distribution [34, 42, 25, 17], and collecting a minority category (tail) sample always leads to more occurrences of common classes (head) [53, 26], resulting in most labels associated with limited instances. The paucity of samples may cause insufficient feature learning on the tail classes [64, 12, 33, 42], and such data imbalance will bias the model towards dominant labels [52, 53, 37]. Hence, the generalization of minority categories is an enormous challenge.
The intuitive approaches such as directly over-sampling the tail [9, 4, 45, 50, 5] or under-sampling the head [18, 4, 20] will cause serious robustness problems. mixup [60] and its extensions [55, 59, 10] are effective feature improvement methods and contribute to a well-calibrated model in balanced datasets [54, 61], i.e., the predicted confidence indicates actual accuracy likelihood [16, 54]. However, mixup is inadequately calibrated in an imbalanced LT scenario (Fig.1). In this paper, we raise a conception called ⇠-Aug to analyze mixup and figure out that it tends to generate more head-head pairs, resulting in unsatisfactory generalization of the tail. Therefore, we propose Uniform Mixup (UniMix), which adopts a tail-favored mixing factor related to label prior and a inverse sampling strategy to encourage more head-tail pairs occurrence for better generalization and calibration.
⇤Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Previous works adjust the logits weight [31, 12, 52, 58] or margin [6, 44] on standard Softmax cross-entropy (CE) loss to tackle the bias towards dominant labels. We analyze the inconstancy of label prior, which varies in LT train set and balanced test set, and pinpoint an inherent bias named Bayes Bias (Bayias). Based on the Bayesian theory, the posterior is proportional to prior times likelihood. Hence, it’s necessary to adjust the posterior on train set by compensating different prior for each class, which can serve as an additional margin on CE. We further demonstrate that the Bayias-compensated CE ensures classification calibration and propose a unified learning manner to combine Bayias with UniMix towards a better-calibrated model (see in Fig.1). Furthermore, we suggest that bad calibrated approaches are counterproductive with each other, which provides a heuristic way to analyze the combined results of different feature improvement and loss modification methods (see in Tab.4).
In summary, our contributions are: 1) We raise the concept of ⇠-Aug to theoretically explain the reason of mixup’s miscalibration in LT scenarios and propose Unimix (Sec.3.1) composed of novel mixing and sampling strategies to construct a more class-balanced virtual dataset. 2) We propose the Bayias (Sec.3.2) to compensate the bias incurred by different label prior, which can be unified with UniMix by a training manner for better classification calibration. 3) We conduct sufficient experiments to demonstrate that our method trains a well-calibrated model and achieves state-of-the-art results on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018.
2 Analysis of mixup
The core of supervised image classification is to find a ✓ parameterized mapping F✓ : X 2 Rc⇥h⇥w 7! Y 2 RC⇥1 to estimate the empirical Dirac delta distribution P (x, y) = 1 N PN i=1 (xi, yi) of N instances x 2 X and labels y 2 Y . The learning progress by minimizing Eq.1 is known as Empirical Risk Minimization (ERM), where L(Y = yi,F✓(X = xi)) is xi’s conditional risk.
R (F✓) = Z
x2X L (Y = y,F✓(X = x)) dP (x, y) =
1
N XN i=1 L (Y = yi,F✓(X = xi)) (1)
To overcome the over-fitting caused by insufficient training of N samples, mixup utilizes Eq.2 to extend the feature space to its vicinity based on Vicinal Risk Minimization (VRM) [8].
ex = ⇠ · xi + (1 ⇠) · xj ey = ⇠ · yi + (1 ⇠) · yj (2) where ⇠ ⇠ Beta(↵,↵),↵ 2 [0, 1], the sample pair (xi, yi), (xj , yj) is drawn from training dataset Dtrain randomly. Hence, Eq.2 converts P (X,Y ) into empirical vicinal distribution P⌫(ex, ey) = 1 N PN i=1 ⌫(ex, ey|xi, yi), where ⌫(·) describes the manner of finding virtual pairs (ex, ey) in the vicinity of arbitrary sample (xi, yi). Then, we construct a new dataset D⌫ := {(exk, eyk)}Mk=1 via Eq.2 and minimize the empirical vicinal risk by Vicinal Risk Minimization (VRM):
R⌫(F✓) = Z ex2 eX L ⇣ eY = ey,F✓( eX = ex) ⌘ dP⌫(ex, ey) = 1 M XM i=1 L ⇣ eY = eyi,F✓( eX = exi) ⌘ (3)
mixup is proven to be effective on balanced dataset due to its improvement of calibration [54, 16], but it is unsatisfactory in LT scenarios (see in Tab.4). In Fig.1, mixup fails to train a calibrated model,
which surpasses baseline (ERM) a little in accuracy and seldom contributes to calibration (far from y = x). To analyze the insufficiency of mixup, the definition of ⇠-Aug is raised.
Definition 1 ⇠-Aug. The virtual sample (exi,j , eyi,j) generated by Eq.2 with mixing factor ⇠ is defined as a ⇠-Aug sample, which is a robust sample of class yi (class yj) iff ⇠ 0.5(⇠ < 0.5) that contributes to class yi (class yj) in model’s feature learning.
In LT scenarios, we reasonably assume the instance number n of each class is exponential with parameter [12] if indices are descending sorted by nyi , where yi 2 [1, C] and C is the total class number. Generally, the imbalance factor is defined as ⇢ = ny1/nyC to measure how skewed the LT dataset is. It is easy to draw = ln ⇢/(C 1). Hence, we can describe the LT dataset as Eq.4:
P(Y = yi) = RR xi2X ,yj2Y 1(X = xi, Y = yi)dxidyjRR xi2X ,yj2Y 1(X = xi, Y = yj)dxidyj = e e C e yi , yi 2 [1, C] (4)
Then, we derive the following corollary to illustrate the limitation of naïve mixup strategy.
Corollary 1 When ⇠ ⇠ Beta(↵,↵),↵ 2 [0, 1], the newly mixed dataset D⌫ composed of ⇠-Aug samples (exi,j , eyi,j) follows the same long-tailed distribution as the origin dataset Dtrain, where (xi, yi) and (xj , yj) are randomly sampled from Dtrain. (See detail derivation in Appendix A.2.)
Pmixup(Y ⇤ = yi) = P 2(Y = yi) + P(Y = yi)
ZZ
yi 6=yj Beta(↵,↵)P(Y = yj)d⇠dyj
= e e C e yi , yi 2 [1, C]
(5)
In mixup, the probability of any (exi,j , eyi,j) belongs to class yi or class yj is strictly determined by ⇠ and E(⇠) ⌘ 0.5. Furthermore, both (xi, yi) and (xj , yj) are randomly sampled and concentrated on the head instead of tail, resulting in that the head classes get more ⇠-Aug samples than the tail ones.

heading: 3 Methodology

heading: 3.1 UniMix: balance-oriented feature improvement
text: mixup and its extensions tend to generate head-majority pseudo data, which leads to the deficiency on the tail feature learning and results in a bad-calibrated model. To obtain a more balanced dataset D⌫ , we propose the UniMix Factor ⇠⇤i,j related to the prior probability of each category and a novel UniMix Sampler to obtain sample pairs. Our motivation is to generate comparable ⇠-Aug samples of each class for better generalization and calibration.
UniMix Factor. Specifically, the prior in imbalanced train set and balanced test set of class yi is defined as Ptrain(Y = yi) , ⇡yi , and Ptest(Y = yi) ⌘ 1/C, respectively. We design the UniMix Factor ⇠⇤i,j for each virtual sample exi,j instead of a fixed ⇠ in mixup. Consider adjusting ⇠ with the class prior probability ⇡yi ,⇡yj . It is intuitive that a proper factor ⇠i,j = ⇡yj/(⇡yi + ⇡yj ) ensures exi,j to be a ⇠-Aug sample of class yj if ⇡yi ⇡yj , i.e., class yi occupies more instances than class yj . However, ⇠i,j is uniquely determined by ⇡yi ,⇡yj . To improve the robustness and generalization, original Beta(↵,↵) is adjusted to obtain UniMix Factor ⇠⇤i,j . Notice that ⇠ is close to 0 or 1 and symmetric at 0.5, we transform it to maximize the probability of ⇠i,j = ⇡yj/(⇡yi + ⇡yj ) and its vicinity. Specifically, if note ⇠ ⇠ Beta(↵,↵) as f(⇠;↵,↵), we define ⇠⇤i,j ⇠ U (⇡yi ,⇡yj ,↵,↵) as:
⇠⇤i,j ⇠ U (⇡yi ,⇡yj ,↵,↵) =
8 >><
>>:
f(⇠⇤i,j ⇡yj
⇡yi + ⇡yj + 1;↵,↵), ⇠⇤i,j 2 [0, ⇡yj ⇡yi + ⇡yj );
f(⇠⇤i,j ⇡yj
⇡yi + ⇡yj ;↵,↵), ⇠⇤i,j 2 [ ⇡yj ⇡yi + ⇡yj
, 1] (6)
Rethink Eq.2 with ⇠⇤i,j described as Eq.6:
exi,j = ⇠⇤i,j · xi + (1 ⇠⇤i,j) · xj eyi,j = ⇠⇤i,j · yi + (1 ⇠⇤i,j) · yj (7) We have the following corollary to show how ⇠⇤i,j ameliorates the imbalance of Dtrain:
Corollary 2 When ⇠⇤i,j ⇠ U (⇡yi ,⇡yj ,↵,↵),↵ 2 [0, 1], the newly mixed dataset D⌫ composed of ⇠-Aug samples (exi,j , eyi,j) follows a middle-majority distribution (see Fig.2), where (xi, yi) and (xj , yj) are both randomly sampled from Dtrain. (See detail derivation in Appendix A.3.)
P⇤mixup(Y ⇤ = yi) = P(Y = yi)
Z
yj<yi
1 ✓Z ⇠⇤i,jU (⇡i,⇡j ,↵,↵)d⇠ ⇤ i,j 0.5 ◆ P(Y = yj)dyj
= (e e C)2 ⇣ e (yi+1) e 2 yi ⌘ , yi 2 [1, C]
(8)
UniMix Sampler. UniMix Factor facilitates ⇠-Aug samples more balance-distributed over all classes. However, most samples are still ⇠-Aug for the head or middle (see Fig.2(green)). Actually, the constraint that pair xi, xj drawn from the head and tail respectively is preferred, which dominantly generates ⇠-Aug samples for tail classes with ⇠⇤i,j . To this end, we consider sample xj from Dtrain with probability inverse to the label prior:
Pinv(Y = yi) = P⌧ (Y = yi)R
yj2Y P ⌧ (Y = yj)dyj
(9)
When ⌧ = 1, UniMix Sampler is equivalent to a random sampler. ⌧ < 1 indicates that xj has higher probability drawn from tail class. Note that xi is still randomly sampled from Dtrain, i.e., it’s most likely drawn from the majority class. The virtual sample exi,j obtained in this manner is mainly a ⇠-Aug sample of the tail composite with xi from the head. Hence Corollary3 is derived:
Corollary 3 When ⇠⇤i,j ⇠ U (⇡yi ,⇡yj ,↵,↵),↵ 2 [0, 1], the newly mixed dataset D⌫ composed of ⇠-Aug samples (exi,j , eyi,j) follows a tail-majority distribution (see Fig.2), where (xi, yi) is randomly and (xj , yj) is inversely sampled from Dtrain, respectively. (See detail derivation in Appendix A.4.)
PUniMix(Y ⇤ = yi) = P(Y = yi)
Z
yj<yi
1 ✓Z ⇠⇤i,jU (⇡i,⇡j ,↵,↵)d⇠ ⇤ i,j 0.5 ◆ Pinv(Y = yj)dyj
=
(e e C) (e ⌧C e ⌧ )
⇣ e yi(⌧+1) e (⌧+yi) ⌘ , yi 2 [1, C]
(10)
With the proposed UniMix Factor and UniMix Sampler, we get the complete UniMix manner, which constructs a uniform ⇠-Aug samples distribution for VRM and greatly facilitates model’s calibration (See Fig.2 (red) & 1). We construct D⌫ := {(exk, eyk)}Mk=1 where {exk, eyk} is (exi,j , eyi,j) generated by (xi, yi) and (xj , yj). We conduct training via Eq.3 and the loss via VRM is available as:
L(eyk,F✓(exk)) = ⇠⇤i,jL(yi,F✓(exi,j)) + (1 ⇠⇤i,j)L(yj ,F✓(exi,j)) (11)

heading: 3.2 Bayias: an inherent bias in LT
text: The bias between LT set and balanced set is ineluctable and numerous studies [12, 53, 57] have demonstrated its existence. To eliminate the systematic bias that classifier tends to predict the head, we reconsider the parameters training process. Generally, a classifier can be modeled as:
ŷ = argmax yi2Y
e P di2D [(WT ) (di) yi F(x;✓)(di)]+byi
P yj2Y e P di2D [(WT ) (di) yj
F(x;✓)(di)]+byj , argmax yi2Y
e (x;✓,W,b)yi P
yj2Y e (x;✓,W,b)yj
(12)
where ŷ indicts the predicted label, and F(x; ✓) 2 RD⇥1 is the D-dimension feature extracted by the backbone with parameter ✓. W 2 RD⇥C represents the parameter matrix of the classifier.
Previous works [12, 53] have demonstrated that it is not suitable for imbalance learning if one ignores such bias. In LT scenarios, the instances number in each class of the train set varies greatly, which means the corresponding prior probability Ptrain(Y = y) is highly skewed whereas the distribution on the test set Ptest(Y = y) is uniform.
According to Bayesian theory, posterior is proportional to prior times likelihood. The supervised training process of (x; ✓,W, b) in Eq.12 can regard as the estimation of likelihood, which is equivalent to get posterior for inference in balanced dataset. Considering the difference of prior during training and testing, we have the following theorem (See detail derivation in Appendix B.1):
Theorem 3.1 For classification, let (x; ✓,W, b) be a hypothesis class of neural networks of input X = x, the classification with Softmax should contain the influence of prior, i.e., the predicted label during training should be:
ŷ = argmax yi2Y
e (x;✓,W,b)yi+log (⇡yi)+log (C)
P yj2Y e (x;✓,W,b)yj +log (⇡yj )+log (C)
(13)
In balanced datasets, all classes share the same prior. Hence, the supervised model (x; ✓,W, b) could use the estimated likelihood P(X = x|Y = y) of train set to correctly obtain posterior P(Y = y|X = x) in test set. However, in LT datasets where Ptrain(Y = yi) = ⇡yi and Ptest(Y = yi) ⌘ 1/C, prior cannot be regard as a constant over all classes any more. Due to the difference on prior, the learned parameters ✓,W, b , ⇥ will yield class-level bias, i.e., the optimization direction is no longer as described in Eq.12. Thus, the bias incurred by prior should compensate at first. To correctness the bias for inferring, the offset term that model in LT dataset to compensate is:
By = log(⇡y) + log(C) (14) Furthermore, the proposed Bayias By enables predicted probability reflecting the actual correctness likelihood, expressed as Theorem 3.2. (See detail derivation in Appendix B.2.)
Theorem 3.2 By-compensated cross-entropy loss in Eq.15 ensures classification calibration.
LB(yi, (x;⇥)) = log h 1 + X yk 6=yi e(Byk Byi ) · e (x;⇥)yk (x;⇥)yi i
(15)
Here, the optimization direction during training will convert to (X; ✓,W, b) + By . In particular, if the train set is balanced, Ptrain(Y = y) , ⇡y ⌘ 1/C, then By = log(1/C) + log(C) ⌘ 0, which means the Eq.12 is a balanced case of Eq.13. We further raise that By is critical to the classification calibration in Theorem 3.2. The pairwise loss in Eq.15 will guide model to avoid over-fitting the tail or under-fitting the head with better generalization, which contributes to a better calibrated model.
Compared with logit adjustment [44], which is also a margin modification, it is necessary to make a clear statement about the concrete difference from three points. 1) Logit adjustment is motivated by Balanced Error Rate (BER), while the Bayias compensated CE loss is inspired by the Bayesian theorem. We focus more on the model performance on the real-world data distribution. 2) As motioned above, our loss is consistent with standard CE loss when the train set label prior is the same as real test label distribution. 3) Our loss can tackle the imbalanced test set situation as well by simply setting the margin as By = log(⇡y) + log(⇡0y), where the ⇡0y represents the test label distribution. The experiment evidence can be found in Appendix Tab.D2.

heading: 3.3 Towards calibrated model with UniMix and Bayias
text: It’s intuitive to integrate feature improvement methods with loss modification ones for better performance. However, we find such combinations fail in most cases and are counterproductive with each other, i.e., the combined methods reach unsatisfactory performance gains. We suspect that these methods take contradictory trade-offs and thus result in overconfidence and bad calibration. Fortunately, the proposed UniMix and Bayias are both proven to ensure calibration. To achieve a better-calibrated model for superior performance gains, we introduce Alg.1 to tackle the previous dilemma and integrate our two proposed approaches to deal with poor generalization of tail classes. Specially, inspired by previous work [24], we overcome the coverage difficulty in mixup [60] by removing UniMix in the last several epochs and thus maintain the same epoch as baselines. Note that Bayias-compensated CE is only adopted in the training process as discussed in Sec.3.2.
Algorithm 1 Integrated training manner towards calibrated model. Input: Dtrain, Batch Size N , Stop Steps T1, T2, Random Sampler R, UniMix Sampler R⇤ Output: Optimized ⇥⇤, i.e., feature extractor parameters ✓⇤, classifier parameters W ⇤, b⇤
1: Initialize the parameters ⇥(0) randomly and calculate By via Eq.14 2: for t = 0 to T1 do 3: Sample a mini-batch B = {xi, yi}Ni=1 R(Dtrain,N ) 4: Sample a mini-batch B⇤ = {x⇤j , y⇤j }Nj=1 R⇤(Dtrain,N ) 5: Calculate UniMix factor ⇠⇤ via Eq.6 6: Construct VRM dataset B⌫ = {exk, eyk}Nk=1 via Eq.7 7: Calculate LB⌫ = E[⇠⇤i,jLB(yi, (ex;⇥(t))) + (1 ⇠⇤i,j)LB(y⇤j , (ex;⇥(t)))] via Eq.11,15 8: Update ⇥(t+1) ⇥(t) ↵r⇥(t)LB⌫ 9: end for
10: for t = T1 to T2 do 11: Sample a mini-batch B = {xi, yi}Ni=1 R(Dtrain,N ) 12: Calculate LB = E[LB(yi, (xi;⇥))] via Eq.15 13: Update ⇥(t+1) ⇥(t) ↵r⇥(t)LB 14: end for

heading: 4 Experiment

heading: 4.1 Results on synthetic dataset
text: We make an ideal binary classification using Support Vector Machine (SVM) [14] to show the distinguish effectiveness of UniMix. Suppose there are samples from two disjoint circles respectively:
z+ = {(x, y)|(x x0)2 + (y y0)2  r2} z = {(x, y)|(x+ x0)2 + (y + y0)2  r2}
(16)
To this end, we randomly sample m discrete point pairs from z+ to compose positive samples z+p = {(x+1 , y + 1 ), · · · , (x+m, y+m)}, and m negative samples z n = {(x 1 , y + 1 ), · · · , (x m, y m)} from z correspondingly, thus to generate a balanced dataset Dbal = {z+p , z n } with P (x, y) 2 z+p = P ((x, y) 2 z n ) = 0.5. For imbalance data, we sample n(n⌧ m) negative data from z to generate z n0 = {(x 0 1 , y 0 1 ), · · · , (x0 n , y0 n )}, so as to compose the imbalance dataset Dimbal = {z+p , z n0}, with P (x, y) 2 z+p P (x, y) 2 z n0 . We train the SVM model on the two synthetic datasets, and visualize the classification boundary of each dataset in Fig.3.
The SVM reaches an approximate ideal boundary on balanced datasets (Fig.3(d)) but severely deviates from the y = x in the imbalanced dataset (Fig.3(a)). As proven in Sec.2, mixup (Fig.3(b)) is incapable of shifting imbalance distribution, resulting in no better result than the original one (Fig.3(a)). After adopting the proposed UniMix, the classification boundary in Fig.3(c) shows much better results than the original imbalanced dataset, which gets closed to the ideal boundary.

heading: 4.2 Results on CIFAR-LT
text: The imbalanced datasets CIFAR-10-LT and CIFAR-100-LT are constructed via suitably discarding training samples following previous works [64, 12, 44, 6]. The instance numbers exponentially decay per class in train dataset and keep balanced during inference. We extensively adopt ⇢ 2 {10, 50, 100, 200} for comprehensive comparisons. See implementation details in Appendix C.1. Comparison methods. We evaluate the proposed method against various representative and effective approaches extensively, summarized into the following groups: a) Baseline. We conduct plain training with CE loss called ERM as baseline. b) Feature improvement methods modify the input feature to cope with LT datasets. mixup [60] convexly combines images and labels to build virtual data for VRM. Manifold mixup [55] performs the linear combination in latent states. Remix [10] conducts the same combination on images and adopts tail-favored rules on labels. M2m [33] converts majority images to minority ones by adding noise perturbation, which need an additional pre-trained classifier. BBN [64] utilizes features from two branches in a cumulative learning manner. c) Loss modification methods either adjust the logits weight or margin before the Softmax operation. Specifically, focal loss [38], CB [12] and CDT [58] re-weight the logits with elaborate strategies, while LDAM [6] and Logit Adjustment [44] add the logits margin to shift decision boundary away from tail classes. d) Other methods. We also compare the proposed method with other two-stage approaches (e.g. DRW [6]) for comprehensive comparisons.
Results. We present results of CIFAR-10-LT and CIFAR-100-LT in Tab.1. Our proposed method achieves state-of-the-art results against others on each ⇢, with performance gains improved as ⇢ gets increased (See Appendix D.1). Specifically, our method overcomes the ignorance in tail classes effectively with better calibration, which integrates advantages of two group approaches and thus surpass most two-stage methods (i.e., BBN, M2m, LDAM+DRW). However, not all combinations can get ideal performance gains as expected. More details will be discussed in Sec.4.4.
To quantitatively describe the contribution to model calibration, we make quantitative comparisons on CIFAR-10-LT and CIFAR-100-LT. According to the definition ECE and MCE (see Appendix Eq.D.2,D.3), a well-calibrated model should minimize the ECE and MCE for better generalization and robustness. In this experiment, we adopt the most representative ⇢ 2 {10, 100} with previous mainstream state-of-the-art methods.
The results in Tab.2 show that either of the proposed methods generally outperforms previous methods, and their combination enables better classification calibration with smaller ECE and MCE. Specifically, mixup and Remix have negligible contributions to model calibration. As analyzed before, such methods tend to generate head-head pairs in favor of the feature learning of majority
classes. However, more head-tail pairs are required for better feature representation of the tail classes. In contrast, both the proposed UniMix and Bayias pay more attention to the tail and achieve satisfactory results. It is worth mentioning that improving calibration in post-hoc manners [16, 63] is also effective, and we will discuss it in Appendix D.2.3. Note that LDAM is even worse calibrated compared with baseline. We suggest that LDAM adopts an additional margin only for the groundtruth label from the angular perspective, which shifts the decision boundary away from the tail class and makes the tail predicting score tend to be larger. Additionally, LDAM requires the normalization of input features and classifier weight matrix. Although a scale factor is proposed to enlarge the logits for better Softmax operation [56], it is still harmful to calibration. It also accounts for its contradiction with other methods. Miscalibration methods combined will make models become even more overconfident and damage the generalization and robustness severely.

heading: 4.3 Results on large-scale datasets
text: We further verify the proposed method’s effectiveness quantitatively on large-scale imbalanced datasets, i.e. ImageNet-LT and iNaturalist 2018. ImageNet-LT is the LT version of ImageNet [49] by sampling a subset following Pareto distribution, which contains about 115K images from 1, 000 classes. The number of images per class varies from 5 to 1, 280 exponentially, i.e., ⇢ = 256. In our experiment, we utilize the balanced validation set constructed by Cui et al. [12] for fair comparisons. The iNaturalist species classification dataset [25] is a large-scale real-world dataset which suffers from extremely label LT distribution and fine-grained problems [25, 64]. It is composed of 435, 713 images over 8, 142 classes with ⇢ = 500. The official splits of train and validation images [6, 64, 30] are adopted for fair comparisons. See implementation details in Appendix C.2.
Results. Tab.3 illustrates the results on large-scale datasets. Ours is consistently effective and outperforms existing mainstream methods, achieving distinguish improvement compared with previous SOTA c-RT [30] in the compared backbones. Especially, our method outperforms the baseline on ImageNet-LT and iNaturalist 2018 by 9.53% and 8.27% with ResNet-50, respectively. As can be noticed in Tab.3, the proposed method also surpasses the well-known two-stage methods [30, 6, 64], achieving superior accuracy with less computation load in a concise training manner.

heading: 4.4 Further Analysis
text: Effectiveness of UniMix and Bayias. We conduct extensive ablation studies in Tab.4 to demonstrate the effectiveness of the proposed UnixMix and Bayias, with detailed analysis in various combinations of feature-wise and loss-wise methods on CIFAR-10-LT and CIFAR-100-LT. Indeed, both UniMix and Bayias turn out to be effective in LT scenarios. Further observation shows that with calibration ensured, the proposed method gets significant performance gains and achieve state-of-the-art results. Noteworthy, LDAM [6] makes classifiers miscalibrated, which leads to unsatisfactory improvement when combined with mixup manners.
Evaluating different UniMix Sampler. Corollary 1,2,3 demonstrate distinguish influence of UniMix. However, the ⇠-sample can not be completely equivalent with orginal ones. Hence, an appropriate ⌧ in Eq.9 is also worth further searching. Fig.4 illustrates the accuracy with different ⌧ on CIFAR-10-LT and CIFAR-100-LT setting ⇢ = 10 and 100. For CIFAR-10-LT (Fig.4(a),4(b)), ⌧ = 1 is possibly ideal, which forces more head-tail instead of head-head pairs get generated to compensate tail classes. In the more challenging CIFAR-100-LT, ⌧ = 0 achieves the best result. We suspect that unlike simple datasets (e.g., CIFAR-10-LT), where overconfidence occurs in head classes, all classes need to get enhanced in complicated LT scenarios. Hence, the augmentation is effective and necessary both on head and tail. ⌧ = 0 allows both head and tail get improved simultaneously.
Do minorities really get improved? To observe the amelioration on tail classes, Fig.5 visualizes log-confusion matrices on CIFAR-100-LT-100. In Fig.5(e), our method exhibits satisfactory generalization on the tail. Vanilla ERM model (Fig.5(a)) is a trivial predictor which simplifies tail instances as head labels to minimize the error rate. Feature improvement [10] and loss modification [6, 58] methods do alleviate LT problem to some extent. The misclassification cases (i.e., non-diagonal elements) in Fig.5(b),5(c),5(d) become smaller and more balanced distributed compared with ERM. However, the error cases are still mainly in the upper or lower triangular, indicating the existence of inherent bias between the head and tail. Our method (Fig.5(e)) significantly alleviates such dilemma. The non-diagonal elements are more uniformly distributed throughout the matrix rather than in
the corners, showing superiority to erase the bias in LT scenarios. Our method enables effective feature improvement for data-scarce classes and alleviates the prior bias, suggesting our success in regularizing tail remarkably.

heading: 5 Related work and discussion
text: Why need calibration? To quantify the predictive uncertainty, calibration [2] is put forward to describe the relevance between predictive score and actual correctness likelihood. A well-calibrated model is more reliable with better interpretability, which probabilities indicate optimal expected costs in Bayesian decision scenarios [43]. Guo et al. [16] firstly provide metric to measure the calibration of CNN and figure out well-performed models are always in lack of calibration, indicating that CNN is sensitive to be overconfidence and lacks robustness. Thulasidasan et al. [54] point out that the effectiveness of mixup in balanced datasets originates from superior calibration modification. Menon et al. [44] further show how to ensure optimal classification calibration for a pair-wise loss.
Feature-wise methods. Intuitively, under-sampling the head [18, 4, 20] or over-sampling the tail [9, 4, 45, 50, 5] can improve the inconsistent performance of imbalanced datasets but tend to either weaken the head or over-fitting the tail. Hence, many effective works generate additional samples [11, 62, 33] to compensate the tail classes. BBN [64] uses two branches to extract features from head and tail simultaneously, while c-RT [30] trains feature representation learning and classification stage separately. mixup [60] and its variants [55, 59, 10] are effective and easy-implement feature-wise methods that convexly combine input and label pairs to generate virtual samples. However, naïve mixup manners are deficient in LT scenarios as we discussed in Sec.2. In contrast, our UniMix tackles such a dilemma by constructing class balance-oriented virtual data as describe in Sec.3.1 and shows satisfactory calibration as Fig.1 exhibits.
Loss modification. Numerous experimental and theoretical studies [46, 12, 53, 57] have demonstrated the existence of inherent bias between LT train set and balanced test set in supervised learning. Previous works [27, 28, 31, 12, 38] make networks prefer learning tail samples by additional classrelated weight on CE loss. Some works further correct CE according to the gradient generated by different samples [52, 36] or from the perspective of Gaussian distribution and Bayesian estimation [19, 32]. Meta-learning approaches [15, 1, 51, 47, 29] optimize the weights of each class in CE as learnable parameters and achieve remarkable success. Cao et al. [6] theoretically provides the ideal optimal margin for CE from the perspective of VC generalization bound. Compared with Logit Adjustment [44] motivated by balance error rate, our Bayias-compensated CE eliminates bias incured by prior and is consistent with balanced datasets, which ensures classification calibration as well.

heading: 6 Conclusion
text: We systematically analyze the limitations of mainstream feature improvement methods, i.e., mixup and its extensions in the label-imbalanced situation, and propose the UniMix to construct a more classbalanced virtual dataset that significantly improves classification calibration. We further pinpoint an inherent bias induced by the inconstancy of label distribution prior between long-tailed train set and balanced test set. We prove that the standard cross-entropy loss with the proposed Bayias compensated can ensure classification calibration. The combination of UniMix and Bayias achieves state-of-the-art performance and contributes to a better-calibrated model (Fig.1). Further study in Tab.4 shows that the bad calibration methods are counterproductive with each other. However, more in-depth analysis and theoretical guarantees are still required, which we leave for our future work.
Acknowledgments and Disclosure of Funding
This work was supported by NSFC project Grant No. U1833101, SZSTI Grant No. JCYJ20190809 172201639 and WDZC20200820200655001, the Joint Research Center of Tencent and Tsinghua.

Review summaries:
This paper studies a practical problem, namely long-tailed recognition. To address this problem, this paper proposes two methods: UniMix and compensated cross-entropy loss. To be specific, UniMix adopts a balanced data sampler to improve the number of head-tail pairs for mixup. Experimental results show the effectiveness of the proposed method.  

The authors proposed a new method based on two approaches, UniMix and Bayias, to sample input-output pairs from unbalanced datasets and use them in Mixup. The authors report experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, showing superior performance in terms of accuracy and calibration scores over a variety of baselines.  While I could get the main idea behind the proposed method, which seems appealing, I had problems understanding all the details, mainly because of a substantial lack of clarity. Sentences are often disconnected which makes it difficult to understand what the authors mean.  My score is borderline but I am open to increase it if the authors can provide a solid rebuttal and substantially improve the paper in terms of clarity.

This paper addressed the problem of long tail recognition using a novel mixup technique. The proposed mixup technique (Unimix) considered the class prior and helps to create a more balanced virtual dataset. The author provide theoretical analysis on why Unimix is a better mixup technique. Extensive experiment also demonstrates the performance gain of using Unimix.

The authors propose two models to improve calibration for long-tailed recognition. 1) a Uniform Mixup is proposed, which adopts an advanced mixing factor and sampler in favor of the minority. 2) the authors propose a Bayias loss to compensate for the prior bias. The proposed method achieves good results on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.