Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
Authors: Zhengzhuo Xu
Zenghao Chai
Chun Yuan
Venue: NeurIPS 2021 Poster (2021)
Keywords: visual recognition
data imbalance
mixup
long tail

Abstract: Real-world data universally confronts a severe class-imbalance problem and exhibits a long-tailed distribution, i.e., most labels are associated with limited instances. The naïve models supervised by such datasets would prefer dominant labels, encounter a serious generalization challenge and become poorly calibrated. We propose two novel methods from the prior perspective to alleviate this dilemma. First, we deduce a balance-oriented data augmentation named Uniform Mixup (UniMix) to promote mixup in long-tailed scenarios, which adopts advanced mixing factor and sampler in favor of the minority. Second, motivated by the Bayesian theory, we figure out the Bayes Bias (Bayias), an inherent bias caused by the inconsistency of prior, and compensate it as a modification on standard cross-entropy loss. We further prove that both the proposed methods ensure the classification calibration theoretically and empirically. Extensive experiments verify that our strategies contribute to a better-calibrated model, and their combination achieves state-of-the-art performance on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.


Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
Zhengzhuo Xu Zenghao Chai'; Chun Yuan 122
Shenzhen International Graduate School, Tsinghua University Peng Cheng Laboratory
xzz200mails 1s. tsinghua. edu. cn, zenghaochai @gmail com, yuancesz. tsinghua. edu. cn
Abstract
Real-world data universally confronts severe class-imaalance problem and ex- hibits a long-tailed distribution, i.c., most labels are associated with limited in- stances. The naïve models supervised by such datasets would prefer dominant labels, encounter serious generalization challenge and become poorly calibrated. We propose two novel methods from the prior perspective to alleviate this dilemma. First, we deduce balance-oriented data augmentation named Uniform Mixup (UniMix) to promote mixup long-tailed scenarios, which adopts advanced mixing factor and sampler in favor of the minority. Second, motivated by the Bayesian theory, we figure out the Bayes Bias (Bayias), an inherent bias caused by the incon- sistency of prior, and compensate it as modification standard cross cross-yntropy loss. We further prove that both the proposed methods ensure the classification calibration theoretically and empirically. Extensive experiments verify that our strategies contribute to a balter-calibrated model and their combination achieves oatt--ofthhealld art performance CIFAR-LT, ImageNet-LT, and iNaturalist 2018.
Introduction
Balanced and large-scaled datasets
4
39 have promoted deep neural networks to achieve re- 21. However, real-world data typically exhibits and collecting a minority category (tail) sample always
markable success many visual tasks 23 48
(LT) distribution KE! 2
leads more occurrences of common classes (head) [ 26 resulting in most labels associated with
limited instances. The paucity of samples may cause insufficient feature learning on the tail classes
42 and such data imbalance will bias the model towards dominant labels Hence, the generalization minority categories is an enormous challenge.
64
22
K
52
37
The intuitive approaches such as directly over-sampling the tail 9
50 5 or under-sampling the
head [8
20 will cause serious robustness problems, mixup 60 and its extensions [55]
59 ITO are
effective feature improvement methods and contribute
well-calibrated model in balanced datasets
[54] 61], i.e., the predicted confidence indicates actual accuracy likelihood [6] [54]. However, mixup
inadequately calibrated in an imbalanced LT scenario (Fig Db. In this paper, raise conception called E-Aug to analyze mixup and figure out that tends to generate more head-head pairs, resulting in unsatisfactory generalization of the tail. Therefore, we propose Uniform Mixup (UniMix), which adopts a tail-favored mixing factor related to label prior and a inverse sampling strategy to encourage more head-tail pairs occurrence for better generalization and calibration.
'Equal contribution. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Joint density plots of accuracy vs. confidence to measure the calibration of classifiers on during training. A well-calibrated classifier's density will lay around the red dot line y x, indicating prediction score reflects the actual likelihood of accuracy. mixup manages to regularize classifier on balanced datasets. However, both mixup and its extensions tend to be overconfident in LT sce- narios. Our UniMix reconstructs a more balanced dataset and coypes-comeensatedd CE erases prior bias ensure bet- ter calibration. Without loss of accuracy, either of proposed methods trains the same classifier more calibrated and their combination achieves the best. How to measure calibration and more visualization results are available Appendix D.2
Figure 1:
0.01
0.00
Actrracy
035
6.0
Confieace
or margin 6 on standard Softmax cross-entropy (CE) loss tackle the bias towards dominant labels. We analyze the inconstancy of label prior, which varies LT train set and balanced test set, and pinpoint an inherent bias named Bayes Bias (Bayias). Based on the Bayesian theory, the posterior proportional to prior times likelihood. Hence, it's necessary to adjust the posterior on train set by compensating different prior for each class, which can serve an additional margin on CE. We further demonstrate that the soyyias-compessteed CE ensures classification calibration and propose a unified learning manner to combine Bayias with UniMix towards a better-calibrated model (see in Fig Furthermore, we suggest that bad calibrated approaches are counterproductive with each other, which provides a heuristic way to analyze the combined results of different feature improvement and loss modification methods (see in Tab
Previous works adjust the logits weight M 12
52
58
In summary, our contributions are: 1) We raise the concept of {-Aug theoretically explain the reason of mixup's 's miscalibration in LT scenarios and propose Unimix (Sec composed of novel mixing and sampling strategies construct a more class-balanced virtual dataset. 2) We propose the Bayias (Sec 3.2 to compensate the bias incurred by different label prior, which can be unified with UniMix by traiming manner for better classification calibration. 3) We conduct sufficient experiments to demonstrate that our method trains well-calibrated model and achieves state-of- offthe-art results on CIFAR-1O-LT, CIFAR-100-LT, ImageNet-LL and iNaturalist 2018.
2 Analysis of mixup
The core of supervised image classification is to find 9 parameterized mapping Fo X e Rexhxw + Y E RCx! to estimate the empirical Dirac delta distribution Ps(2, y)
EN 5iz, yı) of N instances x E X' and labels E y. The learning progress by minimiz- ing [ is known Empirical Risk Minimization (ERM), where L(Y Vi. FolX = xx)) is ss's conditional risk.
I
N
Rs(Fo)
f v. dP
Σ LY= +x))
JEEX
To overcome the over-fitting caused by insufficient training of N samples, mixup utilizes Eq to extend the feature space to its vicinity based on Vicinal Risk Minimization (VRM) X
T E. 3. (1 .y
my
V E yı (1) yj
where& & ~ Beta(a, a), E [0.1], 1]. the sample pair (x,, y.). (Iz, yy) is drawn from training dataset Dirain randomly. Hence, Eq converts Ps Ps(X, Y) into empirical vicinal distribution P 9) N EN v(ā. T I.. ys). where v-)) describes the manner of finding virtual pairs (ž, 7 in the vicinity
of arbitrary sample (I,, Then, we construct a new dataset D, := {[ūx, MM M via Eq 2 and
minimize the empirical vicinal risk by Vicinal Risk Minimization (VRM):
(F)) Jee
{ C -
dP.
1
Σm
M
L
(3)
M
mixup is proven to be effective on balanced dataset due to its improvement of calibration but it is unsatisfactory in LT scenarios (see in Tab E. In Fig mixup fails to train calibrated model,

---

which surpasses baseline (ERM) a little in accuracy and seldom contributes to calibration (far from y = z). To analyze the insufficiency of mixup, the definition of Ê-Aug is raised.
Definition Ç-Aug. The virtual sample (āi. Wi.) generated by E with mixing factor & is defined as a Ç-Aug sample, which is robust sample of class yi (class yj) isf 2 0.5(g < 0.5) that contributes to class (class y2) in model's s feature learning.
In LT scenarios, we reasonably assume the instance number of each class is exponential with parameter À [[2] if indices are descending sorted by Ry,, where yss E [1, CJ and is the total class number. Generally, the imbalance factor is defined as p = By /Tye measure how skewed the LT dataset is. It is easy to draw X = In p/ C 1). Hence, we can describe the LT dataset as Eq
JJmEE.YJJ ((X 24, Y y,)dz,dy, P(Y U.) JuI,E,, 1(X Tú, Y yy)dr,dyj e e --x
A
Au yo E []
(4)
Then, derive the following corollary illustrate the limitation of naive mixup strategy.
Corollary When & ~ Beta a)), E [0, 1), the newly mixed dataset Dy composed of E-Aug samples (Zx,g, Vo.)) follows the same Long-tailed distribution as the origin dataset Dirains where (2,, yi) and (I Uÿ) are randomly campled from Dtroin. (See detail derivation in Appendix A.2
P mixup (Y* = y.) = -P = vi) + P(Y = =y:) JI yitus
Beta(a, a)P(a = W5)ddddy
(5)
A e -ー -- Amh yo E [1.C]
In mixup, the probability of any (Tij, Di.j) belongs to class or class is strictly determined by and (F()) 0.5. Furthermore, both (Ii; y/3 and (3rz- My) are randomly sampled and concentrated on the head instead of tail, resulting in that the head classes get more {-Aug samples than the tail ones.
Methodology
3.1 UniMix: balance-oriented feature improvement
mixup and its extensions tend to generate head- majority pseudo data, which leads to the deficiency on the tail feature learning and results in a bad-calibrated model. To obtain more balanced dataset Dy, we propose the UniMix Factor E/ related to the prior probability of each category and a novel UniMix Sampler to obtain sample pairs. Our motivation is to generate comparable Ç-Aug samples of each class for better generalization and calibration.
UniMix Factor. Specifically, the prior in imbalanced train set and balanced test set of class ys is defined as P Ptrain (Y' = y1) - Ty. and Prest (Y y:) = 1/C, respectively. We design the UniMix Factor či, for each virtual sample ēG,J instead of a fixed & in mixup. Consider adjusting & with the class prior probability y. Tyz- is intuitive that a proper factor Ex, = n.. ( + k ensures T. to be E-Aug sample of class Yj if Ty. 2 πyz. i.e., class y; occupies more instances than class y/-
However, Ei,, uniquely determined by Ty. my,, To improve the robustness and generalization. original Beta(a, ai) is adjusted to obtain UniMix Factor 47 Notice that & close or 1 and symmetric 0.5, we transform it maximize the probability of &L.j TT, (TT, + +,y)) and its vicinity. Specifically, if note & Beta(a, a) f(E: a, a).. we define ET, ~ 2 (Tyg TT. a, at) as:
f(E),
Ty
+ 1:a,;))
E 10,
kyz );
Tye Tas
Ty. + TTa
Eig ~ (Ty.: Ty.. a, a)
(6)
Tyz ;; ar, ax)
Tyz 1]
s(Ei
Sū E
Ty. + Tug
Ty. + T&g
Rethink Eq 2 with f described as Eq
テi, = G. Ty
+
fig) Ti
Ti.j = Ci. yi + (1
We have the following corollary to show how Ei ameliorates the imbalance of D Dirain
3

---

Corollary 2 When fig 2 (T Ty a, a) E [0. the newly mixed dataset D, composed of E-Aug samples (Rig. Di.)) follows a middle-majority distribution (see Fig where (T Vi and (x yi) are both randomly sampled from Dirain. (See detail derivation in Appendix A.3
misup (Y" = = = =v:) I
( (T,, 2 0.5 P(Y = Uj)dys
vn <v
(8)
A (e- - (e-Mlutt)) - - E [1,cc
e
UniMix Sampler. UniMix Factor facilitates {-Aug sam- ples more balance-distributeed over all classes. However, most samples are still Ç-Aug for the head middle (see Fig Actually, the constraint that pair 2;; drawn from the head and tail respectively is preferred, which dominantly generates ξ-Aug samples for tail classes with ft,- To this end, we consider sample 2j from Drrain with probability inverse to the label prior:
0.06
1.05
0.04
Fadter
11013
0.02
Pr PT(Y = y/)
0.01
Pine (Y = y;) Jn PT = yソ yyydy
10 20
30
40
50
60
70
80
90
Figure 2: Visualization of E-Aug samples
When r 1, UniMix Sampler is equivalent to random sampler. т < 1 indicates that zy has higher probability drawn from tail class. Note that 2; is still randomly sampled from Derain. i.e., it's most likely drawn from the majority class. The virtual sample Te, obtained in this manner is mainly a Ê-Aug sample of the tail composite with x; from the head. Hence Corollary is derived:
distribution (C = 100, p == 200) in Corol- lar 32 r-axis: class indices. y-axis: probability of each class. mixup (blue) ex- hibits the same LT distribution as origin (purple). { (green) alleviates such situation and the full pipeline (r = -1) (red) con- structs more uniform distributed dataset. See more results Appendix A.5
Corollary 3 When Ei.j
V yo t Ty Ot, ax),
[0, 1], the newly mixed dataset Dy composed of E-Aug samples (Ze.j.Di..) follows a tail-majority distribution (see 2) where (Ii. is randomly and (I5, Wj) is inversely sampled from Derains respectively, (See detai derivation in Appendix A.4
(f (T. (T, a, a)dsis 2 0.5) Pino yy)ddy;
UnRiMin (Y yk) P(Y =v:)
yi)
Us <.s
ArC - -e-t) (e-vi(rt)) ( Ui E [1,c]
(e a---e--he e- 20 (c-
(10)
With the proposed UniMix Factor and UniMix Sampler, we get the complete UniMix manner, which
constructs a uniform E-Aug samples distribution for VRM and greatly facilitates model's calibration
(See Fig 2 (red) & 1 We construct D,, == {(Tx, 0k > where { [T lk is (āi,j. Vi.5) generated by
(x4, y/) and (z Uy)- We conduct training via Eq and the loss via VRM is available as:
((i...
C(u, Fo Fo (Ti,)) + Fo
(11)
3.2 Bayias: an inherent bias in LT
The bias between LT set and balanced set is ineluctable and numerous studies 12 demonstrated its existence. To eliminate the systematic bias that classifier tends to predict the head, we reconsider the parameters training process. Generally, a classifier can be modeled as:
53 57 have
CajeD((W (1.) ((.)))))))) ((x:))(4))+b.
(.;DWWW)))
1 = arg max WEEY
E co (WW (1)) F( F(20)144)44
arg max W.YY
En V..:D,W.b),,
(12)
where ÿ indicts the predicted label, and Fr; 0) E Px1 the D-dimension feature extracted by the backbone with parameter O. W E IR DxC represents the parameter matrix of the classifier.

---

have demonstrated that it is not suitable for imbalance learning if one ignores such bias. In LT scenarios, the instances number in each class of the train set varies greatly, which means the corresponding prior probability Ptrain = y) is highly skewed whereas the distribution
Previous works 12 53
on the test set tesst (V = y) is uniform.
According to Bayesian theory, posterior is proportional to prior times likelihood. The supervised training process of (x;; W, b) in Eq I2 can regard as the estimation of likelihood, which is equivalent to get posterior for inference in balanced dataset. Considering the difference of prior during training and testing, we have the following theorem (See detail derivation in Appendix BrrendixBBI... B.I.
Theorem 3.1 For classification, let (x;;; 0 W.b) be a hypothesis class of neural networks of imput X = z, the classification with Softmax should contain the inftuence of prior, i.e., the predicted label during training should be:
(.,.D.WW.)), +log (F log (C)
① = arg max MEY Eniey W.,,,,W.),,, +log (Tuz ) +log (C)
(13)
In balanced datasets, all classes share the same prior. Hence, the supervised model vb 0, W.b) b) could use the estimated likelihood P(X = =x\Y = y) of train set to correctly obtain posterior P(Y = X| = z) in test set. However, in LT datasets where P Ptrain (Y = y/3 = Ty and PeeaseY = y:) = 1/C. prior cannot be regard as a constant over all classes any more. Due to the difference on prior. the learned parameters 0. W. b _ O will yield class-level bias, i.e., the optimization direction is no longer as described in Eq 12 Thus, the bias incurred by prior should compensate at first. To correctness the bias for inferring, the offset term that model in LT dataset compensate is:
By log(TT)) + log(C)
(14)
Furthermore, the proposed Bayias B, enables predicted probability reflecting the actual correctness
likelihood, expressed as Theorem (See detail derivation in Appendix B.2
Theorem 3.2 By compensatedd cross-entropy loss in Eq ensures classification calibration.
(y (x; 0)) log [ + Σ yu
e v..) -
(15)
Here, the optimization direction during training will convert to (X; 0, W. + By. In particular, the train set is balanced, IP PPrrsn(Y y) P Ty. =I/C, then B, = log (1/C) + IC) = O, which We further raise that Hy is critical to the classification
means the Eq 12 is balanced case of Eq 13 calibration in Theorem 3.2] The pairwise loss in Eq will guide model to avoid over- over-fitting the tail or fnder-fitting the head with better generalization, which contributes to better calibrated model.
Compared with logit adjustment 4e which is also a margin modification, it is necessary to make clear statement about the concrete difference from three points. 1) Logit adjustment is motivated by Balanced Error Rate (BER), while the Bayias compensated CE loss is inspired by the Bayesian theorem. We focus more on the model performance motioned above, our loss is consistent with standard CE loss when the train set label prior is the same as real test label distribution. 3) Our loss can tackle the imbalanced test set situation as well by simply setting the margin as B, = log( nT)) + log (T)) where the TT represents the test label distribution. The experiment evidence can be found in Appendix Tab D2
the real-world data distribution. 2) As
3.3 Towards calibrated model with UniMix and Bayias
It's intuitive integrate feature improvement methods with loss modification ones for better per- formance. However, we find such combinations fail in most cases and are counterproductive with each other, i.e.. the combined methods reach unsatisfactory performance gains. We suspect that these methods take contradictory trade-offs and thus result in overconfidence and bad calibration. Fortunately, the proposed UniMix and Bayias are both proven to ensure calibration. To achieve a better-calibe caltbreed model for superior performance gains, we introduce Alg to tackle the previous dilemma and integrate our two proposed approaches to deal with poor generalization of tail classes. Specially, inspired by previous work [24] we overcome the coverage difficulty in mixup [60] by removing UniMix in the last several epochs and thus maintain the same epoch as baselines. Note that coypensccompeestted CE is only adopted in the training process as discussed in Sec 3.2
5

---

Algorithm Integrated training manner towards calibrated model.
Input: Dtrain Batch Size N, Stop Steps 7i, T2, Random Sampler R, UniMix Sampler R
Output: Optimized O*, i.e., feature extractor parameters 0", classifier parameters W*,
1: Initialize the parameters ( randomly and calculate By via Eq 14
2: for t.= 0 to Ti do
3: Sample mini-batch B = {a;, V/ i==1 f- R(Dtrain: .N)
4: Sample mini- 8 {2} y LV - R (Dtrain N)
5: Calculate UniMix factor ?" via Eq
6: Construct VRM dataset B, [T Uk k=1 via Eq
f 7: Calculate LB, El[,, S (Vi, (T;;; ()))) (1 Lo, 4G; [1)) via Eq
8: Update ([+1) +- ( - a aV LB.
9: end for
10: for == Ti to T: do
Sample mini-batch B {-i, Vi // + R(D train: .)
12: Calculate LB = E[[ (3:, (x;; θ)) via Eq
13: Update ((++1) +- ( a e LB
14: end for
Experiment
4.1 Results on synthetic dataset
We make an ideal binary classification using Support Vector Machine (SVM) 14] to show the distinguish effectiveness of UniMix. Suppose there samples from two disjoint circles respectively:
= = [xx vx((< (I- mo)² 2 + (y - y0)² < r²) x = (1,,0)((x++ + xo)² + (y + yo)² < 1²]
(16)
To this end, randomly sample discrete point pairs from -+ to compose positive samples
sp {( v.), )..................). (o vth),, and m negative samples x, = {(x 11 ",.,.... (o vm)) from = correspondingly, thus to generate a balanced dataset Drot = { 2, with P ((x,v) E =,
P ((x, y) E 2n 0.5. For imbalance data, sample n(n m) negative data from to generate
A = [(11 (/
(24, ") }}. so as to compose the imbalance dataset Dimbat = ( ,, with P ((z, y) E x ((x,y) y) E ). We train the SVM model on the synthetic datasets,
and visualize the classification boundary of each dataset in Fig
(a) imbalanced scenario
with mixup
(c) with UniMix
(d) balanced scenario
Figure 3: SVM decision boundary on the synthetic balanced dataset (Fig B(d) and imbalanced dataset (Fig kb) B(c).. The theoretical classification boundary of the synthetic dataset is y=-z. "o"
represents generated pseudo data, where blue and green represent belong to = and = respectively.
The SVM reaches an approximate ideal boundary on balanced datasets (Fig d but severely deviates from the y = - v the imbalanced dataset (Fig 3(a)p. As proven in See mixup (Fig 3(b) is incapable of shifting imbalance distribution, resulting in no better result than the original one (Fig ((a).. After adopting the proposed UniMix, the classification boundary Fig 3(c] better results than the original imbalanced dataset, which gets closed to the ideal boundary.
] shows much

---

4.2 Results on CIFAR-LT
The imbalanced datasets CIFAR-IO-LT and CIFAR-100-LT are constructed via suitably discarding training samples following previous works 64 decay per class in train dataset and keep balanced during inference. We extensively adopt P E {10, 50, 100, 200} for comprchensive comparisons. See implementation details in Appendix C.I
12
+ 5
The instance numbers exponentially
Comparison methods. We evaluate the proposed method against various representative and effective approaches extensively. summarized into the following groups: a) Baseline. We conduct plain training with CE loss called ERM as baseline. b) Feature improvement methods modify the input feature to cope with LT datasets. mixup [600 convexly combines images and labels to build virtual data for VRM. Manifold mixup [55] performs the linear combination latent states, Remix ITO conducts the same combination
images and adopts tail-favored rules on labels. M2m [ converts majority images minority ones by adding noise perturbation, which need an additional pre-trained classifier. BBN 64 utilizes features from two branches in cumulative learning manner. e) Loss modification methods either adjust the logits weight or margin before the Sofimax operation. Specifically, focal loss [38]
and CDT [58] re- re-weight the logits with claborate strategies, while LDAM [6] and Logit Adjustment [44 add the logits margin to shift decision boundary away from tail classes. d) Other methods. We also compare the proposed method with other two-stage approaches (e.g. DRW [6] for comprehensive comparisons.
CB 12
Table 1:
Top-1 validation accuracy(%6 (%) of ResNet-32 CIFAR-10/100-D E2E: end to end training. Underscore: the best performance in each group. t: our reproduced results. i: reported results in reported results in 588. Our calibration ensured method achieves the best performance.
64
Dataset
E2E
CIFAR-IO-LT
CIFAR-100-LE
p (easy
hard)
10
200
10
50
100
200
ERM
86.39
74.94
70.36
66.21
55.70
44.02
38.32
34.56
mixupt
87.10
77.82
73.06
67.73
58.02
44.99
39.54
34.97
Manifold Mixup
65
87.03
77.95
72.96
56.55
43.09
38.25
Remix 10
88.15
79.20
75.36
67.08
59.36
46.21
41.94
36.99
M2m 33
x
87.90
78.30
58.20
42,90
BBN
x
88.32
82.18
79.82
59.12
47.02
42.56
Focal
38
86.55
76.71
70.43
65.85
55.78
44.32'
38.41
35.62
Urtasun al 477
82.12
76.45
72.23
66.25
52.12
43.17
38.90
33.00
CB-Focal
12
87.10
79.22
74.57
68.15
57.99
45.21
39.60
36.23
7-norm"
30
87.80
82.78F
75.10
70.30
59.10
48.23
43.60
39.30
LDAM
86.96
79.84
74.47
69.50
56.91
46.16
41.76
37.73
LDAM+DRW'
X
88.16
81.27
77.03
74.74
58.71
47.97
42.04
38.45
CDT [58
89.40
81.97
79.40
74.70
58.90
45.15
44.30
40,50
Logit Adjustment
44
89.26
83.38
79.91
75.131
59.871
49.76
43.89
40.871
Ours
89.66
84.32
82.75
78.48
61.25
51.11
45.45
42.07
Results. We present results of CIFAR-10-L1 and CIFAR-100-LT in Tab I Our proposed method achieves state-of-the-an art results against others on each with performance gains improved as gets increased (See Appendix D.I. Specifically, effectively with better calibration, which integrates advantages of two group approaches and thus surpass most two-stage methods (i.e., BBN, M2m, LDAM+DRW). However, not all combinations can get ideal performance gains as expected. More details will be discussed in Sec
method overcomes the ignorance in tail classes
To quantitatively describe the contribution to model calibration, make quantitative comparisons on CIFAR- 10-LT and CIFAR-I00-LT, According to the definition ECE and MCE (see Appendix well-calibrated model should minimize the ECE and MCE for better generalization and robustmess. In this experiment, we adopt the most representative p E (10, 100) with previous mainstream state-of- tht-thh-at methods.
Eq D.2 D.3
The results Tab show that either of the proposed methods generally outperforms previous methods, and their combination enables better classification calibration with smaller ECE and MCE, Specifically, mixup and Remix have negligible contributions to model calibration. As analyzed before, such methods tend to generate head-heal pairs in favor of the feature learning of majority
7

---

Table 2: Quantitative calibration metrie of ResNet-32 CIFAR- test Smaller ECE and MCE indicate better calibration results. Either of the proposed methods achieves a well-calibrated model compared with others. The combination of UniMix and Bayias achieves the best performance.
Dataset
CIFAR-1O-LTT
CIFAR- 100-1-
10
100
10
100
Calibration Metric (%)
ECE
MCE
ECE
MCE
ECE
MCE
ECE
MCE
ERM
6.60
74,96
20.53
73.91
22.85
34.50
38.23
87.22
mixup 60
6.55
24.54
19.20
37.84
19.69
38.53
32.72
50.46
Remix 10
6.81
22.44
15.38
27.99
20.17
32.99
33.56
50.96
LDAM+DRW
11.22
45.92
19.89
49.07
30.54
55.57
42.18
64.78
UniMix (ours)
6.00
25.99
12.87
28.30
19.38
33,40
27.12
41,46
Bayias (ours)
5.52
20.14
11.05
23.72
17.42
28.26
24.31
39.66
UniMix-Bayias (ours)
4.74
13.67
10.19
25.47
15.24
23.67
23.04
37.36
classes. However, more head-tail pairs are required for better feature representation of the tail classes. In contrast, both the proposed UniMix and Bayias pay more attention the tail and achieve satisfactory results. It is worth mentioning that improving calibration in post-hoc manners [16 63 is also effective, and we will discuss Appendix D.2.3] Note that LDAM even worse calibrated compared with baseline. We suggest that L.DAM adopts an additional margin only for the ground- truth label from the angular perspective. which shifts the decision boundary away from the tail class and makes the tail predicting score tend to be larger. Additionally, LDAM requires the normalization of input features and classifier weight matrix. Although a scale factor is proposed to enlarge the logits for better Softmax operation [56]. it is still harmful to calibration. It also accounts for its contradiction with other methods. Miscalibration methods combined will make models become even more overconfident and damage the generalization and robustness severely.
4.3 Results on large-scale datasets
We further verify the proposed method's effectiveness quantitatively large-scale imbalanced datasets, iee. ImageNet-LT and iNaturalist 2018. ImageNet-LT is the LT version of ImageNet 44 by sampling subset following Pareto distribution, which contains about 115K images from 1, 000 classes. The number of images per class varies from 5 to 1, 280 exponentially. i.e., p 256. In our experiment, utilize the balanced validation constructed by Cui et al. [12 for fair comparisons. The iNaturalist species classification dataset 255 is large-scale real-world dataset which suffers from extremely label LT distribution and fine-grained problems 25 66 It is composed of 435, 713
images over 8, 142 classes with p = 500. The official splits of train and validation images 6 are adopted for fair comparisons. See implementation details in Appendix C.2
30
Table 3: Top-1 validation accuracy(%) of 1esNeet-10/500 ImageNet-L.T and ResNet-50 iNaturalist 2018. E2E: end to end training. t: our reproduced results. J: results reported in origin paper.
Dataset
ImageNet-LT
iNaturalist 2018
Method
E2E
ResNet-10
ResNet-50
A
ResNet-50
A
CE
35.88
38.88
60.88
CB-CE
12
/
37.06
+1.18
40.85
+1.97
63.50
+2.62
LDAM
/
36.05
+0.17
41.861
+2.98
64.58t
+3.70
OLTR
X 35.60
-0.28
40.36
+1.48
63.90
+3.02
LDAM+DRW
X
38,22
+2.34
45.75
+6.87
68.00
+7.12
BBN
64
X 41.80
66.29
+5.41
RTT
30
X
+5.92
47.541
+8.66
67.601
+6.72
Ours
42.90
+7.02
48.41
+9.53
69.15
+8.27
Results. Tab illustrates the results on large-scale datasets. Ours consistently effective and out- performs existing mainstream methods, achieving distinguish improvement compared with previous SOTA C-RT (30 in compared backbones. Especially, our method outperforms the baseline on ImageNet-LT and iNaturalist 2018 by 9.53% and 8.27 % with ResNet-50, respectively. As can be
noticed in Tab the proposed method also surpasses the well-known two-stage methods achieving superior accuracy with less computation load in concise training manner.

---

4.4 Further Analysis
Effectiveness of UniMix and Bayias. We conduct extensive ablation studies in Tab to demonstrate the effectiveness of the proposed UnixMix and Bayias, with detailed analysis in various combinations of feature-wise and loss-wise methods on CIFAR-1O-LI and CIFAR- -O0-TT. Indeed. both UniMix and Bayias turn out to be effective in LT scenarios. Further observation shows that with calibration ensured, the proposed method gets significant performance gains and achieve state-of-the-ard results. Noteworthy, LDAM [6] makes classifiers miscalibrated, which leads to unsatisfactory improvement when combined with mixup manners.
Table Ablation study between feature-wist and loss- wis- methods. LDAM is counterproductive to mixup and its extensions. Bayias-compensateed CE ensures calibration and shows excellent performance gains especially combined with UniMix.
Dataset
CIFAR- I0-IT
CIFAR-100-LIT
(easy + hard)
100
200
100
200
Mix
Loss
Top1 Ace
A
Top1 Acc
A
Top1 Acc
Top1 Ace
A
None
70.36
66.21
38.32
34.56
mixup
CE
73.06
+2.70
67.73
+1.52
39.54
+1.22
34.97
+0.41
Remix
CE
75.36
+5.00
67.08
+0.87
41.94
+3.62
36.99
+2.43
UniMix
CE
76.47
+6.11
68.42
+2.21
41.46
+3.14
37.63
+3.07
None
LDAM
74.47
69.50
41.76
37.73
mixup
LDAM
73,96
-0.15
67.89
-1.61
40.22
-1.54
37.52
-0,21
Remix
LDAM
74.33
-0.14
69.66
+0.16
40.59
-1.17
37.66
-0.07
UniMix
LDAM
75.35
+0.88
70.77
+1.27
41.67
-0.09
37.83
+0.01
None
Bayias
78.70
74.21
43.52
38.83
mixup
Bayias
81.75
+3.05
76.69
+2.48
44.56
+1.04
41.19
42.36
Remix
Bayias
81.55
+2.85
75.81
+1.60
45.01
+1.49
41.44
42.61
UniMix
Bayias
82.75
+4.05
78.48
+4.27
45.45
+1.93
42.07
+3.24
Evaluating different UniMix Sampler. Corollary 2 demonstrate distinguish influence of UniMix. However, the E-sample can not be completely equivalent with orginal ones. Hence, an appropriate Eq D is also worth further searching. Fig illustrates the accuracy with different r on CIFAR- and CIFAR-100-LT setting 10 and To0. For CIFAR-IO-LT (Fig 4(a) 4b)).
=1 possibly ideal, which forces more head-tail instead of head-hear pairs get generated to compensate tail classes. In the more challenging CIFAR- CIFAR-100-LT, t = O achieves the best result. We suspect that unlike simple datasets (e.g.. CIFAR- 10-LT), where overconfidence occurs head classes, all classes need to get enhanced in complicated LT scenarios. Hence, the augmentation is effective and necessary both on head and tail. 7 0 allows both head and tail get improved simultaneously.
002
160
400
400
(a) CIFAR-1O-LTT-I
(b) CIFAR-I0--0 10-I.T-100.
(c) CIFAR 100-1--10.
(d) 1FFAR-100LLT-10M
Figure 4: Comparison of top-1 validation accuracy(% %) of ResNet-32 on CIFAR-LT when varying in Eq g for UniMix. The histogram indicates average results in repeated experiments.
Do minorities really get improved? To observe the amelioration on tail classes, Fig visualizes log-confusion matrices on LIFAR-100-LT- 100. In Fig 5(c) our method exhibits satisfactory general- ization on the tail. Vanilla ERM model (Fig 5(a) is trivial predictor which simplifies tail instances as head labels to minimize the error rate. Feature improvement [10] and loss modification (6 58 methods do alleviate LT problem to some extent. The misclassification cases (i.e., diagonal elements) in Fig (())5(()))) (dd) become smaller and more balanced distributed compared with ERM. However, the error cases are still mainly in the upper lower triangular, indicating the existence of inherent bias between the head and tail. Our method ((ig)(ee) significantly alleviates such dilemma.
The non-diagonal elements are more uniformly distributed throughout the matrix rather than in

---

66
99 33
66
90
33
66
95
33
95
33
66
95
06
95
199
W
W
(a) ERM
(b) LDAM+DRW
(c) CDT
58
(d) Remix 10
Ours
Figure 5: The log-confusion matrix on CIFAR- 1---R-O0--T---- 100 validation dataset. The r-axis and y-axis indicate the ground truth and predicted labels, respectively. Deeper color indicates larger values.
the corners, showing superiority to erase the bias in LT scenarios. Our method enables effective feature improvement for data-scarce classes and alleviates the prior bias, suggesting our success in regularizing tail remarkably.
5 Related work and discussion
Why need calibration? To quantify the predictive uncertainty, calibration is put forward to describe the relevance between predictive score and actual correctness likelihood. A well-calibrated model is more reliable with better interpretability, which probabilities indicate optimal expected costs Bayesian decision scenarios 43 Guo er al. [16] firstly provide metric to measure the calibration of CNN and figure out well relfoperfrreed models are always in lack of calibration, indicating that CNN sensitive to be overconfidence and lacks robustness. Thulasidasan el [54] point out that the effectiveness of mixup in balanced datasets originates from superior calibration modification. Menon 44 further show how to ensure optimal classification calibration for pair- loss.
et al.
Feature-wise methods. Intuitively, under-sampling the head 18 4 20 or over-sampling the tail 9
5 can improve the inconsistent performance of imbalanced datasets but tend to either weaken the head or over-fitting the tail. Hence, many effective works generate additional samples to compensate the tail classes. BBN
45
50
4 uses two branches extract features from head and tail simultaneously. while c-RT 30 trains feature representation learning and classification stage
|
22
10 are effective and easy-implement feature-wise methods that convexly combine input and label pairs to generate virtual samples. However, naïve mixup manners are deficient in LT scenarios as we discussed in Sec In contrast, our UniMix tackles such a dilemma by constructing class balance-oriented virtual data as describe in Sec G.T. and shows satisfactory calibration as Fig exhibits.
separately. mixup [60] and its variants 555
59
57 have demon- strated the existence of inherent bias between LT train set and balanced test set supervised learning. make networks prefer learning tail samples by additional class- related weight on CE loss. Some works further correct CE according to the gradient generated by or from the perspective of Gaussian distribution and Bayesian estimation
Loss modification. Numerous experimental and theoretical studies 46
Previous works [27] 28
31
12,
different samples 52
optimize the weights of each class in CE as learnable parameters and achieve remarkable success. Cao et al. G theoretically provides the VC generalization bound. Compared with Logit Adjustment [44] motivated by balance error rate, our Bayias-compensated CE eliminates bias incured by prior and consistent with balanced datasets, which ensures classification calibration as well.
19
3
Meta-learning approaches [I5I 55 17 29
ideal optimal margin for CE from the perspective
Conclusion
We systematically analyze the limitations of mainstream feature improvement methods, i.c., mixup and its extensions the label-imbalanced situation, and propose the UniMix construct more class- balanced virtual dataset that significantly improves classification calibration. We further pinpoint an inherent bias induced by the inconstancy of label distribution prior between long-tailed train set and balanced test set. We prove that the standard cross- cross-entropy loss with the proposed Bayias compensated can ensure classification calibration. The combination of UniMix and Bayias achieves state-of-the-an performance and contributes to better-calibrated model (Fig TT. Further study in Tab shows that the bad calibration methods are counterproductive with each other. However, more in-depth analysis and theoretical guarantees still required, which we leave for our future work.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NSFC project Grant No. U1833101, SZSTI Grant No. JCYJ20190887 172201639 and WDZC202008200206550000 the Joint Research Center of Tencent and Tsinghua.
References
[1] Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, pages 3981-3989, 2016.
[2] Peter L Bartlett Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 2006.
[3] Mariusz: Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Fimerr, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort. Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End end learning for self-driving cars. CaRR, abs/1604.07316, 2016.
[4] Mateusz Buda. Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
[5] Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 872 881. PMLR, 2019.
[6] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with labelldistributtionaaaar margin loss. In Advances in Neural Information Processing Systems NeurlPS 2019, pages 1565-1576, 2019.
[7] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2/th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721-1730. ACM, 2015.
[8] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 416-422. MIT Press, 2000.
[9] Nitesh Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002.
[10] Hsin-Ping Chou, Shih-Chich Chang, Ju Pan, Wei Wei, and Da- Da-Cheng Juan. Remix: Rebalanced mixup. In Computer Vision ECCV 2020 Workshops, volume 12540, pages 95-110. Springer, 2020.
[11] Peng Chu, Xiao Bian, Shaopeng I. Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In Computer Vision ECCV 2020 16th European Conference, volume 12374 Lecture Notes in Computer Science, pages 694-710. Springer, 2020.
[12] Yin Cui, Menglin Jia, Tsung-Yi Lin. Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 9268-9277. Computer Vision Foundation / IEEE, 2019.
[13] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 1190-4699) Computer Vision Foundation IEEE, 2019.
[14] Theodoros Evgeniou and Massimiliano Pontil. Support vector machines: Theory and applications. In Machine Learning and Its Applications, Advanced Lectures, volume 2049 of Lecture Notes in Computer Science, pages 249-257. Springer, 2001.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, 2017, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 2017.
[16] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321 -1330. PMLR, 2017.
11

---

[17] Agrim Gupla. Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmenta- tion. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, pages 5356-5364.
Computer Vision Foundation IEEE, 2019.
[18] Hui Han. Wenyuan Wang. and Binghuan Mao. Borderline-smoter A new over-sampling method in imbalanced data sets learning. In Advances in Intelligent Computing, International Conference on Intelligent Computing, ICIC 2005, volume 3644 of Lecture Notes Computer Science, pages 878-887. Springer, 2005.
[19] Munawar Hayat, Salman H. Khan, Syed Waqas 7amir, Jianbing Shen, and Ling Shao. Gaussian affinity for max-margin class imbalanced learning. In 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, pages 6468-6478 6478. IEEE, 2019.
[20] Haibo He and Edwardo A. Garcia. Learning from imhalanced data. IEEE Trans. Knowl. Data Eng-, (199):1263-128.. 2009.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, pages 2980 2988. IEEE Computer Society, 2017.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 770-778, LEEE Computer Society, 2016.
[23] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In IEEE Conference on Computer Vision and Patten Recognition, CVPR 2019, pages 358-567. Computer Vision Foundation / IEEE, 2019.
[24] Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. CoRR, wbs/1909.09111111 09100,994,,,00 2019.
[25] Grant Van Horn, Oisin Mac Andha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 8769-8778. IEEE Computer Society, 2018
[26] Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. CoRR, Atss/709.004550 2017.
[27] Chen Huang. Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, pages 5375-5384. IEEE Computer Society, 2016.
[28] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans. Puttern Anal. Mach. Intell., 42(1 1):2711-2794, 2020.
[29] Muhammad Abdullah Jamal, Matthew Brown, Ming- Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, pages 7607- 7616. IEEE, 2020.
Ming-Hsuan
[30] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
[31] Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous Ahmed Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. JEEE Trans. Neural Networks Learn. Syst. (9(8)33573-358) 2018
[32] Salman H. Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and I Ling Shao. Striking the right balance with uncertainty. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 103-112. Computer Vision Foundation TERR, 2019.
[33] Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced elassification via major-to-minor translation. In 2020 IREE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, pages 13902. IEEE, 2020.
[34] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32-73 2017.
12

---

[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[36] Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In The Thirty-Thire AAAI Conference on Artificial Intelligence, AAA! 2019, pages 8577 8584. AAAI Press, 2019.
[37] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang. Chunfeng Wang. Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10991-1 1000, 2020.
[38] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, ICCV 2017, pages 2999-3007. IEEE Computer Society, 2017.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, volume 8693, pages 740-755. Springer, 2014.
[40] Hao Liu, Xiangyu Zhu, Zhen Lei, and Stan Z. Li. Adaptiveface: Adaptive margin and sampling for face recognition. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, pages 11947-11956. Computer Vision Foundation / IEEE, 2019.
[41] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 6738-6746. IEEE Computer Society, 2017.
[42] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiyyun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition an open world. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 20, 2019, pages 2537-2546. Computer Vision Foundation / IEEE, 2019.
[43] Juan Maroñas, Roberto Paredes, and Daniel Ramos. Calibration of deep probabilistic models with decoupled hayesian neural networks. Neurocomputing, 407:194-205, 2020.
[44] Aditya Krishna Menon. Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
[45] Sankha Subhra Mullick, Shounak Datta, and Swagatam Das. Generative adversarial minority oversampling. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 1695-1704. IEEE, 2019.
[46] Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, 2020.
[47] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proceedings of the 35th International Conference on Machine Learning, (CM1.2012 volume 80 of Proceedings of Machine Learning Research, pages 4331 4331-4340. PMLR, 2018.
[48] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R C-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pages 91 1999, 2015.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Feii Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):21 1-222, 2015.
[50] Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. Computer Vision ECCV 2016 14th European Conference, volume 9911, pages 467-482. Springer, 2016.
[51] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zonghen Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pages 1917-1928, 2019.
[52] Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, pages
IEEE, 2020.
13

---

[53] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Prucessing Systems 2020, NeurIPS 2020, 2020.
[54] Sunil Thulasidasan, Gopinath Chennupati. Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pages 1 13888-13899, 2019.
[55] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paze and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, volume 97, pages 6438 6447.
PMI.R, 2019.
[56] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017. pages 1041-1049, ACM, 2017.
[57] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, 2020.
[58] Han-Jia Ye, Hong-Your You Chen, De-Chuan Zhan, and Wei-Lan Chao. Identifying and compensating for feature deviation in imbalanced deep learning. CoRR, 0hs/2001-013333 2020.
[59] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Scong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 6022-6031. IEGE, 2019.
[60] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference Learning Representations, ICLR 2018. OpenReview.net, 2018.
[61] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021.
[62] Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In AAAI, 2021.
[63] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, pages 16489-16498. Computer Vision Foundation JEEE, 2021.
[64] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, pages 97 9716-9725. IEBE, 2020.
14


Reviews:
[{"review_id": "dJXHPJC8AZ", "reviewer": "Reviewer_1uZE", "summary": "The authors propose two models to improve calibration for long-tailed recognition. 1) a Uniform Mixup is proposed, which adopts an advanced mixing factor and sampler in favor of the minority. 2) the authors propose a Bayias loss to compensate for the prior bias. The proposed method achieves good results on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this paper is limited:\n\nThis work is very similar to previous papers [1] and [2]. Concretely, the proposed Bayias loss (Eq.15) is the same as logit adjustment [1] (Eq. 11 in [1]) for long-tailed recognition. [2] analyze long-tailed recognition from the calibration view. Algorithm 1 in this paper is also the same as [2], i.e., adding mixup in Stage-1 and using re-balanced loss in Stage-2. What's more, the whole paper emphasizes calibration. However, the authors just give visualization results (Fig. 1, Fig. 3, and Fig. 5) to illustrate.  Why not show quantitative results (Expected Calibration Error, Brier Score, and Probabilistic sharpness) in experiment part? A lot of theoretical analysis seems meaningless.\n\n\nThe experiment results are not significant:\n\nBalanced Softmax [3] (experiment results please refer to [4]) can get **45.1%** top-1 accuracy on CIAFR-100-LT, $\\rho=100$. While the proposed Bayias loss just gets **43.52%** (**both without mixup**).  Balanced Sofemax [3] (experiment results please refer to [4]) **without mixup** can get **69.8%** top-1 accuracy on iNaturalist 2018. While the proposed Bayias loss **with mixup** just gets **69.15%**. \n\nFor calibration, the proposed method is **overall lower** than MiSLAS [2] in terms of **both Expected Calibration Error and top-1 recognition accuracy** on all long-tailed benchmarks. It is a reasonable and related fair comparison because both two models add mixup operation.\n\n\nOther questions:\n\n1) Based on Algorithm 1, I don't think the proposed method is E2E. Since it also contains two stages, which is similar to LDAM+DRW (Table 1: LDAM+DRW E2E  [x])\n\n2) Both UniMix Factor and UniMix Sampler generate samples, which is mainly a $\\xi$-Aug sample of the tail composite with $x_i$ from the head. I don't think more balanced $\\xi$-Aug samples will lead to better model recognition performance.\n\n3) What is the definition of $f$ in Eq. (6)?\n\n\nRef:\n\n[1] Long-tail Learning via Logit Adjustment, ICLR 2021.\n\n[2] Improving Calibration for Long-Tailed Recognition, CVPR 2021.\n\n[3] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeuIPS 2020.\n\n[4] Disentangling Label Distribution for Long-tailed Visual Recognition, CVPR 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper aims to address long-tailed recognition. There seems no obvious negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "Tl5_a944s1h", "reviewer": "Reviewer_V6xE", "summary": "This paper studies a practical problem, namely long-tailed recognition. To address this problem, this paper proposes two methods: UniMix and compensated cross-entropy loss. To be specific, UniMix adopts a balanced data sampler to improve the number of head-tail pairs for mixup. Experimental results show the effectiveness of the proposed method.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper reveals an interesting problem in existing mixup-based long-tailed recognition (LT) methods, i.e., they will generate more head-head pairs since there exist more samples from the head classes. Such a problem may lead to more imbalanced learning.\n2. To address the above problem in existing mixup-based LT methods, this paper proposes an intuitive UniMix strategy.\n3. The proposed methods are theoretically analyzed.\n4. Extensive results show the effectiveness of the proposed method.\n\nCons:\n1. When this paper analyzes the bias issue of mixup, it assumes the instance number is exponential with $\\lambda$. I wonder whether the result is the same when the instance number is not exponential with $\\lambda$, since real-world datasets may not follow such a way.\n2. For the proposed compensated cross-entropy loss, it is very similar to the balance softmax [1], which is also obtained from the Bayesian inference's perspective. Moreover, similar ideas of logit adjustment and rebalanced mixup have been deeply explored in the work [2,3]. Although this is not the drawback of this paper, I do not think this loss can be regarded as an important contribution.\n\n    [1] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.\n\n    [2] Long-tail Learning Via Logit Adjustment, ICLR 2021.\n\n    [3] Remix: Rebalanced Mixup. arXiv, 2020.\n\n3. In the final algorithm, there are actually two phases, i.e., first mixup training for T1 steps, and then non-mixup training for T2-T1 steps. I understand this is a trick to improve performance [4], but I wonder how effective the first phase is, since the main contribution of this paper is at phase 1, i.e., UniMix.\n\n    [4] Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks, AAAI 2021.\n\n4. As proposed in this paper, mixup leads to bias distribution (since generating more head-head pairs) and has a negligible contribution to model calibration. Regarding this, why do existing mixup-based methods also obtain very promising performance?  Does this mean that model calibration may not be an essential metric for the long-tailed recognition problem? \n\n5. Following question 4, according to Tables 3 and D1, LDAM is harmful to model calibration. However, LDAM is also effective and helps handle the imbalance issue. Does this mean that a method with worse model calibration does not essentially mean a bad long-tailed method, and a method with better model calibration does also not mean a better long-tailed method? These two concerns are important since this paper is based on model calibration.\n\n6. Why not compare to MiSLAS [5]. This paper analyzes mixup for model calibration and thus is very related to the submission. One interesting thing is that from the results reported in two papers, although MiSLAS only uses Mixup and ignores the head-head pair bias, it obtain even better accuracy and lower ECE than the method in this submission. I understand there may exist some differences in methodology (1-stage method v.s. 2-stage method), but it would be better for authors to also discuss MiSLAS and compared with it.\nIn addition, one interesting thing is that MiSLAS finds that mixup is beneficial to representation learning and bad for classifier learning. From this perspective, the head-head bias seems to mainly influence classifier learning. Could you discuss this？\n \n    [5] Improving Calibration for Long-Tailed Recognition, CVPR 2021.\n\n***\n\n****Post Rebuttal****\n\nThe most important contribution of this paper is that it theoretically and empirically reveals an interesting problem in existing mixup-based long-tailed recognition (LT) methods, i.e., they will generate more head-head pairs that may lead to more imbalanced learning.\n\nTo address the argued problem, this paper proposes a new method, consisting of (1) a Unimix strategy for generating more tail pairs and (2) a Bayias loss for rebalancing the optimization. Nevertheless, the performance improvement of Unimix over mixup is limited, while most performance improvement is derived from the Bayias loss (c.f. Table 3 in the submission and the new results in the response). Moreover, the novelty of the Bayias loss is limited, considering the logits adjustment loss [1] and balanced softmax loss [2]. For the balanced test set, these losses perform similarly. Although the authors argue that the extension of Bayias loss with $\\log(C)$ to $\\log(\\pi')$ can help to handle imbalanced test class distribution when the test class distribution $\\pi'$ is known [3], I think that it is very difficult to obtain the test class distribution in real-world applications [4], so the practicability and contribution of the extension are limited.\n\nConsidering the above advantages and limitations, the paper seems really a borderline paper and is difficult to decide. Therefore, I return to consider the value and contribution of this paper to the LT community. In my view, although the performance and novelty of the proposed method are not that promising, this paper theoretically reveals an important head-head bias problem in the existing mixup-based LT method, which is insightful to the LT community and even inspires more augmentation-based LT methods. **Considering that performance-only judgment should not be encouraged in the current AI community and novelty can be varied, I believe the contribution of this paper is good enough and decide to increase my score to 7**. Even so, I encourage the authors to further improve the proposed method for better performance, keep exploring augmentation-based LT methods, and try to solve more practical test-agnostic LT task in the future. Thanks again for your effort.\n\n[1] Long-tail Learning Via Logit Adjustment, ICLR 2021. \n\n[2] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.\n\n[3] Disentangling Label Distribution for Long-tailed Visual Recognition. CVPR, 2021.\n\n[4] Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts with Self-Supervision. ArXiv, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the main review for the concerns of this paper. I would like to see the response from the authors. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10+"}, "extra_text": {}}, {"review_id": "LPMY_GOyLrS", "reviewer": "Reviewer_GN8J", "summary": "The authors proposed a new method based on two approaches, UniMix and Bayias, to sample input-output pairs from unbalanced datasets and use them in Mixup. The authors report experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, showing superior performance in terms of accuracy and calibration scores over a variety of baselines.\n\nWhile I could get the main idea behind the proposed method, which seems appealing, I had problems understanding all the details, mainly because of a substantial lack of clarity. Sentences are often disconnected which makes it difficult to understand what the authors mean.  My score is borderline but I am open to increase it if the authors can provide a solid rebuttal and substantially improve the paper in terms of clarity.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n---------\n\n- The method seems to follow a different rebalancing approach compared to other work (e.g. Remix [1]).\n- The empirical results show a substantial improvement in terms of accuracy over other baselines.\n- Comparison in terms of ECE scores confirm that the method improves calibration.\n\nWeaknesses\n----------\n\n1) The paper lacks in terms of clarity and it is often difficult to understand what the authors mean. The paper needs to be substantially improved from this point of view.\n\n2) One of the main claims of the authors is that the proposed method improves the model calibration w.r.t. Mixup on unbalanced datasets. The plots reported by the authors seem to suggest that this is true, but those plots show the accuracy VS confidence tradeoff, that can be misleading. In Appendix D.2 the authors report quantitative measurements in terms of ECE and MCE. It has been showed that ECE scores suffer of several shortcomings [3]. I think that it would be useful to provide a quantitative comparison also in terms of other metrics like ACE and TACE scores [2], which are more robust than ECE.\n\n3) It is unclear why best performances and calibration are obtained by joining UniMix+Bayias. The authors provided an ablation study in Table 3 but there is not a proper discussion to explain these results.\n\n4) The authros claim (lines 42-48) that one of the main contributions is the introduction of the $\\\\xi\\\\text{-Aug}$ notation for Mixup (Section 2). I am struggling to understand where is the novelty, since it seems to me that this is a fairly common notation used to describe Mixup. Could the authors expand on this point?\n\n5) All the experimental results do not report the standard deviation over accuracy. Are the reported results just on a single seed? Please provide additional details regarding this point.\n\n6) There is not a proper discussion of the shortcomings of this method. It seems to me that joining UniMix+Bayias has a higher computational overhead compared to other approaches (e.g. Remix [1]). I would like to see a discussion about this point.\n\n\nReferences\n----------\n\n[1] Chou, H. P., Chang, S. C., Pan, J. Y., Wei, W., & Juan, D. C. (2020, August). Remix: Rebalanced mixup. In European Conference on Computer Vision (pp. 95-110). Springer, Cham.\n\n[2] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring Calibration in Deep Learning. In CVPR Workshops (Vol. 2, No. 7).\n\n[3] Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not reported. This work has minimal societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "89zxq3rbKA", "reviewer": "Reviewer_Xiu3", "summary": "This paper addressed the problem of long tail recognition using a novel mixup technique. The proposed mixup technique (Unimix) considered the class prior and helps to create a more balanced virtual dataset. The author provide theoretical analysis on why Unimix is a better mixup technique. Extensive experiment also demonstrates the performance gain of using Unimix.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \na.\tThe paper is well written and all the figures are well illustrated.\nb.\tThe theoretical analysis is interesting and provide strong support to the proposed method.\nc.\tThe experiment is thorough and cover different aspects of the ablation studies.\nd.\tThe performance is promising\n\nMinor weakness\na.\tAlthough the author somehow shows the performance on head/medium/tail classes using the confusion matrix, it will be nice to provide the number and table in the final version. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of long-tailed visual recognition. They proposed two complementary methods to address the imbalance problem: (i) a data augmentation scheme based on the Mixup and (ii) an approach to compensate for bias in class prior. The final proposed pipeline is 1 stage, as opposed to 2-stage which has featured prominently in recent work. Results are presented across several different image classification datasets, showing superior performance compared to existing work in most cases. 

During the review process there was a lot of discussion regarding the relative performance compared to the recent CVPR’21 paper, “Improving Calibration for Long-Tailed Recognition” AKA MiSLAS. This paper was published after the NeurIPS submission deadline, and thus the relationship between the two works did not factor into the final decision. However, the authors are strongly encouraged to cite it, and include the new 2-stage comparison to MiSLAS from the response for completeness. In addition, they should more clearly clarify the relationship to the logit adjustment work from ICLR’21. 

The reviewers noted that while the technical contributions were not overly significant, the main insight of the paper (that naive mixup introduces “head to head” class bias, i.e. a bias towards head majority pseudo data) is well characterized and will be of interest to the long-tail community. There were also concerns about the clarity of the exposition and some missing discussion. Many of these issues were cleared up in the discussion, but the authors are strongly encouraged to address these issues with the writing for the final version. This is very important as it will hopefully significantly improve the clarity of the paper. The paper should also be proof-read for grammar (e.g. Line 129). 

In the end, three of the reviewers were broadly supportive of the paper, but one recommended rejection (based mostly on the relationship to MiSLAS - see above). This AC, agrees with the consensus of the reviewers and supports the paper being accepted.  

Decision: Accept (Poster)