Enhancing Robustness to Class-Conditional Distribution Shift in Long-Tailed Recognition
Authors: Keliang Li
Hong Chang
Shiguang Shan
Xilin CHEN
Venue: Accepted by TMLR ()

Abstract: For long-tailed recognition problem, beyond imbalanced label distribution, unreliable empirical data distribution due to instance scarcity has recently emerged as a concern. It inevitably causes Class-Conditional Distribution (CCD) shift between training and test. Data augmentation and head-to-tail information transfer methods indirectly alleviate the problem by synthesizing novel examples but may remain biased. In this paper, we conduct a thorough study on the impact of CCD shift and propose Distributionally Robust Augmentation (DRA) to directly train models robust to the shift. DRA admits a novel generalization bound reflecting the benefit of distributional robustness to CCD shift for long-tailed recognition. Extensive experiments show DRA greatly improves existing re-balancing and data augmentation methods when cooperating with them. It also alleviates the recently discovered saddle-point issue, verifying its ability to achieve enhanced robustness.

Published in Transactions on Machine Learning Research (02/2024)
Enhancing Robustness to Class-Conditional Distribution Shift in Long- Tailed Recognition
LoileTailed
Keliang Li
seviang.i@vvipl.ict..o
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
Hong Chang
changhhong@kcc..cc.om
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
Shiguang Shan
sgshan@ict.ac.com
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
Xilin Chen
alchen@ictac..om
Institute of Compating Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
Reviewed on OpenReview: https: // openreview. net/ form? id= n2gAD8Fázk
Abstract
For long-tailed recognition problem, beyond imbalanced label distribution, unreliable pirical data distribution due to instance scarcity has recently emerged as concern. It inevitably causes Class-Coniitioal Distribution (CCD) shift between training and test. Data augmentation and head-to-tail information transfer methods indirectly alleviate the problem by synthesizing novel examples but may remain biased. In this paper, we conduct a thorough study on the impact of CCD shift and propose Distributionally Robust Augmenta- tion (DRA) to directly train models robust to the shift. DRA admits a novel generalization bound reflecting the benefit of distributional robustness to CCD shift for long-tailes tailed recog- nition. Extensive experiments show DRA greatly improves existing re-balancing and data augmentation methods when cooperating with them. It also alleviates the recently discov- ered saddle-point. issue, verifying its ability to achieve enhanced robustness.
1 Introduction
Visual recognition has recently achieved significant progress, driven by the development of deep neural networks (He et al. 2016) as well as manually balanced datasets (Russakovsky et al. 2015). However, real- world data usually exhibits long-tailed distribution over classes (Liu et al.
2019 Van Horn & Perona 2017
leading to undesired estimation bias and poor generalization (Zhou et al. 2020 Cao et al. 2019 Kang et al. 2019). Recently, considerable efforts have been made address imbalanced label distribution issue through
re- re-balancing strategy. Two-stage methods (Samuel & Chechik 2021 Cao al. 2019 Zhou al. 2020 and
logit adjustment (Menon et al., 2020 Ren et al. 2020) have shown significant performance improvement over intuitive direct re- re-balancing (Zhang et al. 2021b), like re-sampling and re-weighting (Kang et al. 2019
Zhou ct al. 2020), and bocome important components of recent solutions (Zhang et al. 2021a Zhong et al.
2021). The ralionality of re-balancing strategies (Cao et al. 2019: Park et al.) 20021 Ren el al. 2020) has been theoretically proved (Menon et al.) 2020). under the following assumption: the class distribution P(y) shifts from training test. (usually class-imbalanced training but class-balancee in test), while the Class-
Conditional Distribution (CCD) P(rly) keeps consistent, i.c. Praas(<ll) Prese(x)y) (Menon al. 2020:
Ren et al. 2020

---

Published in Transactions on Machine Learning Research (02/2024)
Nevertheless, the consistent CCD assumption hardly holds in real-world long-tailed scenarios, attributed to instance sparsity. For tail classes where the samples are extremely scarce, estimating Ptrain (aly) by
empirical CCD is unreliable at all and the shift between them cannot be ignored, as we demonstrate later in
this paper. To address instance scarcity, some works (Kim et al. 2020 Zhou et al. 2022 Liu et al. 2019)
ERM
DRW
1.0
1.0
Accuracy
2.6
92/84 24
0.6
8CC:91.33
3.4
0.4
PC softmax
CRT
3.0
Accuracy
2.6
0.6
56/87-9
3CC:77 76/777777733
5
6
8
2
3
6
Class index
Class index
Figure 1: Accuracy CIFARIO-LT (imbalance ratio: 100) with without removing CCD shift. All methods (as introduced in Section 3.2 show significant improvement after removing shifts, especially for classes with fewer instances, verifying that the empirical CCD distributions of tail classes are more unreliable. Shaded regions show 95% CTs over 5 runs.
explore transferring information from head to enhance tail classes, e.g., by pasting image patches or mixing features (Park et al. 2022 Hong et al. 2022). However, head-to-tail transfer relies on the assumption (Chu ee al. 2020 Kim et al. 2020) that (1) head class information can help to recover tail class distribution, and (2) class-generic and class-specific information can be decoupled and re-mixed to generate novel tail class samples. Actually, head-to-tail transfer cannot convincingly recover the real CCD. Even if class-generis information can be extracted perfectly, it may not be suitable the tail classes e.g. the class geas-ieeerric context PPar et al. 2022) shared by head classes, grassland, clearly contradicts the true distribution of tail-class animals that live in caves.
Other works (Xu et al. 2021 Wei et al. 2022 Zhong et al. 2021 Li et al. 2021 Ahn et al. 2023 dealing with instance scarcity employ data augmentation designed for balanced datasets to increase tail class diversity. However, there are concerns rogarding these methods. Their target is generating novel instances instead of alleviating CCD shift. directly, potentially leading to biased augmented examples. Moreover, existing data augmentation methods require specific modifications to adapt long-tailed distribution setting (Li et al.
2021 Xu ct al. 2021 Park ct al. (022). This complicates the joint use of data augmentation and ro-balancing strategies, e.g. Mixup-based ang.eentation (Zhang et al. 2017) degrades margin- based re- be-balancing methods
[Ren et al. 2020 Cao et al.] 2019) in most cases (Xu et al. 2021 Park et al. 2022). Therefore, exploring data augmentation methods that can directly address CCD shift and cooperate with existing strategies is promising.
To this end, we first conduct an empirical study to quantify the impact that CCD shift has on long-tail recognition. We verify that CCD shift is a critical factor limiting the performance of long-tail tail recognition, and highly correlated with the class cardinality, intuitive phenomenon in state-of-the-an re- re-balancing methods and highlight the importance of instance-level discriminative data augmentation and logit adjustment methods in addressing CCD shift. Based on them, we design a novel data augmentation method, Dislrlbutionall Robust. A Augmentation (DRA), that directly targets CCD shift and adapts to the current instance and training dynamics. Specifically, we train models be more robust to unrcliable empirical distributions rather than attempting
in Figure [ Moreover,
experiments explain the counter-
restore the underlying tail distribution, as expected in previous works (Hong al. 2022 Park et al. 2022). We achieve this goal through Distributionally Robust Optimization (DRO) (Sinha et. al. 2017 Lin el. al. 2022 with class-aware
2

---

Published in Transactions on Machine Learning Research (02/2024)
uncertainty set, which allocates robustness based on the unreliability of each class. We further prove that, under mild conditions, our training objective bounds the balanced loss on real CCDs.
Our main contributions are highlighted as follows:
We investigate CCD shift issue in long-tailed recognition quantitatively, and provide new insights from this view into main-stream re-balancing and data augmentation methods.
We propose a thcorctically sound method, Distributionally Robust Augmentation (DRA), to directly address instance scarcity problem and make models more robust to CCD shift.
Extensive experiments on long-tailed classification benchmarks demonstrate that DRA can cooperate with and enhance existing re- balancinng and data augmentation methods. DRA can also improve the confidence calibration (Guo al., 2017)) and alleviate saddle-point issuc (Rangwani et al. 2022a)
2 Related Works
2.1
Re- Re-balancing methods for long-tailed data
We review a broader scope of re-balancing methods that address the prior shift of P(y). Intuitive direct ro-sampling or ro-weighting by froquencics is shown getting marginal improvement (Zhang et al.] 2021bb
Cui al. 2019 Lin et al.) 2017) while two-stage methods, e.g. deferring use of re-balancing strategies
(e.g. DRS DRW (Cao et al.
2019
Zhou et al., 2020; Park et al. 2021i) and decoupling methods which CRT. LWS (Kang et al. 2019h overperform them significantly (Zhong
only re- ra-balance the classifier (e.g.
et al.
2021 Zhang et al.]
2021a).
Another line introduces class-wise margin (Cao et al.
2019 Wu et al.
2021 or temperature (Ye et al.
2020) to cross-entropy loss. Logit adjust.mett (Menon et al. 2020) utilizes
margin-based loss and post- hoc adjustment to modify logits, equivalent to PC softmax and Balanced softmax
respectively (Hong al. 2021 Ren al. 2020), which has boen proved consistent with balanced accuracy (Menon et al.] Though two-stage methods and logit adjustment have become important components
Though
of recent solutions (Samuel & Chechik 2021 Zhang et al.) 2021a Park et al. 2022 Hong et al. 2021 Zhou
et al. 2023), there still remain questions to them, e.g. why deferring re- balancing is critical and why the consistent adjustment is sub-optimal. A concurrent work (Wang et al. 2023) sheds light on these questions by a systematical Fisher consistency analysis for re-weighted margin-based loss(Cag et al. 2019 Kini et al. 2021) while Section will give explanations from the view of CCD shift. Additionally, re-balancing methods suffer from saddle points for loss of tail classes RRnggaani et al., (022a)), lcading to poor generalization and instability to distribution shift Chift(Dauphin et al., 2014). Rangwani et al. (2022a) and Zhou et al. (2023) use SAM technique(Fored et al., (020)) to improve re-balancing methods by allowing model to converge to flat minima. Surprisingly, our proposed DRA also alleviates the saddle-point issue effectively, indicating that DRA obtains more robust models.
2.2 Data augmentation and information transfer for long-tailed data
Data augmentation (DA) on balanced datasets (Cubuk et al. 2020 Yun et al,
2019 such as Mixup (Zhang
et al.
2017) and ISDA (Wang et al. (201)). have been adopted improve long-tailed learning 4Ahn et al.
2023
Xu al. 2021 Zhong et | |021: Li et al. 2021). Moreover, images Zada et al. 2022) features (Li
et al.
2022a perturbed by noise and out-of-distribution data (Wei et al. 2022 could surprisingly serve
augmentation(FA) for imbalanced data, though they are harmful to balanced training, Head-to-tail transfer
are highly explored recently (Yin et al.] 2019 Liu et al.) 2021 2020: Kim et al., 2020). GIT (Zhou et al. 2022) uses GAN [Goodeelow el al. 2014 to learn nuisance transformatiins to transfer invariance for tail
classes while CMO (Park et al. 2022 randomly pastes image patches head classes on tail classes to transfer context: information. Recently Ahn al. (2023) and Hong et al. (2022) claim that class-wise or instance- aware augmentation more effective, which is consistent with our findings in Section 3 The relation and difference between related data augmentation works and our proposed method will be discussed later.
3

---

Published in Transactions on Machine Learning Research (02/2024)
2.3 Ensemble methods for long-tailed data
Ensemble methods for long-tailed recognition are based on multiple experts to take full advantage of limited data. Zhou et al. (2020) gradually fuse two branches for balanced and re-balanced label distributions re- spectively by a cummlative learning strategy while Cai et al. (2021) tend to make experts cope with specific and overlapping class splits to complements for each other. RIDE/Wang et. al.] 2020) proposes strategy to route experts dynamically, and explicitly encourages the diversity of them. SADE et al. 2022) allows each expert to adapt to various label-distribution and aggregates them by the consistency of their predictions in the test time, and could generalize to various test label distributions. In this work, we see ensemble methods as orthogonal to our proposed DRA, as introducing distributional robustness to ensemble methods is complex and under-explored even for balanced datasets. Exploring how much ensemble methods adapt to CCD shift and enhancing them from CCD shift view are left for future works.
2.4
Distributionally Robust Optimization (DRO)
DRO (Kuhn et al. 2019 Delage & Ye. 2010 Shaffeezadeh- Abadeh et al. 2019) aims at generalization under distribution shift. It focuses on worst-case risk minimization across a set of potential distributions, known as uncertainty sets, determined based on various computations, c.g. Wassorstein distance (Kuhn et al. 2019),
F-divergence (Namkooung & Duchi 2016) and moments constraints (Delage & Ye, 2010). WRM (Sinha et al.) 2017) gives a general solution to the DRO problem with Wasserstein distance uncertainty set. It converts the Lagrange penalized problem to min-max optimization with constant Lagrange multiplier (Algorithm
2 in Appendix). Lin et al. (2022) surveys recent studies on DRO. Our DRA generalizes WRM by using class-wise uncertainty set and generating sequence of examples in the inner-optimization which shows superiority over WRM theoretically and empirically. In the scope of long-tailed recognition, DRO-LT (Samuel & Chechik 2021) utilizes DRO to alleviate the bias in feature learning which is closely related to our proposed DRA. However, is significantly different from ours the motivation, method, and adaptability. See detailed discussion in Section
3 Consiionddiional Distribution Shift in Long- Tailed Recognition
In this scction, we conduct empirical studies on unreliable class-conditional distribution (CCD) estimation in long-tailed recognition. We first introduce problem setup. Then we proposed new metric to measure the offset between empirical CCD and the ideal CCD. Finally, we present our findings and analysis of CCD shift issue based on the experimental results.
3.1 Problem setup
In clasiffcation/rocoguttioon sctting, a labeled instance a pair (x, y), where x E X R. and y takes value in [L] = {l.....] L} with L the number of classes. We consider a 6-parameterized classifier f, e.g. a neural network, which outputs fo(x)) TR to estimate P(y]z). Denote training set as {(xi,, y:) ] ~ Parain; with Perain =Eee(r] P(rls)))))s) N, the number of instances from class j, PN,LT Eje(r the empirical distribution while (N),L)()),;) = N, Enn= 8(x, v&) where 5 is the Dirac Dirac-distribution. In the long-tailed scenario, P(y) is imbalanced, and we assume that P()1) > ... P(YL). In practice, the model is desired to perform well on all classes for e.g. detecting rare species or making fair decisions (Van Horn
& Perona 2017 Hinnefeld et al. 2018). Hence, the distribution for evaluation is usually class- class-balanced balanced i.e.
Prest Σ 2) P(rly,), In other words, the goal is minimizing the balanced risk (Menon et al. 2020
Rhal
1
Σ
Palys (yj # arg max (Jo(z)),)
(1)
T
vE(T.]
jE
Σ
Er(zl,,) arg maxx(f(x))) 1z)l.
L
vE(L)
JELL]
4

---

Published in Transactions on Machine Learning Research (02/2024)
3.2 Quantifying CCD shift
In previous work, lo estimate Reat, P(æly;) is approxinated by the empirical CCD Menon el al. 2020: Vapnik 1991) (r),)) which is intuitively unreliable due to scarce instances of tail classes. The tollowing proposition explains it more formally.
Proposition 1. For loss function 1x, v) for any L-Lipschitz respect to P(zly) is condi- lional distribulion on IRm and Py(xly) is the empirical distribulion of P(xly) estimaled from N in- stances. Denote A [r(s)][]==]]) < oo for some a > 0, then C11 02 depending on A, a, we have Er(ew)(z Epx(stmll( v)]l with probability at least c//ett) ( mintmm))))
It says with very few instances e.g. in tail classes, training performance does not ensure generalization, the shift between empirical distribution and true distribution cannot be ignored. That is the so-called CCD shift. The proof of this proposition can be found in Appendix A.1 We also provide visualization and analysis of a toy example in Appendix B.1 to validate Proposition and the consequence of CCD shift. Unfortunately, unreliability of CCD cannot be directly computed from above bound due to unavailable real distribution in practice.
Nevertheless, there is a way to quantify CCD shift: estimating CCD with more instances besides the long- tailed datasct as an oracle, which more reliable by Proposition [ Specifically, we choose CCD from the balanced dataset as the oracle, i.e. sampling from the following distribution
Premove shi)e(z,))) = Σ
(2)
JEIL]
named as removing shift sampling. Note that, by sampling the same number of instances as original long- tailed datasets from distribution (2) the CCD is changed while label distribution is kept still for a valid ablation. When training with removing shift sampling,
novel instances are seen by the model. One can use the accuracy gap between models trained with and without this sampling the CCD shift bet training and test. If the empirical CCD is reliable enough to estimate the true CCD, more reliable CCD introduced by Premove shift would not bring significant improvement and the accuracy gap metric would be small. Conversely, a large gap, as on the long-tailed datasets in our experiments, means significant CCD shift.
metric to measure
Specifically, we perform ablation experiments on CIFARI0/100-LT (imbalance ratio:100) without or with removing shift sampling to investigate the effect of CCD shift on vanilla Empirical Risk Minimization (ERM) Vapnik 1991 and other representatives of main-stream re- balanccinn methods including
Deferred Re- Weighting (DRW) CCoo et al. 2019) uses class-balanced re-weight at the later stage of training i.e. after training an initial model by ERM, as a representative of deferred re-balance
methods [Zhou et al. 2020 Cao et al. 2019 Park et al. 2021
Classifier Re- Training (CRT) Kang et al. 2019 re-trains the linear classifier of an initial model
Irained by ERM with the feature extractor frozen, as a representative of decoupling methods (Kang et al. 2019 Zhang et al. 2021a Zhong et al. 2021).
Softmar (PC softmax) HHong et al. 2021 modifies the logits according to the training in the inference phase as ff (z); =
((y)), serving as a special case of post-hoc logit adjustment (Menon et al. 2020) and a representative of logit adjustment methods (Hong et al. 2021 Menon et al. 2020 Li et al. 2022b).
We put more details of empirical study on removing shift sampling in Appendix B
5

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Accuracy decoupling method CRT on CIFARIO-LT under different settings without or with removing shift sampling. Underline means accuracy imbalanced features and small font denotes difference between ce-balanced and imbalanced features. More results are provided in Table (in Appendix B
feature re-balance+
classifier adjust
base
removing shift
CRT
77.72
87.07
DRW
CRT
75.97-1.75
+8.42+1..5
DRW
78.06
89.01
88
(ERM, basel
(DRW. bese)
86
accuracy
84
(ERM, removing shitt)
82
(DRW, removing shif)
80
IDRW, basel
78
remove shift
IERM, base)
INSAl nemove
(ERM, removing shift)
0.8
1.2
1.6
2.0
T
(DRW, removing shift)
CIRARIO
Figure 2: Accuracy CIFARIO-IT with varying in post-hoc adjustment. After removing shift, gets optimal performance while it is sub-optimal with shift of Plzly).
Figure 3: ECE and feature deviation distance with or without removing shift and re-balancing on CIFARI0/100-LT.
3.3 Empirical results and analysis
CCD shift matters and is class-wise. Figure | and Figure (in Appendix B quantitatively show how much CCD shift affects the performance long-tailed recognition. The accuracy curves all methods (vanilla ERM and re-balancing methods) exhibit significant gaps in performance with without removing CCD shifts and the gap is much larger in classes with fewer instances. This confirms that CCD shift 1) is key factor that limits the performance in long-tailed recognition; 2) is class-wise and more severe in tail classes. It motivates us to explicitly address the unreliable empirical distribution and consider varying reliability across classes.
Decoupling methods benefits by avoiding more severe CCD shift. A surprising phenomenon in decoupling methods im-balanced features are more generalizable than re- balanced ones, as in Table But it disappears once the CCD shift removed, with re- re-balanced features leading to better performance than imbalanced ones. We explain that ro-balancing in feature learning assigns higher weights to more unrcliable tail classes thus exacerbating overall unreliablility of the empirical distribution i.e. decoupling overperforms simply re- re-balancing without introducing more CCD shift. While decoupling avoids aggravating CCD shift in feature learning, the shift remains and subsequent classifier adjustment may still suffer from it, leaving space to be improved as in Samuel & Chechik 2021 Zhang et al. (021a): Wang et al. (2021).
Sub-optimality of Fisher-consistent post-hoc adjustment is blamed to CCD shift. Recall that post-hoc logit adjustment adjusts logits to fo(x), log((((y))) The parameter T = 1 is proved Fisher- consistent, however, sub-optimal in experiments (Menon et al. 2020). We demystify this contradiction from the empirical study of CCD shift. Figure 2 shows logit adjustment gets best performance with 5 much bigger than 1 and, however, with removing shift, 1 becomes optimal. We explain that rare instances from tail
We choose DRW get balanced features instead RW (direct weighting) for RW gets the same result whether with removing shift sampling or not: harms first-stage learning. The related explanation be found in Appendix

---

Published in Transactions on Machine Learning Research (02/2024)
classes are regarded as "out-of-distribationn examples and given low confidence to the true class (Wang et al. 2021). So more adjustment to tail classes, i.e. >> 1, adapts to hardly recognizable samples due to CCD shift. This explains the effectiveness of searching adjustment parameters in margin-based adjustment loss [Li et al. 2022b Kini et al. 2021) as well. Morcover, our findings suggest that adjustment and data augnentation shall be inslance-level discriminalive to compensale for sumples hard to be recognized due to CCD shift, agrecing with sample-aware augmentation of recent works (Hong et al. 2022).
CCD shift and imbalance label distribution jointly affect confidence calibration and feature deviation. We additionally study the influence of CCD shift on two issues that are found affected by long-tailed distribution by prior works, confidence calibration (Xu ct al. 2021 Zhong ct al. 2021) and feature deviation (Ye et al.; 2021p. Xu et al.) (2021); Zhong et al. (2021) find that long-tailed distribution makes the calibration of models worse i.e. the output score inaccurate to estimate real likelihood, while Yo et al. (2021) observes that the distance of feature centers (average of all instances) between training and testing data is enlarged by imbalanced distribution, especially for classes with fewer instances. We use ECE (Fxpected Calibration Error) and feature deviation distance as metrics for these issues. See Appendix B.5 for introduction to these metrics and more discussion. From Figure the main observation is that both confidence calibration and feature deviation are co-influenced by CCD shift and inbalanced label distribution. Utilizing removing shift sampling or re- re-balancing method (e.g. DRW) alone obtains much smaller ECE and feature deviation distance, while joint use of them gets even better results. These results imply that CCD shift is complementary to label imbalance, and addressing CCD shift is crucial for not only recognition but also other tasks requiring reliable feature representation.
4 Distributionally Robust Augmentation against CCD Shift
4.1 Formulating class-aware robustness via min-man max optimization
Shift of CCD has been shown significantly limiting the generalization under long-tailed distribution even with existing ro-balancing strategies. Though existing data augmentation could synthesize novel instances for tail classes, they do not target CCD shift directly, and thus may be still biased by the shift and cannot restore the real CCD convincingly (Hong et al. 2022 Ahn et al. 2023).
To this end, we explore addressing this problem from another perspective: making models robust to CCD shift (Some prior augmentation methods can also be understood in this scope, please refer to our discussion among those and ours See 4.3). We utilize Distributional Robust Optimization (DRO) (Sinhal eet al.) 2017 Delage & Ye 2010) to obtain robustness against potential CCD shift, considering the worst risk among set of potential distribution shift. near the empirical distribution instead of trusting it totally
[Menon et al. 2020 Xu et al. 2021 Cao et al. 2019). Formally, we aim at minimizing the DRO risk:
RDRO = L Σ sup Ep, [o(x,))] JELL PEP,,
(3)
with P3 a probability measure on X x [L] (o(2,)) = ((fo(x); y), and P, = (P,lW. (NILT(Z)),) < r Wa Wasserstein distance induced by cost function c((21, y1 (I22, y2)) = ca (21, x2) + oc 1 [wr where (x(21, is nonnegative, lower semi-continuous, and satisfies C, (x, x) 0 as valid cost function to make the Wasserstein distance well-defined (Sinha et al., 2017: Villani 2009)- Additionally, we assume Ca (2),)))) is continuous and (x())))) is 1-strongly convex for any 32 and typically set (x(,), = || - x2 12 as it meets the above properties establish our following theoretical derivation. Different from previous hods (Sinha et al. 2017), the radii of uncertainty sets P, depend instance numbers, which implics different reliabilities by Proposition D Specifically, they are decrcased with the increasing number of instances, i.e. T1 < ... < <TL, to make the model robust enough without over-pessimism (Frogner et. al. 2021).
classes to adapt to imbalanced
Two critical questions about the DRO risk are: how to make it tractable and whether effective in solving CCD shift. For the former, we convert the intractable (3) on the set of potential distributions a min-max optimization problem on the training set:
7

---

Published in Transactions on Machine Learning Research (02/2024)
Theorem (Simplified version of Theorem 7 For loss function (o(x.)) v) continuous æ for any y, our DRO risk equals
RDRO-
inf H1 Σ 2:20JELL]
Agry + (py sup
(o(#),))) >3 c(z, (x,
(4)
T. JELL]
with 2 = (I) E EXx
See the full Theorem and proof in Appendix A.2 Motivated by previous DRO (inha et al. 2017 we
relax A to hyperparameters and omit rj in to simplify the optimization procedure. Then our objective changes from to:
F(0) Z Σ
sup [o(2,,)) > We(Pi,,Ner,)) JELL]
(5)
sup (lo(x,,0z) - X3 c(2, (a,,
(6)
L
==(xx185)
JE[L]
The final objective 6 actually equals to the Lagrange penalty 1 for the original problem as above. We will explain the optimization process in Section 4.2
To show the rationality of the DRO risk, will see that with mild conditions the risk on balanced label distribution with real CCD is partially bounded by our converted objective G
Theorem 3 (Simplified version of Theorem 14). For any positive multipliers {Aybse(A) with mild conditions for the dluss-conditional distribulion P(r Wj) of each cluss iee. cerlain order moment erists, wilh probabilily
the balanced risk is bounded by
Roat S + Σ (,C((N;)))
(7)
jEIL]
with C(N;;)) a constant only depending on Nj and
In fact, n) is from the convergence rate between empirical CCD and real CCD (ourniir & Guillin 2013), indicating the unreliability empirical distribution of each class. It decreases as N2 increases.
The bound shows that CCD shift is essential for long-tailed distribution and our method provides principled solution to the CCD shift. It also reflects our insight of class-aware radius, since more robustness for tail class j by scaling up xy would reduce more risk as the factor C(N,) is larger. We put the full theorem, its proof and related remarks of the above bound in Appendix A.4
4.2 Learning with sequence of augmented data
The min-max optimization 6 can be solved by two bilevel steps: solving the inner max-optimization and computing the outer loss by its optimal solution:
min- Σ E
(o aug ( ((u
(8)
jE(r.
s.t. (x agg mu)) arg (1o Us) by c(2, (2) ==(#..12)
(9)
Specifically, we can apply gradient descent and obtain the last example in the inner optimization loop as aza yaugl I This process can be understood as data augmentation adaptive to current model and training instance, as indicated by the objective: to (zz, Vz) can be seen mining harder examples for the current
In fact, with c, u y for each (z. infinity.
as gu # y turns objective of inner optimization negative
8

---

Published in Transactions on Machine Learning Research (02/2024)
loop:
Vala(z), c((,, vr), (1)
the last example
From the M-th loop
loops
/v
3;
examples
From the k-th...,Miii loops
forward
pepper
lo 196)
outer-optimization inner-optimization
Figure 4: An illustration of DRA. Augmentation examples x; aMg generated from original data 2; in M inner loops (with two choices of the last example sequence of examples) used new training instances (z. yi) to train robust model against potential distribution shift. The order numbers O-A indicate the process of DRA in an overall iteration.
model, with c(2, (x,z) y)) constraining examples not too far away from the original instance and by determining the magnitude of the constraint.
However, as lound our experiments (shown in Figure 5 this strategy causes unstable optimization and bad performance when Ay is relatively small, since brings smaller convergence rate in the inner optimization thus the gradient computed for the last point biased for outer optimization. See more detailed discussion this result in Remark in Appendix A.3 To overcome these limitations, propose DRA Algorithm
(as illustrated Figure which uses a sequence of examples, i.e. the last 8 == M k points, from the loops of inner-optimization, annotated as "a sequence of examples" in Figure 4 The overall optimization becomes:
1 min T
Σ EPy.Lr(elss) jE(M]
1
lo (2 [ugg
(10)
(E(s]
s.t. s.t.((x aug y y Hem from the last loops in max (lo (L Be) Ay clz, v)). == (3).
(11)
Figure 4 and Algorithm (in Appendix C.2 explain the inner optimization detail by illustration and pseudo-code respectively.
Empirically, method achieves better performance with more stable optimization (as shown in Figure 5 and Table 5) Theoretically, i.e. as increases, the following convergence bound becomes more tight.
prove a sequence of examples make the optimization more stable
Theorem (Simplified version of Theorem 9h. Under common conditions from WRM Sinha et al. 2017 there cxist constants C₁ < 1, C2 making below convergence bound valid:
1-(C))" CGe + +
Σ E EI tELd
(12)
T
&Nootch
NuatchT
with Noatch the batch size used and is the step tolerance of inner optimization in Algorition
The bound is formally presented in Theorem g
Appendix
A.3 where more theoretical analysis and
ments provided.
4.3 Discussion with prior works
Relation other augmentation methods. As clarified before, DRA pursucs robustness CCD shift by generating novel examples as data augmentation. Some prior augmentation works can be understood improving robustness to family of potential distributions, e.g. GIT (Zhou et al.] 2022 be seen

---

Published in Transactions on Machine Learning Research (02/2024)
increasing robustness to the family of transformed (by e.g. luminance changes and rotations) empirical distributions while Open sampling (Wei et al. 2022) introduces additional open-set data, bringing robustness to potential unseen subpopulation shift (Duchi et al. 2022). However, the enhanced robustness relies on specific handcrafted operations or auxiliary data c.g. transformations in GIT and open-set datasets in Open Sampling, which may incur additional bias and cannot address CCD shift. Our DRA is a more principled method considering robustness under a ball of distribution discrepancy (Wasscrstein distance (Sinha et al., 2017)), which is theoretically substantiated to handle CCD shift without additional modules and data.
Prior work DRO-LT (Samuel & Chechik 2021 also utilizes DRO to improve long-tailed recognition against instance scarcity. Here, we summarize the main differences between DRO- LT and our proposed DRA. 1) Molivation: our work offers a comprehensive analysis of CCD shift from instance scarcity on existing long-tailed solutions, while DRO-LT aims at remedying biased representation of tail classes in decoupling methods (Kang et al. (2019), which a derived issue of CCD shift, as found in our study (Sec (.3). 2) Method: DRO-LT considers feature-level uncertainty set of Gaussians which is oversimplified to cover the real test distributions and thus fails to give theoretical analysis between enhanced robustness and generalization. DRA considers uncertainty sets under Wassertein distance, which admits novel generalization bound and preserves the advanlage of DRO under Wasserstein distance, i.e. customized robustness by special cost function (Villani 2009; Sinha et al. 2017). Additionally, DRO-LT estimates its objective by a margin-based loss, which slows convergence similar to other margin-based loss YYe et al. 2020; Li et al. 2022b), while the convergence of DRA does not require additional training phase and epochs. 3) Adaptability: DRA can easily integrate with various prior methods while DRO-LT is coupled with centroid classifier and classifier re-training strategy. The latter may not be applicable to other tasks, e.g. object detection (Zhang et al. 2021a), and centroid classifier is rarely used compared with softmax classifier.
Relation to DRO-LT.
5 Experiments
Experiments on long-tailed classification tasks show that DRA significantly improves existing re-balancing strategies, even on strong baselines trained with mixup and RandAngment. More thorough analysis shows the effectiveness of DRA on addressing confidence calibration (Guo et al. 2017) and saddle-point issue (Rangwani et al. b022b Complete experimental details and more results including visualization ion of examples from DRA
are provided in Appendix C
5.1 Experimental setup
Datasets. We conduct experiments on CIEARIO-LT, CIFARIOO-LT, Liny--maaeNNet-DD (Cao et al. 2019 CelebA-5 (Kim et al. 2020) and ImageNet-LT (Luu et al. 2019). The imbalance ratio of CIFAR-LT and Tiny-ImgeNNet-LIT is set to 100. As in prior studies (Cao et. al. 2019 Wei et al. 2022), we report the Top-1
accuracy the test set for Celeb-5 and on the validation set for other benchmarks. Results of ImageNet-LT are reported three splits of classes: Many-shot (more than 100), Medium-shot (20-100) and Few-shot (less
than 20). For specific experimental settings for various datasets, please see Appendix C.1
Comparative methods. We compare our mothods with four groups of related methods: (I) re-balancing
methods including Decoupling (Kang el al. 2019). Logit Adjustment (Menon al. 2020). PC softmax (Hong
let al.
2021), DRS/DRW (Cao et al.)
2019). and (II) their variants: LADE (Hong et al. 2021). IB Park
ett al.
2021 b Vector loss (Kini al. 2021), 1, Auto-balance (Li et al. 2022b). CDT (ee et al. 2020 MFW (Ye
et al.
2021 4 SAM (Rangwani et al. 2022b CC-SAM et al.
2023); (III) data/featun
augmentation 2023), Open-sampling (Wei
methods (DA/FA including Mislas (Zhong et al.] 2021), CUDA (Ahn et al.
let al. 2022), DRO-LT (Samuel & Chechik 2021), GCL(Li et al.
2022a): (IV) head-to-tail transfer methods
including M2m Kim et al. 2020). GIT (Zhou et al.
2022). SAFA (Hong et al.]
2022), CMO (Park et al.
2022
Samuel & Chechik 2021 we apply DRA in the later stage of training since we need an initial model that has fit. the data distribution for DRA to generate
Implementation of DRA. Inspired by (Kim et al. 2020
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Comparisons of accuracy (%) with Mixup and RandAugment on different benchmarks. Balanced Softmax noted as BS. means results with tation degrades Balanced Softmax on ImageNet-LL so DRA is not used with the union of them. i align our training epochs (i.c. 100/300) with CUDA CUDA/DRO-II DRO-II respectively for fair comparison. f means results from the original papers, others are reproduced by us. The best results are in bold.
respectively. *: Mixup-basee augmen-
ImageNet-LT
Method (Mixup/ RandAng)
CIFARIO-LT
CIFARI00-LT
Many
Med
Few
All
CE-DRS
79.7 / 80.89
47.08 46.95
61.76 / 61.13
49.69
49.61
27.14 26.02
50.11
50.13
NE-DRW-SAME
80.6
44.6
56.6
45.8
28.1
47.1
CE DRW.DWWWUUE CUDA
46.76
61.8
48.3
30.3
51.0
DE-DRW-CMO
81,98
47.05
60.8
48.6
35.5
51.21
SAFA
80,48
46,01
63.8
49.9
33.4
53.1
CE-DRS DRS Owr
80.59 82.59
47.39
47.68
61.77 / 61.30
50.23
49.73
27.66
26.14
50.89
50.42
CE. DRW + CUDA Ours
81.83
47.71
61.93
49.45
31.11
51.74
BS
81.23 81.22
46.63
16.81
63.0
63.16
49.80
50.85
27.62
30.35
51.5*
62.275
BS-CurMix
46.46
BS-CUDA
81.35
46.89
61.9
49.2
32.3
51.6
BS-CMO
82.30
46.72
62.0
49.1
36.7
52.31
BS + Ouns
82:21 81.83
48.11
47.69
63.32
51.22
30.87
/ 52.77
BS + CUDA + Ours
82.50
48.71
62.23
49.84
33.07
52.321
DRO-LTT
82.6
47.31
64.0
49.8
33.1
53.5
BS (300 epocla)t
82.19
47.97
63.83
51.53
/ 32.08
13.600
BS (300 epochss)t
34.46
PC Softmax
81.29
46.12
16.20
63.09
50.37
51.15
30.57
30.36
52.41
PC Softmax Ours
82.41
81.41
46.35
46.77
63.09
62.68
51.36
51.89
$2.20
31.58
53.27
/ 522899
CRT
62.77 / 64.11
49.51
50.77
31.75
28.46
51.77 / 52.36
CUDA)
62.3
47.2
28.4
50.2
MISLAS
82.1
47.21
63.3
51.21
33.73
53.16
CC-SAM
82.41
48.77
61.4
49.5
37.1
52.4
GCL
81.37
47.32
54.12]
CRT Ours
64.52
6121
50.87
50.71
33.01 31.81
53.65 / 52.89
augmentation examples. For the class-wise multiplier Az we set X3 x N S with B,S 2 where B determines robustness difference over classes and determincs overall robustness level, which reduces the multipliers of DRA to only two hyperparameterss regardless of the number of classes considered. M, Qinner in DRA (as in Algorithm D are set by our default value and not considered as hyperparameters for tuning while k is set M/2 M corresponding to sequence single example respectivcly. Full implementation details of DRA are in Appendix C.2
5.2
Main results
DRA cooperating with re-balancing strategies. As summarized in Table 3 and 12 iin Appendix DRA improves re-balancing baselines significantly on the five benchmarks, building stronger baselines for them. The improvement on CIFAR-LT more significant under the setting of prior work (Zhou et al.
C.3
2022 with smaller batch size, e.g. overperforming previous state-of-the-art LADE (See Table 14 in the Appendix C.4). These strongly supported that the distributional robustness introduced by our DRA benefits generalization under CCD shift. Moreover, the gains by DRA on ImageNet-LT is not only from few-shot but also from many-shot and medium-shot, verifying distributionally robustness helps the generalization of head classes as well.
DRA cooperating with existing data augmentation. Recently, solutions for long-tail fail learning often rely augmentation schemes from balanced datasets (Samuel & Chechik 2021 Park et al. 2022 Li et al. 2022a). This makes it unclear whether the improvement from them comes from alleviating CCD shift or, better regularization inherent and induced bias of relied data augmentation. To accurately evaluate DRA and make a lair comparison, we examine DRA's 's performance in conjunction with two data augmentation from balanced learning, Mixup [Zhang et al. 2017) and RandAngment (Cubuk et al.) 2020). and a recent strong data augmentation solution designed for long-tailed recognition, CUDA(Ahn al. 2023 As in Table 2 DRA even brings significant improvement for these strong baselines trained with other data augmentations CIFAR-LT and ImageNet-I.TT achieving superior comparable performance to state-of-the-at data
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Comparison of re-
sults on CelcbA-5.
Table 4: Comparison of accuracy (%) on different benchmarks. Balanced
Softmax is noted as BS. The best results are in bold and small red font denoles performance gain.
Method
Acc
ERM
72.7
ImageNet-L.F
LDAM-DRM
74.5
Method
CIFARIO-ET
CIFARIOD-TT
Method
Many
M2m- DDAM
75.6
FRM
40.28
FRM
66.84
46.05
Open Sampling
78.6
PC softmmax
44.34
PC softinax
30.51
51.18
CE-DRS
73.1
+Ours
80.39+0.81
44.8440.50
+uus
62.41
49.72
31.89
51.6440-46
+Ours
75.2
BS
79.65
45.17
BS
62.70
48.81
31.62
51.82
LDAM-DRS
75.9
+Ours
80.33 0.68
45.71 00.54
+Ours
63.09
19.58
32.17
52.42 10.00
+Ours
80.1
DIS
77.21
44.16
CIT
61.59
47.70
30.32
50.3
PC softmax
76.2
+Ours
78.89 1.68
41.87 1..71
Ours
61.81
49.13
31.57
51.37 (1.07
+Ours
77.1
LDAM-DRS
78.71
45.12
Balanced Softana
76.8
+Ours
79.75
15.54
+Ours
77.6
augmentation or head-to-tail transfer methods. Note that DRA integrates with these data augmentation without modifications but surpasses existing solutions based them, e-g. Mislas and CMO. This reflects that existing methods are still biased by CCD shift even with modifications for adapting to long- taildd recognition.
5.3 Ablation study
Class-aware uncertainty set. In the implementation of DRA, class-aware multiplier are set as Ny Nj x S with hyperparameters and S. Through various robustness are assigned while 3 = 0 gives every class the same robustnoss, roducing DRA to existing WRM mothod et al. 2017). As shown in Figure and Figure 0 with different values of ß and S, can reveal the ellects of class-aware radius and validate effectiveness of ours. In Figure 5 class-aware radius exhibits significant superiority under multiple choices of S while class-consiitent radius (in WRM) only slightly improves the performance. Moreover, Figure 6 indicates aware radius (3 > 0) brings significant improvement over class-consistent radius (3-0) and I need to be adjusted carefully to ensure higher robustness for CCD shift.
The number of augmentation examples. To verify the effect of using a sequence points as augmen- tation examples in DRA to calculate outer gradient, we set k in DRA to M M/2(a sequence) or M (single example) to obtain comparative results. Figure 5 shows the effectiveness of DRA with different numbers of augmentation examples. As the bound 12 indicates, a sequence of augmentation examples is more stable than the last point with relatively small S. Morcover, we probe if the effectiveness of our strategy is from using a sequence or examples not the last one. Specially, we compare variant of randomly selecting an example within the last M/2 iteration, noted as 'random example' in Figure 5 The result of this variant is worse than sequence of examples most of the time but overperforms the last example under smaller S, supporting our insight reflectod by (12) that only using the last example is not stable for optimization.
Other cost functions Other than our typical cost function d(-,-) (2), 1 + y23 with (x(21, ||, 22113, we conduct experiments other C₁ E {1|=1 x2llØIp 1,2.4, 1,2) in prior worksSinaa et al. 2017). However, they are not 1-strongly convex fails to apply convergence guarantee 12) or the guarantce from WRM (Sinha et al. 2017). For these cost functions, as in Table we indeed observe that the inner optimization becomes unstable and may cause lower performance than baseline as heuristics in these cases, also effectively
or even diverge. Fortunately, a sequence of examples(k=MM/ alleviates this issue and gains consistent improvement, serving as appropriate augments for only the last example(k==M-1), indicating our strategy generalizing to more cost functions that prior work(Sinha et al. 2017) could not..
12

---

Published in Transactions on Machine Learning Research (02/2024)
5.10
141
CE DCS ORAGHOUE BAM-)
CK und OPAEHOLI randon 1066
CE LRS
44.3
64.5
14.4
712
714
10
100
Figure 5: Accuracy on CIFARI00-LT with re- spect to different choices of hyperparameters S and under / 0 or 3 = 0.30. Random example means randomly selecting onc from the last M/2 examples.
Figure 6: Accuracy of DRA CIFARI0/100-LT with respect to different choices of 3 under S = 70 or S = 100.
Table 5: Accuracy on HIPARI0/100-TH with respect to different choices of cost functions in the form of (21, x- 2) = || - 22 Ilp while c((11, y1 (122, 92)) (21, + [117/2)) The baseline is CE-DRS and 'nan' means
Table 6: ECE (%) on ImageNet-IT. Balanced Softmax is noted as BS. Smaller ECE means better calibration.
the training process does not converge.
w/o mixup
w/ mixup
Method
(r, k)
CIFARIO-LT
CIFARIOO-IT
+DRA
+DRA
p=1
2
p=1
4
CRT
8.78
4.91
5.90
2.79
(2, M-1)
nan
78.89
77.78
nan
44.57
44.45
(2, M/2)
77.83
78.56
44.86
44.31
PC softmax
6.81
4.40
4.21
3.88
(1, M-1)
56.78
73.24
77.34
40.57
44.64
44.72
6.29
3.04
(1, M/2)
79.12
78.07
77.41
44.81
45.12
44.75
5.4 More thorough study
DRA on confidence calibration. Inspired by our discovery that CCD shift degrades confidenco calibra- tion, we evaluate DRA's impact on confidence calibration with the Expected Calibration Error (ECE) metric Guo al., 2017). As in Table [ and Figure 13 14 (Appendix C.5). it kind of surprising that robustness CCD shift from DRA not only improves calibration but also improves the well-calibrated models with Mixup most of the time the ImageNet-LT and CIFAR-LT (See complete discussion in Appendix C.5
Alleviating the saddle-point is- sue. To further prove that DRA would obtain solutions more robust to CCD shift, we analyze the loss landscape of solutions from DRA. Re- cent work(Rangwan) et al., 2022b) an- alyzes the loss landscape of long- learning on wise loss and finds that solutions converge to saddle point in the landscapes of tail classes, which demonstrates poor generalization and instability to distribution shift (Duphhi et al. 2014 We eval- nate DRA by the metrics proposed by Rangwani et al. (2022b), minimum negative cigenvalue of Hessian Aminl and non-convexity ratio Amum on CIFARIO-LT. As in Figure 7 DRA obtains solutions with signif- Amaz
333
30-
Figure 7: Comparison on Amin and Ameaur avoid convergence to non-convex region, as indicated by lower Xmin and Amam]
Amin
DRA-trained models
Amax
icantly smaller Amin and |2mia|, |, which means that DRA finds more flat solutions and quantitatively prove that it alleviate saddle point issue and obtain more robust models.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Table 7: Training cost comparison among DRA and previous methods on CIFARIO0-LT, All values are in seconds. For two-stage methods, time per epoch and epochs for each stage are reported. Additional Time means extra time for pre-training modules beyond end-to-end training. The ratio of training time baseline (CE-DRS) is also reported in overall time.
Method
Time per epoch
Additional
Time
epochs
Overall Time
ERM/CE-DRS
2.85
200
637.0 (1.0x)
GIT
2.85/15.34
32891.2 637.0
160+-40
34010.2 (53.4x)
M2m
2.85/24.75
160+-40
2109.0 (3.3x)
CUDA
5.26
200
1083.0 (1.7x)
RIDE(3 experts)
27.59/6.
200+-5
5681.0 (8.92x)
DRO-LT
2.85/62.3
200++000
6867 (10.78x)
SAM
4.95
200
1044.0 (1.64x)
CC-SAM
18.46 + 4.85
200+-30
3837.5 5 (6.02x)
CE-DRS-DRA(KMM11
2.85/15 19.37
160+-40
1280.6 (2.01 x)
(E-DRS-DRA(--122
2.85 26.45
160+-40
1543.8 (2.42x)
Training efficiency. We make comprehensive comparison of training cost among DRA and related previous methods on CIFAR100-LT in Table All the experiments are conducted on an Nvidia RTX 2080Ti GPU. From the results, we find that 1) long-tailed solutions using DA/FA DA/FA/multi-cxpor /multi-cxpert more or less increase the cost of training, e.g. CUDA and DRO-LT require additional time to compute augmentation score or class prototype each epoch respectively while our DRA introduces an extra inner optimization. 2) The additional training cost. of DRAA is mild as DRA is only equipped in the later stage of training (e.g. the last 40 epochs CIFARI00-LTY and enjoys our theoretical convergence guarantee (i.e. need for more epochs for convergence like DRO-LT), the overall cost. is about only 2x CE-DRS/ERM. which is not big burden, comparable to recent works CUDA (1.7x) and SAM (1.64x), and much less than existing solutions c.g. DRO-LT (10.78x) and GIT (53.4x). For other datasets, as the usage ratio DRA lower e.g. 30 epochs in the whole 180 epochs on ImageNet-LTT, the extra training cost would lower.
6 Conclusion
In this study, we present empirical evidence suggesting that unreliable distribution estimation in long- tailed recognition, exhibiting as shift of P(xly), is a key factor to limit performance even with re-balancing methods. Our findings indicate that only regarding long-tailed learning as label distribution shift problem is sufficiently comprehensive and reviewed existing methods from the view of CCD shift. Through our proposed DRA, show training a model robust to potential distribution shift. tailed recognition, aligning with theoretical guarantees. Moreover, highlight the potential of extending our concepts to the recently proposed multi-domain long-tailed settings Tang et al.) 2022 Gu et al. 2022). as appropriate distribution robustness can simultaneously improve instance scarcity and domain shift issues, which is promising to explore for future works. There is some space to improve DRA, c.g. more precise uncertainty set for potential shift or more efficient solutions to solve bi-level optimization. Overall, to some degree, our work sheds light on the bottleneck of long-tailes recognition and encourages the exploration of further solutions addressing the issue of unreliable empirical estimation.
effectively improve long-
Acknowledgments
The authors would like to thank Ruibing Hou and Nan Kang for feedback on various drafts and helpful discussions. This work partially supported by National Key R&D Program of China no. 20212D0111901, and National Natural Science Foundation of China (NSFC): 62376259.
14

---

Published in Transactions on Machine Learning Research (02/2024)
References
Sumyoong Ahn, Jongwoo Ko, and Se-Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. arXiv preprint arXiv:2302. 05499, 2023.
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputinng 5(4-5):185-196 1993.
Arindam Banerjee, Inderjit Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 112-121, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with babel-distrbbutioo-aaaae margin loss. Aduances neural information processing systems, 32, 2019.
Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. European Conference on Computer Vision, pp. 95-110. Springer, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceed- ings, Part XXIX 16, pp. 694-710. Springer, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. blass-balanced loss based on effec- tive number of samples. In Proceedings of the IEFE/CVF conference on Computer Vision and Pattern Recognition, pp. 9268 9277, 2019.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dinensional non-convex optimization. Advances in neural information processing systems, 27, 2014.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-61) 2010.
John Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses for latent co- variate mixtures. Operations Research, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint Xix::010000441,, 2020.
Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields, 162 738, 2015.
Charlie Frogner, Sebastian Claici, Edward Chien, and Justin Solomon. Incorporating unlabeled data into distributionally robust learning. Journal of Machine Learning Research, 22(56):1 46, 2021.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hossian cigenvalue density. In International Conference on Machine Learning, pp. 2232-2241. PMLR,
2019.
Ian J Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua. Bengio. Generative adversarial nets. In NIPS, 2014.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In International Conference Machine Learning, pp. 2821-3830. PMI.R, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Xiao Gu, Yao Guo, Zeju Li, Jianing Qiu, Qi Dou, Yuxuan Liu, Benny Lo, and Guang- Yang. Tackling long-tailed category distribution under domain shifts. arXiv preprint or arXi::2277 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Henry Hinnefeld, Peter Cooman, Nat Mammo, and Rupert Deese. Evaluating fairness metries in the presence of dataset bias. arXiv preprint arXiv:1809. 09245. 2018
Yan Hong, Jianfu Zhang, Zhongyi Sun, and Ke Yan. Safa: Sample-ddaplive feature augmentation for long- tailed image classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avie, Isruel, October 23-27, 2022, Proceedings, Part XXIV, pp. 587-603. Springer, 2022.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang, Disentan- gling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference Computer Vision and Pallern Recognation, pp.
2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 13896-1333 13995,, 2020.
Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label- imbalanced and group-sensitive classification under overparamcterization. Advances in Neural Information Processing Systems, 34:18970 18983, 2021.
A Krizhevsky. Learning multiple layers of features from tiny images. Moster's thesis, University of Tront, 2009.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Seaffeezadeh-Abadeee Wasser- stein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pp. 130 166. Informs, 2019.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 6929 6938, 2022a.
Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Autobalance: Opti- mized loss functions for imbalanced data. arXiv preprint arXiw:2901.01212, 2022b.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference Computer Vision and Pallern Recognition, pp. 5212-522 5221, 2021.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEFE/CU C/VF Conference on Computer Vision and Pattern Recognition, pp. 6918 6928, 2022c.
Fengming Lin, Xiaolci Fang, and Zheming Gao. Distributionally robust optimization: A review on theory and applications. Numerical Algebru, Control & Oplimization, 12(1):159, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEFF International Conference Computer Vision, pp. 2980 2980-2988, 2988, 2017.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Bo Liu, Haoxiang Li, Hao Kang, Gang Hua, and Nuno Vasconcelos. Gistnet: geometric structure trans- fer network for long-tailed recognition. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 8209 8218, 2021.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long- tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 2970 (970-22999990 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in open world. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognilion, pp. 2537 2546, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
Hongscok Namkoong and John Duchi. Stochastic gradient methods for distributionally robust optimization with -ddvergences. Advances in neural informalion processing systems, 29, 2016.
Yurii Nesterov. Introductory lectures convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998.
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balance loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 735 744, 2021.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6887-6896, 2022.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, and R. Venkatesh Babu. Escaping saddle points for effective generalization on class-imbalane imbalanced data, 2022a.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on ilass-imbalanced data. Advances in Neural Information Processing Systems, 35:22791 22805, 2022b.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balancod meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33: 4175 4186, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255) 252, 2015.
Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9504, 2021.
Soroosh Shafforzadeh-Abadchh. Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1-68, 2019.
Alexander Shapiro. On duality theory of conic linear problems. In Semiiinfimite programmingg pp. Springer, 2001.
Aman Sinha, Hongscok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robust- ness with principled adversarial training. arXiv preprint N.t:1710.110177 2017.
Suvrit Sra. A short note on parameter approximation for von miscs-fisher distributions: and a fast imple- mentation of (x). Compulational Statisties, 27:177-190, 2012.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. Invariant feature learning for generalized lnng-tailed classification. Xie preprint arXiv:2007 2022.
Grant Van Horn and Pictro Perona. The devil is in the tails: Fine-grained classification in the wild. an preprint Xiv: 179997094455 2017.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-taile recognition by routing diverse distribution-awar experts. arXiv preprint arXim:2010.0188 2020.
Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for long-tailed visual recoguition. arXiv preprinl arXiv:2112.077 12.0722, 2021.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Chong Wu. Implicit semantic data augmentation for deep networks. Advances in Neural Information Processing Systems, 32, 2019.
Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified general- ization analysis we-weighting and logit-adjustment for imbalanced learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling: Exploring out-of-listribution data for balannicng long-tailed datasets. In International Conference on Machine Learning, pp. 23615 23630. PMLR, 2022.
Tong Wu, Ziwei Liu, Qingqin Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-tailed distribution. Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recogrition, pp. 8659 8668, 2021.
Zhengzhuo Xu, Zenghao Chai, and Chun Yuan. Towards calibrated model for long-tailed visual recognition from prior perspective. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, 2021. URL http:////opprrrreeeeeeeeerrrrrre
VOZAfN-BOA_
Han-Jia Ye, Hong-
Hong-You You Chen, Do-Chuan Zhan, and Wei-Lan Chao. Identifying and compensating for feature deviation in imbalanced deep learning. arXiv preprint Xix::2001.01. 011385, 2020.
Han-Jia Ye, De-Chuan Zhan, and Wei-Lau Chao. Procrustean training for imbalanced deep learning. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 92-102, 2021.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with redrs-tepresented data. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognétion, pp. 5704-5713, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023 6032, 2019.
Shiran Zada, Ttay Benou, and Michal Trani. Pure noise to the rescue of insufficient data: Improving imbal- anced classification by training on random noise images. In International Conference on Machine Learning, pp. 25817-25833 PMLR, 2022.
Hongyi Zhang, Moustapha Cissc, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint [xXi::1710.0941.. 2017.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 2361-2262 22600, 2021a.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. Xiv preprint orXi::2100.05996 2021b.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for Lest-agnostic long-tailed recognition. Advances in Neural Information Processing Systems, 35:34077 35:34077-34090, 2022.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 10489-16490 2021.
Allan Zhou, Fahim Tajwar, Alexander Robey, Tom Knowles, George Pappas, Hamed Hassani, and Chelsea Finn. Do doep networks transfer invariances across classes? In International Conference Learning Representations, 2022. htttss://openrevieet ott.
Boyan Zhou, Quan Cui, Xin-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVE /CF conference on Computer Vision and Pattern Recognition, pp. 9719-97229 9728, 2020.
Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Am Heng, and Wei Gong. Class-condiitonal sharpness-award minimization for deep long-tailed recognition. In Proceedings of the IEEE/CKE Conference on Computer Vision and Pallern Recognition, pp. 3499 3499-3509, 2023.
A Proofs and More Theoretical Analysis
Lemma 5 Cenitoroovic--Rubister Theorem).
(Villani 2009 5.16) For Wassernstein distance We be- [7 [iaal
tween two distributions Q and Q', Lip(f) suPay.m2
-2ll it admits
Qc(Q,Q'==
sup Lsp(a)<1 Rm
(E))((d))
J
(()) (dE).
Lemma 6 (Measure concentration). (Fournier & Guillin 2015), Theorem 2) For a probability distribution
P on I and PN is the empirical distribution of N instances i.id sompled from P, c(.-) is a cost function inducing the Wasserstein Distance W We(e(a1, x22 || -22llp with p A F) (e(x, < oo for some a > O and Do RRm, then 7e,,0, 02, depending on A, a, PPW.(P, Pw) < c) -n holds, for
{1,2} is special case),
Logles/n) /a/m NN log(cs/n) C2. log(a//1) / if log(c1/n)
E(n)
C2
C2N
02
A.1 Proof of Proposition
Proof.
Ep(el)) y)} (px = L |Ep(tt)) (((x,))/L) - ((x,)))((
SL (.(P(x))), Pa (*ly))
The last equality from Lemma
And directly substituting n in Lemma by _f
gets the
result..
19

---

Published in Transactions on Machine Learning Research (02/2024)
A.2 Theorem and proof
Theorem 7. For loss lo (z, 1 (X' [T]) - IR continuous with respect to for any y and volid cost
function we have
RDRO- = inf A,20.e(L]
T
DArrt + E
sup 10(2,, Uz) Ag c(z, (2,
(13)
SELL
Considering Lagrange penalty with non-negative multipliers {Ay}sete) have the reformulation:
F(0)
1
E
SupEy (o(2.))) X3 Wc(P,, PNLn.)
(14)
L jELL] Pj
Σ
sup 2xxxx1
1o(x; Uz) - X3 c(2, (r,
(15)
L JE[4]
Proof. Our proof generalizes the proof in Sinha et al. 2017). which is a special case Xn C' of ours.
Using the duality firstly, we have
RDRO - Σ
sup Ep, [1)
jE(N) PEEP,
inf PyEp, A;20
1 L
Σ
E Uol v)) - XX,,, Px.r(())))) + Xyr,}
SE[L]
1
inf (Agry
sup PjeP
-
-y, PN,
2,
L
jE(L]
The last equality is based on strong duality which will be explained later. All we need to prove now is
1
Σ EE [o(a, v) Px.r(tl))
sup AEP,
L
JE(L]
Σ
sup (=.
10( v=) A3 c(2, (2, (x)
JE(L]
Actually, we have
1 Σ (o(x
sup T PEEP SE(L)
(x.r(c)))))))))
Σ
/ (() Ao
inf MExtP,,Px.er / c((),EE) MidEi, dE2)
= sup PEP,, ;E[L]
L
1
Σ
/ lo(E.v) - -x,o((E v), M((dE, y). d(2)
sup (P
JEIL)
1
Σ
1x(x,, Vz) X)c((Ez) Ux). M(dEi,dE)
sup PEEP) (P
Σ
sup
JE(L)
Σ ErK.TT JE(L]
sup 2E..
(0(x.. U2) - by clz, (2.
The first equality from the definition of Wasserstein distance and the last equality is from sUP==teww (ex 1o( Uz) Xy o((z is independent of &1. We now explain that the inequality is actually
20

---

Published in Transactions on Machine Learning Research (02/2024)
equality. For every i, we can find 27 (22, for every c > 0, k E N satisfying
lo(=;,))) vi) by c(2] (2i,
sup 2x
10 Vv) X3 c(2, (zi, Wi))
if
sup
10(x2, >3 c(z, (mi, < oo
(
==
U1) - - -0(2,, (x)) 2
sup (.
(0(x)1))) - >3 c(z, (1., = dc
Let M() = N EielN 6(2 (Fi, Wil), and for arbitrariness of 6 or k; the equality is established.
Finally, we prove the strong duality. Let P3 = PNLTS (xly), so Wo(P), pNur (xly)) 0. Therefore, {0....) 0} E
Σ / |lE 2(IE, Q is probability measure on X}, R is the inner point of the set {b E 10, J サ=
1
satisfying the slater's condition of a standard conic programming duality result ((Shapiro) 2001 3.4), thus the strong duality is established.
A.3 Theorem and proof
To analyze the convergence of DRA as in Theorem g one needs an assumption on convexity of the cost function c, on which the cstimation of convergence rate on inner optimization counts. Our choice of the cost function(i.e. |||] - x2112 + x (v)) naturally satisfies the assumption.
Assumption 8. The cost function 2221;22) (x1, oo [() satisfies that ca(:. is 1-strongly convex for any 20.
Theorem 9. Under similar conditions to WRM (Assumption
10 13 Let be the number of it- erations in outer optimization, AF = F(o) - infnetry F(n)), Nbatch is the batchsize, with stepsize of outer optimization a = mint 2i.q; Noatch and inner stepsize ainner = x- Lo = Loo + Los To2
x
with Loz, Loo, I.z0, Tezz; a depending on I. For Algorithm 1. let 8 M k is the number of examples, X minjels/4j. G max Los, Leo, Lzo, Lezh, then for {0,
(0))=0 in the outer optimization, we have
1 T
Σ E SEM
- (1 A >-bea) )a 6T
+ / Nbotch
1
(16)
8N Nbatch
A-baz
Laz
Proof. With similar mild assumptions WRM (inha et al. tionally Robust Angmentation Algorithm I enjoys guarantee of convergence, It finds a e-stationary point of model parameters 0 with O(1/e²) iterations (as fast as SGD Ammai 1993)).
2017 below, one prove that our Distribu-
Assumption 10. The loss 1: 0 x Z(= X' x [4] + IR satisfies the smoothness conditions:
Vol(0,2) 2) - Vol(e', 22) || $ Lao |0 - 1' V.1(0.2 2) V 2 } |||
< Lea ||I - 2=11.
Vol(0,2) - vol(O, |1. < Toz || - =1, V=1(0,=) V_100', =)|| < T.z0 |10 - 0¹
Il, is the dual norm of the norm ||:|| (as our settings, both are 11-112).
We also borrow lemma from WRM as follows:
Lemma 11. (Sinha et al. 2017), Lemma 1) Let f:0x Z + R be differentiable and n-strongly concave in
2. define F(0) = sup f8, Let go( (8, 2) and g= 2) = vzf(0, z) and f satisfies Assumption
10 with replacing with f. Then J differentiable and VoJ Vol (0, 20)) where 2" =0) = sup f(0,2) z).
And we have
L28 10 02
(0,) - =0(2)) <
n
21

---

Published in Transactions on Machine Learning Research (02/2024)
and
V - v/ S (Leo + LOoLoR 110 - 02ll-
Under Assumption 10 the primal of inner optimization 1(0,2) 2,c(2, 20) is (A, - Luz strongly concave, that $j,0 s 1(0, 2) - 2yc(2, 20) is well-defined and satisfics the condition of Lemma 11 Thus, we have Vosup:ez 10, z) Xyc(2, 20) = Volo(=j,o) and VoF(0) is L.oLor -Tipschi. Then we make an assumption on the variance of the gradient the outer-optimization, which is a common condition when analyzing the convergence of SGD.
x,-
Assumption 12. For any sampled training set { U:) VelNN holds
E Fl VoF(O)]| o²,
where "y.. arg sug lo(2) Ase(z, (2:,
With the preparation above, we begin to prove Theorem 9 Let h2(0,, 2; (xI, = (0,=)) - (x:,, In Algorithm 1 the gradient we use to update is actually
Σ
Vohn, (xft", ..)) (... B.))
Nbatoh (E(Neacct]]
LE(+)
Since the gradient of F(0) is Lg-smooth,
F(0++1) <F(O.) (VF(()), 0c+1 01) + Lg 1 - -0:122
=F(O,) a(1 Laa || +a
a(1 + Lock
2 (VF(O.))F(()) -9t)
2
Laa²
[IF(0.) -gelle-
2
We need to estimate F(0:) 9t in the equation above. Let
ge
1(0,, 20.,))))00))))
Nbatct
iE Noutekk
With the help of gt. estimate F(O,) ge more easily, The difference between 9e and of comes from we use a scries of points to compute the gradient to A and the inner optimization cannot obtain optimal solution precisely. The difference between V (0,)) and of is from just sampling a batch from all training instances.
We compute the difference between Je and 97 as 8 = 9t gi then we have
|
2
Vol(Or, (I .w))) Val(Or, ;))
Nhatch iE
[E(()
2
22
2
(alth vi) - 21
Nbatch
Noutek
LE(s)
Assumption 13. The inner-optimization just reaches the linear convergence rate
Nesterov 1998 of gradi-
ent descent in strongly-convex optimization i.c. [[ .y:)
i13 < (1 X-
G (T)) vs) 2 1ll < c.
We make this assumption under the motivation that the last example is enough when the convergence much faster than the convergence bound that the strongly-convex condition gives. However when the convergence of innerooptimization slow i.c. just reaches the convergence bound, using sequence of examples can gain benefits.
22

---

Published in Transactions on Machine Learning Research (02/2024)
We have [|x vi) 21 S x..-1.--. With another assumption that inner-oplimization reaches the the inner
linear convergence rate of gradient descent strongly concave optimization optimization is at least À - Stongly concave, we have
(Nesterov 1998
[|x] v.) .b) 25 51 A
Luz ) Xy. G
1
L1
1-(1- 3 gear 4haz
As result, || S &Nbacch
A-hix
L=
Substituting with de, we have
F(0++1) <F(O,) [IFF(0))1 a 5 -2-2-111 L. 116.122
- VF(O.) 9:) + Loa (11::11 (VV -
For Lemma 11 9 unbiased estimation to VF((,) so Flge VF.0.) = Using Assumption 12
control the invariance of the estimation and taking expectations of the above formula, we have
E[F(0(+1) F(6.)) 2
-
a
3 -a
1 1- 1- (1- - (1-2-pa)) 4L 2
Nbatch Laa 0² +
49 sNbatch
d-has G
Laz5
where we use 2 get LoO0 < 3 a.
Summing by t, we get
AF T Σ EIVPOON
1
2 2 aT Nbatch
6
1 (1
A- A-bizz
4L0="
Isga? +
G
s.Nattch
d T122
Substituting with the stepsize a, the conclusion is obtained.
Remark 1. Actually, in the inequality (16), considering that XÚ is small but concavity is established (Aş A-L < 1, then the first term of the bound reduces with the increase the number of Lez), we assume G examples thus the overall optimization becomes more stable.
Remark 2. Previous WRM can be regarded as using the last point to compute gradient for outer-optimization, since if inner-oplimization is concave with small radius (corresponding to big X2), it is the optimal solution, which leads to unbiased estimation for the gradient of outer optimization(Sine the situation that with small Äj, the inner optimization is relatively slow and concavity may even not established. Therefore we propose a more moderate way by using a sequence of examples from the iterations of inner-optimization instead of making a locally optimal point the only example. Theorem gives evidence to our above insight.
et al. 2017). Considering
A.4 Generalization bound and optimality of augmentation examples
In addition, give bound the estimated balanced risk on real distribution and the optimality of examples generated by DRA under mild conditions.
Theorem 14 (Generalization bound). Assuming 1: θ x mmm x [L]) + R is continuous and for every class j, let. the real class-condilional distribution P(ælyy) salisfies 3x > O and ao E Rm A = FEpell,) oo with ez a valid cost function, then 3c,, C2 only depending on A, a, for any multiplier (Ayhse(L), with probability 1 11 it holds
<
Rbat,1 S Σ
1. (A, maxt log(es//) p/mm llglel (og(es//)),nn 1103 C2N,
1
C2N, coN,
SElL]
+ EPH.ar(ele)l 2=(0e.Ns)xxxx(
sup
[(x,, - A, c(z, (x, w))}
23

---

Published in Transactions on Machine Learning Research (02/2024)
in which Roal.1 := = i. Erole (uc y)} is the balanced risk on test distribution Piest EJELL] P(zlus) estimated by loss function (x.y).
Proof. Using Theorem
for any multiplier {AybsE(Z); we have
sup Eg (z, 1
Σ
AyT3
L
I
JE(L] PjEP,
jE[L]
+ E
sup 9.
1x Uz) A3 c(2, (z, v)),
in which ry is the radius of Pj-
And with our assumption, P(r)u;) satisfies conditions of Lemma G
So let rj
max {( (ogla)//) ya/m, ymmm (logles/m) [/a.. By applying Lemma with the probability of 7 the dis-
e2N, (o(P(x)))) (N,Er(Z)s;)) < ry and we have p(r E Pj = [P,]WW(PP, PN < r, Thus, it
tance
establishes
Σ SE[L]
1 E F( L
1 L
Σ sup E (((,,))) jE[L] PEEP
and we get the conclusion.
Remark 3. This bound shows with high probability the balanced risk real distribution could be bounded by our objective (the last term in the right) plus constant depending the multiplier A and the number of instances from each class, which serves generalization gap, decreasing with the number of instances Nj. It indicates A controls a trade-off between robustness and performance. If we do not enforce any robustness i.e. the multipliers \Ú are extremely large, DRA actually equals to ERM. In this situation, large X leads to considerable generalization gap, especially for the tail classes. Hence, we conclude that models robust to CCD shift benefit long-tailed recognition. Conversely, making X extremely small near zero, admits trivial bound meaning the model refuses to make predictions, which well-known as "over-pessimism" in DRO Frogner et al. 2021
The above bound also shows new viewpoint beyond existing theoretical results of long-tailed recognition in the communily. We make some remarks for the bound as follows:
1. It is more flexible and tight than the results of prior DRO works (inna et al., 2017 Kuhn et al. 2019). It adjusts wise robustness by Xj for each class j, while prior work only admits a special case of X; = for all j E [Z].
2. It is closer to practical training compared to the Fisher-consistenue result by logit adjustment (Menon et al. 2020 Fisher-consistency theory from Theorem in (Menon et al. 2020) only states that a Bayes-optimal classifier would be obtained by the minimization of logit adjustment loss under a balanced label distribution with real CCDs.
3. It is no longer dependent on the capacity of the hypothesis class, which is extremely large for modern neural networks, while the bound in Theorem 1 of (Cao et al. 2019 is.
Proposition 15 (Optimality of augmentation examples). Assuming 1 θ x (R" X (L]) R is contin- wous and Assumption g is satisfied. T,((x. v)) == arg tax (x Uz) Xÿ c(2, ()) is the optimal solution of inner optimization, M is the set of probobility measure on X x [L] and ( NLT., E(se)=w)) ((T,((+;,)))) the empirical distribution consisting of them. Then for any multiplier (}A}ELL)) we have
( = arg max Ep,, [(x v)] - Px PyEM
which indicates the empirical distribution consisting of ideal optimal solutions of inner optimization of class is the optimal perturbed distrilution of the Lagrange penally problem.
24

---

Published in Transactions on Machine Learning Research (02/2024)
Proof.
sup EØ, [1, y) >3 XjW.(P3, PN,cr(zlU;)) PyEM
= -Erner
sup
Ux - X3 + c(2, (2,
(EXX
L X3
(TT) Mi)), (24,
((:))=)))
s
w) Wy us). PN,L (xlu)))
The first oquality is from Theorem and the second is from the definition of (xly)) The inoquality is from the definition of Wasserstem distance. For the inequality in the opposite direction is trivial, we establish the conclusion.
7
A.5 Proof of 17 in the toy example Proof.
1
Σ log( EMIL
N
Z
VE[N.]
=0) ksp x-
while (I, v)] is the entropy of vml distribution (2) k.((K) Sra 2012
And the conclusion is
established.
B Detailed Empirical Study CCD Shift
B.1 toy example to validate CCD shift
In Sec we argue that even with Fisher-consistent loss function, loss computed on empirical distribution with scarce instances unreliable estimate the risk on real test distribution, as Proposition indicates, exhibiting the shift between empirical CCD and the real CCD. To further clarify the concept of CCD shift, we visualize and analyze loy example.
We assume that there exists data from three classes with imbalanced ratio of 10 5 5: 1, sampled from an ideal feature space, i.c. the feature x is located on 3-sphere: 5 = {xr E | =1. He is the feature prototype of class c, located on the vertices of a regular simplex i.e. uniformly distributed on the hypersphere (Li
et al. 2022c Graf et al. 2021). For each class c E {0, 1, 2}, the instances are sampled from the distribution P(x(c), a vMF distribution dastribbutionBBanerjee el. al. 2005) on the S with the density p(z(c) = emm"" " Z is the normalization constant 2m(e"-e-") Specifically, set == 1 and sample N₂ 10 instances for class 2.
Figure visualizes the sampled instances {= their normalized mean projected to S 2a
T with d
N. Ei 2 and the prototype For class 2 that has fewest instances, the normalized mean of the sampled instances deviates from the prototype w2,with cosine similarity of 0.7424, significantly lower than similarities other two classes. These indicates the empirical distribution of the class with scarce instances is not reliable estimation to the real distribution of the class, Le. CCD shift.
4² [
Furthermore, one can check the shift more formally, by computing the value of a specific loss function on the empirical CCD real CCD. Taking 1(x, y) Eeetal 1 log(p)og(p(()))) as the loss function, the difference between empirical CCD Py (x/c) = 1 N. EiENN.) (xx)) and real CCD P(xlc) is
EPpw v) FEP (x. will luf A(s))
(17)
25

---

Published in Transactions on Machine Learning Research (02/2024)
Ovarika of Claaass
Class
Camgle Sun
prcorvec of
Comple Gen13
1.04
5.7
325
370
327
625
10
275
648
4.
Soitar Aungoo
748
Figure 8: Illustration of CCD shift by a toy example. Subfigures (a), (b) and (c) correspond to class O, 1, and 2. We report average similarity and loss difference between empirical CCD and real CCD de on 50 runs. The visualization is from one run and its similarity and loss difference are also reported. The estimated average mean of tail class 2 deviates from the prototype significantly, along with less similarity and larger de, indicating the unreliability of empirical CCD from scarce instances.
noted as d(c; k), where A(k) Jq (k)
-5( is a constant and Jn (x) is Bessel functions of the first kind (Sra 2012 2
See the derivation of (17) in Appendix A.5 As in Figure 8 the tail class 2 exhibits much more difference de than class 0, 1 with more instances, acting as en evidence to Proposition [ Empirical CCD cannot istittate
real CCD reliably for classes with few instances, therefore CCD shift cannot be disregarded in long tailed
rocognition.
B.2 Removing shift sampling
Detailed description of removing shift sampling. Many benchmarks of long-tail recognition, such as
CIFARI0/100-LT (Cao et al., 2019) and ImageNet-LT (Liu al. 2019), are down-sampled from balanced dataset are sampled respectively from
datasets. Formally, instances from balanced datasets and
(N) 1) = Σ
Phat(M;) Pay
(18)
JELL]
PNILT(I,)) = Σ
(19)
JELL]
Here PN, (rlly))) Noat.; Σ E( 612; v1) and (x)u))()))s) NET, Ejen= 8(1; v:). Besides, Poat is uniform and PI)(y) v) is imbalanced. To conduct an ablation comparison with respect to CCD, we replace the CCD in 19 with PN, (z(l)), which is more reliable by Proposition 1, as an oracle for real CCD. While or (ar (x) may not approximate real CCD well, as long as the approximation brings significant improvement on performance (as in our experiments and see more in next section Appendix B.2) we can identify CCD shift according to Proposition 1. That is, sampling from the following distribution
Premove shife(I.y) = Σ
JELL]
which is our proposed removing shift sompling in Section 3.2
Assumption removing shift. To conduct experimental comparisons on CCD, we propose removing shift sampling in Section 3.2 An idcal situation for comparison on CCD is to obtain real CCD P(zly) and keep imbalanced label distribution P(y). In fact, due to no access to the real P(æly), we make the empirical CCD PN, (xly) serves as an oracle to substitute real CCD P(xly). At the same time, we keep imbalance by using the same label distribution Pv) as long-tailed datasets so that the label distribution is imbalanced in every batch and it is in the whole training. In this way, more unique instances are seen by the model but the number of samples from each class, i... label distribution, keeps unchanged in all batches
26

---

Published in Transactions on Machine Learning Research (02/2024)
during training. So under assumption, we make fair ablation study on CCD with proper control of other factors. It is possible that (rlba cannot estimate real CCD well, but as long as the oracle brings improvement in performance, we can blame CCD shift correctly. The counterexample appears in extreme cases when the oracle does not make improvement c.g. using re-weighting with removing shift sampling in the training process, where the removing shift sample does not bring improvement. In that case, we cannot tell whether the CCD shift does not appear thus should not be blamed or our oracle is too weak to identify the CCD shift.
B.3 Experimental setting
We perform ablation experiments on CIFAR-LT without or with removing shift sampling to investigate the effect of CCD shift on vanilla ERM [Vapnik 1991) and some representalive re-balancing methods: DRW
Cao et al. (2019) CRT (Kang et al.
2019 and PC softmax (Menon et al. 2020 Hong et al. 2021
Implementation of removing shift sampling. We do removing shift sampling on original balanced dataset CIPAR10/100 (Krizhevsky, 2009). We use a dataloader with a class-iballanced sampler which samples with the same class probability PaT(y) as the long-tailed dataset and samples uniformly within cach class. The accumulated probabilities of instances from each class equals the predetermined class frequency and every instance within class has the same sampling probability. To this end, we implement the sampler multinomial distribution sampler in the same way that used to generate CIFAR-LT in Cao et al.] (2019) Different from the generation of long-tailed dataset, we sample from the whole balanced dataset and get class-inbalaneed instances in every batch. In addition, We keep the same amount of data as the long-tailed dataset while only change the cansicondiiional distribution via removing shift in the ablation experiment.
More implementation details. We train ResNeet-32 (He et al., 2016) with batch size 256 optimized by SGD with momentum 0.9 and weight decay 5 10-4 with warm-up scheduler, consistent with the setting of our experiments on DRA. The learning rate decays by factor of 0.01 at epochs 160 and 180 with an initial rate 0.2. We evaluate Top-1 accuracy on the original validation set of the datasets, following the common protocol in long-tailed recognition (Cao et al. 2019).
Methods in experiments. We select ERM as baseline, and DRW, PC softmax (equals to post-hoe logit adjustment) and CRT as representative re- re-balancing methods.
1. ERM. ERM means empirical risk minimization Vapnik 1991 We train a model on long-tailed datasets with cross-entropy loss without any strategy.
2. Balance. For easy reference, we also train a model under our setting on the original CIFAR10/100.
3. DRW. We implement DRW (Cao al. 2019) following the original paper. We train in regular way in the early phase and re-weight the loss by the inverse of the predetermined class frequencies only in the last phase during training. We typically re-weight the loss function starting at 160 epochs.
4. RW. RW mcans directly re-weighting the loss by the inverse of the predetermined class frequencies. We re-ghttht the loss in the whole training phase. In ablation experiments on decoupling methods, we utilize DRW and RW to obtain re-balanced features and ERM to obtain imbalanced features in the first stage of training.
5. PC softmax. We implement PC softmax following (Hong et al. 2021 by training the model by ERM for 200 epochs and applying adjustment at testing.
6. CRT. We implement CRT following (Kang et al., 2019 We train the whole model for 200 epochs and ro-train the classifier for 10 epochs with reighithting loss. We restart the learning rate when re-training the classifier.
LWS. We implement LWS following (Kng et al. 2019 We assign weights to classifiers with learnable scale factors i.e. W, ft W,, [Z.] We train the whole model for 200 epochs and
27

---

Published in Transactions on Machine Learning Research (02/2024)
ERM
DRW
Accuracy
0.8
0.6
0.2
10
50
90
100
PC softmax
CRT
Accuracy
0.8
0.8
0.4
0.4
70
10
50
90
Class index
Class index
Figure Accuracy on CIFAR100-LT without or with removing CCD shift. It shows similar results that models trained with removing shift sampling get significant performance improvement.
Table 8: Accuracy of different methods on CIFAR-LT without or with removing CCD shift
CIFAR100-LT
CIFARIO-LT
removing shift
base
removing shift
base
ERM
50.7
40.28
80.37
73.24
DRW
63.97
44.41
89.01
78.04
PC softmax
58.61
44.34
87.90
79.58
CRT
Balance 52.82
43.48
87.07
77.72
66.73
91.53
then train factors Ji for 10 epochs by re-weighting the loss. We lune the learning rate on different datasets when learning scale factors as we found is somewhat sensitive.
B.4 More results and analysis on CIFAR100-LT
On a whole, results on CIFAR100 agree with those on CIFARIO-LT. In Figure and Table removing shift sampling improves the performance of re-balancing methods and ERM on CIFAR100-LT, which indicates again that shift of CCDs is the key to limit long-tailed recognition.
As for decoupling methods, Table shows re-balanced features outperforms those uniformly trained sig- nificantly on two decoupling methods CRT and LWS. It agrecs with our explanation of why decoupling methods work: first-stage learning without re-balancing avoids more severe CCD shift while re-balancing with removing shift could benefit fcature learning.
With removing shift sampling - = is optimal, while without the best 7 much bigger than 1. This result again agrees with our supposition that logit adjustment gets sub-optimal experiments and leaves space be improved due to CCD shift.
Results of logit adjustment on CIFAR100-LT in shown Figure 10
B.5 Confidence calibration and feature deviation under CCD shift
As two recently studied issues in long-tailed recognition, confidence calibration (Guo et al. 2017) and feature deviation Ye et al. 2020) have drawn an amount of attention. We conduct. experiments to invest igale how the shift of CCDs affects these two issues.
Confidence calibration. Confidence calibration is to make model prediction estimate true correctness likelihood well, which important in many applications. It has been proved that the calibration of neural
28

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Accuracy of decoupling methods, CRT and LWS, on CIFAR-IT with different features, without or with removing shift. LWS shows similar results to CRT: re-balance harms feature learning while benefits feature learning with removing shift.
CIFARI00-LT
CIFARIO-LT
feature re-balance
classifier adjust
removing shift
base
removing shilt
base
CRT
52.82
43.48
87.07
77.72
DRW
CRT
58.39
42.36-1.12
88.42+1.33
75.97-1.75
LWS
57.67
44.05
87.55
76.21
DRW
LWS
61.95+4..22
43.63-0.42
$8.95+1144
75.36-0.84
DRW
63.97
44.41
89.01
78.04
RW
CRT
48. 18.11-4.71
28.48 15.0
79.51-7.56
71.43.6.29
RW
LWS
49.03-8.64
29.76-14.22
78.75-8.80
71.78-4.43
RW
49.92
30.48
80.34
72.78
Balance
66.73
91.53
accuracy
50
45
40
base
remove shift
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
Figure 10: Accuracy CIFAR100-LT with varying in post-hoc adjustment. After removing shift 1 gets optimal performance similar to that on CIFAR10-ITT.
Table 10: ECE on CIFAR-LT of different methods without or with removing shift sampling.
CIFARIO-LT
CIFARI00-LT
removing shift
base
removing shift
base
ERM
6.4
15.5
15.91
29.75
DRW
2.00
10.77
2.19
22.16
PC softmax
2.55
9.25
2.73
19.43
CRT
4.40
14.54
19.5
26.63
Balance
1.77
3.89
29

---

Published in Transactions on Machine Learning Research (02/2024)
DRAViemove)
Uset Accuro
Skp Accurity
AccetkCy
Accuracy
222
SC T9
PCE
IcErremaee
CRT
CAThremevel
1.0
One
Sup
1.-
2.5
2.0 0.2
2.6
0.5
3.0
3.0
0.2
0.4
3.6
300
0.0
0.2
0/1
0.6
0.8
3.0
0.0
0.2
0.4
0.5
08
3.0
Figure 11: Reliability diagrams of different models trained on CIFAR1O-LT. "(remove)" means using remov- ing shift sampling and PCE means PC softmax. The gap between model confidence and error probability is reduced by removing shift sampling significantly.
Table 11: Average feature deviation distance CIFAR-LT of different methods without or with removing shift sample
CIFARIO-LT
CIFAR100-LT
removing shift
base
removing shift
base
ERM
0.83
1.27
2.11
3.18
DRW
0.693
1.20
1.96
2.90
Balance
0.73
2.10
networks is generally bad and long-tailed distributions make neural networks even more miscalibrated and over-confident (Guo et al.) 2017 Zhong et al.] [2021). Expected Calibration Error (ECE) is widely used to measure the calibration predictions, ECE is calculated as:
model. With all instances divided into B interval bins of equal size by their
B Σ
S N
ECE
(ace(S.) - coo.ff.f)))))
in which S; is the set of instances whose predictions fall into the i-th bin. acc(-) and conf(-)) compute the the accuracy and estimated confidence on S; respectively. As shown in Table 10 re- ce-balancing methods improve ECFs except CRT on CIFAR-100 and removing shift sampling improves the calibration of different. models significantly. However, using both ro-balancing mothods and removing shift sampling is still worse (in ECE) than that balanced dataset. We suppose the reason lies in that we still cannot perfectly model real P(æly) even with our sampling method the whole sample size is the same down-sampled long-taile tailed dataset, fewer than original balanced dataset. Figure 11 shows the reliability diagrams of different models trained on CIFARIO-LT. With removing shift sampling, the uncertainty estimation of models is significantly improved.
Feature deviation. Feature deviation phenomenon found in long-tailed recognition by recent works YYe et al.] 2020 2021). That is the average distance of features learned from long-tailed training dataset usually exhibits imbalance between training and test and the distances of tail classes are larger than those of
30

---

Published in Transactions on Machine Learning Research (02/2024)
Ealkecd EX3
Feature devation distance
Feature devatton distarce
-00
DRW
2.-
0.2
Index
Class Index
(a) CIFARIO-LT
(b) CIFAR100-LT
Figure 12: Class-wisc feature deviation on CIFARIO-IT/CFARIOO with removing shift sampling.
Even with removing shift sampling and re-balancing it still exhibits imbalanced feature deviation both CIFARIO-LT and CIFAR100-LT.
head classes ovidently. Ye ct al. 2020 proposes a feature deviation distance dis(j) to measure the deviation for class j:
R mean(Sx(190 (2 )))) mean( (go (2 (21
dis(jige) 1
in which Il-112 is T.2-norm, SK means sub-sampling K instances from class R is the number of sampling rounds, and is the feature extractor. And for convenient comparison, we also use a class-wise mean of feature deviation distance, called average feature deviation distance:
dis
Σ SE[T]
dis(j:go).
(20)
L
Table 11 shows feature deviation is affected by shift of CCDs as removing shift sampling improves this metric the whole while the re- balancing method (i.e. DRW) also generally alleviates the feature deviation no matter with removing shift or not. Besides, training equipped with removing shift scemingly (regardless of re-balance or not) approachs or outperforms the balanced result, e.g. 0.69/0.83 vs 0.829 on CIFAR10-LT and 1.96/2.11 vs 2.10 on CIFAR100-LT. One may conclude that imbalance does not make an obvious difference on feature deviation. In fact, this because features of head classes benefit from training on imbalance distribution, thus being more compact. As result, using average feature deviation distance the mcasure, re-balance seems not critical for alleviating feature deviation.
Figure 12 shows wisss-wise feature deviation is affected by imbalance obviously as well. The feature deviation distances of different classes still exhibit imbalance with removing shift and there exists significant gap of feature deviation distance of tail classes between balanced training and re-balanced training with removing shift. We suppose that is because re-balancing methods used the experiments still leave space improve feature deviation, e.g, it has been discovered that the imbalanced logit distribution harms feature learning in the beginning stage of training (Ye et al.) 2021
C Complete Experiments DRA
C.1 Experimental details
CIFAR100-LT and CIFAR10-LT. Unless specifically stated, use the setting from (Hong al. 2021 CIFAR10/ 1000-L by default. Under this sctting, we apply SGD with batch size 256 and the base learning rale 0.2 to train a ResNet-32 (He et al., 2016) model for 200 epochs, following (Hong et al. 2021 We employ
the linear warm-up learning rate schedule for the first five epochs and reduce at epochs 160 and 180 by a
factor of 0.01 and use the same weight docay 0.0005 as previous works (Cao et al.) 2019 Zhou et al. 2022). For the setting of [Zhou et al. 2022) in the Table 14 the batch size and learning rate are changed into 128 and 0.1 respectively, and weight decay set 0.0002, keeping the same with (Zhou et. al. 2022 to make
31

---

Published in Transactions on Machine Learning Research (02/2024)
fair comparisons with previous (orks et al., 2021 Kim et al. 2020 Zhou et al. 2022 we find their hyperparameters are sensitive with the batchsize and weight decay.
Tiny-ImageNet-LT. Following (Park et al.) 2021 we apply SGD with batch size 128 and weight decay 0.0002 to train a ResNet-18 model for 100 epochs. We set the base learning rate to 0.1 and reduce it at epochs 50 and 90 by factor of 0.1. Unless specifically stated, we perform training for 180 epochs by default.
ImageNettLLT. Following (Hong et al.) 2021), we apply SGD with batch size 128 and weight decay 0.0005 to train ResNet-50 model. We perform cosine learning rate scheme with initial learning rate of 0.05, Unless specifically stated, we perform training for 180 epochs. We report the results of training 90/180 epochs in Table 13
CelebA-5. Following Kim et al. 2020 Wei et al. 2022), we apply SGD with batch size 128 and weight decay 0.0002 to train a ResNet-32 model for 200 epochs. We employ the linear warm-up learning rate schedule for the first five epochs and reduce it at epochs 160 and 180 by a factor of 0.01 with an initial learning rate of 0.1.
Reproducing baseline results. We combine our DRA with various existing methods and compare their results with other baselines methods in long-tailed recognition. For baselines combined with DRA, we implement them based on their official codes with default. parameters in their modules under our training setting for fair comparison, such as model architecture and optimization hyperparameters. There are a few results disagreeing with original numbers reported in their papers, caused by their different training settings from ours, e.g. we get 44.34 top-1 accuracy of PC softmax on CIFAR100-LT while the reported number is 45.3 in (Hong et al., 2021), which is because the baseline implemented by us is not strong them i.e top-1 accuracy of ERM 40.28 40.1 and they only perform experiments under one random seed. For other baselines, we reproduce the results using publicly available official codes or direct use numbers reported in the original papers. For all results we implement or reproduce based on official code, we run five times with different seeds and report the average.
Study for saddle-point issue. We use the official code_ of (Rangwani et al. 2022a) to estimate Amin and Amaz, eigenvalues of Hessian by calculating the Eigen Spectral Density (Ghorbani al.) 2019). The Eigen Spectral Density is calculated on the average loss of training instances from specific class. Specially, Rangwani et al. 2022a) uses the PyHessian library, which uses Lanczos algorithm to compute the complete Hessian eigenvalue density efficiently. Amin and Amar are obtained from the complete Hessian eigenvalue density.
C.2 Implementation details of DRA
The data augmentation process of DRA is summarized in Algorithm ] while the prior work WRM Sinha
et al. 2017 is illustrated by Algorithm 2 for reference.
Candidate value of class-wise multiplier. Noting that the classwware multiplier X has a negative correlation with the radius of uncertainty set so a positive correlation with the number of instances of class j, thus we set Xラ- normalize(N, $1 S with 8, 2 and normalize meaning scale the vector to unit vector. B determines robustness difference over classes([=0 reduces to the class-consistent À as in Algorithm 2 and S determines overall robustness level. In this way, we could reduce the multipliers of DRA to only two byperparameters. We set M = 10 and Oannner = 0.1 in DRA. As for the hyperparameter k the number of iterations starting to be used to augment. examples, we set as half of the whole number of iterations M heuristically or M M-1, 1, which is equivalent. to directly utilizing WRM as the solution of our primal problem. We do not fine-tune to get an optimal number but only select from the above two choices, which, however, gets pretty good result.
Using stage of DRA. Inspired by (Kim al. 2020 Samuel & Chechik 2021 , we apply DRA in the
later stage of training since we need an initial model that has fit the data distribution for DRA to generate augmentation examples. Specifically, for CIFARIO-LT, CIFAR100-LT and CelebA-5 we start to use DRA
32

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm DRA: augmenting with sequence of examples
Input: (() Nostek }, a batch of instances
Input: (Ay}se(n), the class-aware multipliers
Output: Augbatch {<,
.y) .))))
Naatch (MM-)
Augbalch + ()
5: for == 1...., Nbutch do
6: x, f Ti, 1 O, 96 f (2e, 0, ... 00)
7: while <M and |19;1 > do
Dr = V.)y,c((a), Vr), (mi, vi)) - ((f(()))))))))) v:) (Compute gradient of inner-oplimization)
1+1 x ainner 9, {Update augmentation examples by gradient]
4;
10:
if 2 then
11:
Augbatch concate((Augbatch:;
(Save sequence of examples)
12: end if
13:
1=1+1
14:
end while
15:
end for
16: return Augbatch (Return augmented instances for outer-optimization)
Algorithm 2 WRM (Sinha et al. 2017): augmenting with the last example
1: Input: {Z;;,) M }, batch of instances
2: Input: {A}, the multiplier
3: Output: Augbatch = {x aug .y:)) Nontok
4: Augbatch + {}
for 1...., Nbatch do xf x4, 1 0, 93 +- (2e, 0, .))
while 1 << M and |1/
do
8:
g Vadc((a/, vi), (ii, v)) ([fo(z)), v:) (Compute gradient of inner-optimization)
I. [+1 + ainner of {Update augmentation examples by gradient) }
9:
10:
1=1+1
11: end while
12:
Augbatch = concale((Auuunnn (Augbatthh; (1))):))) (Save the last example)
13:
end for
14: return Augbatch (Return augmented instances for outer-optimizaatonn)
33

---

Published in Transactions on Machine Learning Research (02/2024)
160 epochs, for Tiny- Lin-llmageNet-LI start to use DRA on 90 epochs. For ImageNet-LT we start to use DRA on 80 epochs when training for 90 epochs while we start to use DRA on 150 epochs when training for 180 epochs. For methods that trained for 300 epochs, DRA is started to be used on 240 epochs. And for CRT, we only use DRA the training of the second stage.
Setting of hyperparameters. Following Hong et al. 2021 Kim et al. 2020), we tune the hyperpa- rameters of DRA by grid search on the validation set. WRM uses the following strategy to determine the hyperparameter multiplier: À C. (x while C 0.04 a constant and E (z) [||=|||2] is the average norm of all instances in training set. However, this strategy is not helpful in determining the hyperparam- 248.39 CIFARIO-LT while it is 222.42
eters S. For example, (x))l)][l=) numbers are similar but the optimal S found by our grid search on the two sets disagrees a lot. Morcover, scaling the optimal S on CIFARIO-LT by the proportion of (x) [|]] does not work well either. How lo determine the hyperparameters for DRA more elegantly could be a future research point to further improve our method.
CIFAR100-LT, The two
C.3 Full results on Tiny ImageNet-LT and ImageNet-LT
The experimental results on Tiny ImageNet-LT and ImageNet-LT are shown in Table and Table 13 spectively.
re-
Table 12: Comparison of top-1 accuracy (%) Tin- f means results from original papers, means reproduced results from official codes, means requiring more training epochs. Best results are in bold and small red font denotes performance gain.
Method
Tiny-ImageNet-II
ERM
34.20
CE-DRS
36.02
LDAM-DRS
37.66
GITT
21.99
LDAM-DRW
37.43
MFW'1
35.4
CDT
37.9
IB;
40.40
CE-DRA
35.11 +0.91
CE-DRS-DRA
36.18+0.11
LDAM-DRS-DRA
39.07+1.41
C.4 Full results on CIFAR10/100-LT
The experimental results and comparisons with prior methods on CIFAR10/100-11 are shown Table 14 See See 5.1 for reference of methods compared.
C.5 DRA on confidence calibration
Inspired by the discovery in empirical study that confidence calibration is effected by both imbalance and shift of CCDs, we expect DRA would relieve over-confidence of models. From our experiments, it is kind of surprising that DRA not only improves calibration but also improves the well-calibrated models by Mixup most. of the time. We use ECE /uo et al. 2017 as the measure.
scems just with re-balancing methods model cannot be calibrated well and DRA actually gets smaller ECE on these methods, which validates again that confidence calibration is affected by shift of CCDs and DRA improves by making the model more robust to the shift.
In Figure 13
34

---

Published in Transactions on Machine Learning Research (02/2024)
Table 13: The full performances on ImageNet-LTT. The small red font denotes performance gain. | means results from the original paper.
Method 90epochs
Many
Medium
Few
All
ERM
65.1
35.7
6.6
LADE
60.34
47.37
27.82
49.20
Auto balancet
49.09
CRT
61.22
47.52
26.41
49.52
+Ours
61.63
47.53
30.61
50.25
+-0.72
PC softmax
60.4
46.7
23.8
48.9
+Ouss
60.38
46.81
24.23
49.11+-022
Logits Adjustment
60.62
47.33
27.53
49.25
+Ours
60.63
47.84
27.60
49.49+0.24
180epochs
ERM
66.84
40.89
11.54
46.05
LADE
62.80
49.76
33.4
52.14
CRT
61.59
17.70
30.32
50.3
+Ours
61.81
49.43
31.57
+1.37+11.77
PC softmax
62.13
49.25
30.51
51.18
OOrs
62.41
49.72
31.89
51.644-0.c
Logits Adjustment
62.70
48.81
31.62
51.82
+Ouss
63.09
49.58
32.17
52.42 +0.60
Comparison of top-1 accuracy (%) on CIFARI0/100-LT. means results from original papers, $ mcans reproduced results from official codes, means requiring more training epochs. Best results are in bold and small red font denotes performance gain.
Table 14:
Method
CIFARIO-LT
CIFAR100-LT
Method
CIFARIO-LT CIFAR100-LT
Under the setting Hong et al. (2021)
Under the setting of Zhou et al.
(2022)
ERM
73.24
40.28
ERM
70.45
38.21
LDAM-DRW
78.67
45.05
M2n-LDAM;
77.50
42.20
LADE;
79.87
45.39
Vector losst
80.77
42.15
Ours
75.08+1.84
41.37++1.09
Auto-balancef
78.85
43.30
GIT-LDAM
78.49
43.49
Open samplingt
79.05
42.86
CE-DRS
77.21
44.16
CE-DRS
75.87
41.21
+Ours
78.8941.08
44.87+0.71
+Ours
77.38+1.51
42.61+1.40
LDAM-DRS
78.71
45.12
LDAM-DRS
77.47
42.78
+Ours
79.75+11.4
45.54
+Ours
78.76+1.29
43.53+0077
PC softmax
79.58
44.34
PC softmax
77.92
41.86
+Ours
80.39 +0.81
40.54
+Ours
79.28
43.44++1.59
Logit Adjustment
79.65
45.17
Logit Adjustment
78.07
42.08
+Ours
80.33+0.68
45.71- -0544
+Ours
79.13+11.06
42.96+0.88
35

---

Published in Transactions on Machine Learning Research (02/2024)
CE-CHS-ORA
0.6
0.3
0.2
0.2
105-10.57
8.5-8.78
ECE+2
PCE
PCE DRA
PCE
PCE ORA
Gep AcOuracy
Gap Accuncyy
Cap Accuracy
Gap Accuracy
0.6
0.5
100
8.04
0.2
0.4
0.8
1.0
0.0
0.4
0.6
0.
0.0
0.4
2.0
Ceriidanc
Comtiiuanca
Confidercce
Confiddacce
(a) CIFARIO-LT
(b) CIFAR100-LT
Figure 13: ECE (%) and reliability diagram on CIFARIO-IT/CIARIIOLLTI DRA gets better calibration performance marginally. PCE means PC softmax.
Besides, considering the discovery in recent works (Zhong et al. 2021 Xu et al. 2021 that Mixup has significant positive effect on calibration, conduct experiments to measure ECE with both Mixup and DRA. Figure 14 shows DRA further improves calibration on the basis of Mixup most of time. The only exception is CE-DRS on CIFARIO-LT, in which DRA weakens calibration but boosts its accuracy as shown in Table 2. It seems with both Mixup and DRA the rogularization on logits too strong obtain good calibration the model gives confidence even lower than the accuracy instead of usual over-confidence on neural network (Guo et al. 2017). These results also raise interesting question: Do calibration and accuracy agree in long-tailed recognition? Or more specifically, does good calibration mean good model in long-tailed recognition (Xu et al. (021) Actually, our experiment shows that it is possible to boost performance while weakening calibration.
C.6 Visualization of and discussion on examples generated by DRA
To get more precise understanding, visualize the examples generated by DRA, in Figure 15 It exhibits that DRA adds pixel-wise transformation to instances in order to generate harder examples data augmentation. Some of these transformations tend to erase conspicuous features in an instance by adding "noise". In this way, model is encouraged to extract broader features instead of overfitting features it has learned. For example, DRA seems to add "noise" to crase the ladder from "fire engine" in the first row. As ladder is a salient feature for fire engine, and this augmentation makes model attend to more recognizable features c.g. water tank and red color. Similarly, in the sccond row, DRA adds "noise" on the beaks and claws of the bird and encourages the model to take in more features to help recognition instead of over-depending the two features.
In addition, more surprisingly, DRA can catch semantic information while discard nuisance information by itself. As the fourth row, the yellow sweet peppers happen to be transformed to green while pixels out of the peppers almost kept unchanged. In other words, DRA makes a transformation of color, and the transformed instance still belongs "sweet pepper" class. In this way, DRA performs transformations that keep the semantics unchanged. As shown in the bottom two rows, DRA could give transformations on the background (nuisance information) and keep semantic information: almost all of the pixel-wise transformations in the
36

---

Published in Transactions on Machine Learning Research (02/2024)
CF-GRS
0.6
Expcial boocle/Auoora Accrn))
0.2
0-2
FCR
FC6m4.48
3.22
PCE
PCE ORA
PCE
PCE DRA
Cap
Gop Accuracy
Cap Accuracy
Gap Accurayy
Accuracy
0.2
6.4
0.0
0.6
0.2
0.4
0.8
3.0
0.4
1.0
0.2
0.8
Confiddance
Corridarce
Confislance
Carfiderio
(a) CIFARIO-LT
(b) CIFARI00-LT
Figure 14: ECE (%) and reliability diagram CIFARIO-LITCCIARRIO-I with Mixup. DRA even improves the calibration of models with Mixup, which have obtained small ECE and been calibrated well. PCE means PC softmax.
last but one row lie in the sky, and the semantic object "road" isn't changed. In the last row, the pixels of sofa are not transformed while the background is changed.
From these results, infer that sceking harder examples for robust model sometimes agrees with transfor- mations that keep the semantics of instances. For example, background intensity could serve as examples to obtain distributional robustness under a few situations, and this can be an explanation of why GIT improves performance by adding transformations keeping semantics of instances. However, when transformations that keep semantics are not that effective to gain distributionally robustness, DRA tends to seck other examples with obvious features are erased.
From the observation that examples from DRA do have keep semantics and seem like instances with "noise", hope DRA could suggest for unknown distribution shift instead of utilizing information to reduce the shift, e.g. GIT, ISDA (Zhou
motivation for data augmentation: making more robust model
ct al. 2022 Wang et al. 2019 Li et al. 2021 This motivation, making model robust to shift, could give hints to explain the rationality of existing data augmentations that don't have to keep semantics, e.g.
adding pure noise to instances as data augmentation (Zada et al. 2022) and those based on Mixup (Zhang
et al. 2017
Chou et al. 2020 Zhong al. 2021 Xu 2021
37

---

Published in Transactions on Machine Learning Research (02/2024)
j-1
3
5
8
10
IOPATRED
Figure 15: Examples generated by DRA CIFAR-LT. Left most column original examples and the number of iterations. DRA tends to add pixel-level transformations to obtain harder examples while some of these transformations agrec with semantic information.
38


Reviews:
[{"review_id": "HOpTAccYKn", "reviewer": "Reviewer_v3rC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n$\\cdot$ The long-tail image recognition problem is an important one in practice.\n\n$\\cdot$ The empirical comparison between DRA and existing baselines is solid and extensive. \n\n**Weanesses**\n\n1) The concept of CCD shift is not clearly explained in the paper. I understand the mathematical meaning of CCD, but the removing CCD shift experiment is quite confusing to me (see Q1 below). Since the CCD is a core concept in this paper, it should be better explained via some visualization. The paper only shows accuracy of removing CCD, but the exact concept of CCD is not properly illustrated. Specifically, it is not clearly stated that removing CCD shift in Section 3.3 brings more training data and the description in the paragraph **Implementation of removing shift sampling** is not clear. \n\n2) The effect of DRA is claimed to mitigate CCD shift. However, this effect is not supported by experimental results. The experiment shows that some performances are improved by DRA, but the reason behind this improvement is not necessarily connected to removing CCD shift. In other words, is there any quantitative measure for the reduced CCD shift in the experiment? It would be more convincing to add such a measure. \n\n3) The performance of DRA as shown in Table 2 cannot support its effectiveness in some cases. For instance, on ImageNet-LT, the performance of DRA is not as good as CMO in Few data setting, and not as good as MISLAS in both Few and Med setting. Could the authors add the discussion on the lower performance of DRA in such cases? In addition, in the previous version of the submission, the performance of PC Softmax and PC Softmax + Ours in ImageNet-LT Few with Mixup is 32.42 and 32.0. Why does the performance of PC Softmax and PC Softmax + Ours become 30.57 and 32.20 in this version? \n\n**Additional comments**\n\nNote that the three points are all discussed by the authors in the response of a previous conference, where the authors state that they **would add a discussion of above issue (the three concerns) in the next version of our manuscript**. However, those updates are not shown in this updated version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a distributionally robust augmentation (DRA) to alleviate the so-called class-conditional distribution (CCD) shift in long-tail image recognition. The effect of removing CCD shift is first unveiled by an empirical study and then DRA is proposed to alleviate this CCD shift problem in long-tail image recognition. The proposed DRA generates adversarial examples by maximizing the loss function while keeping the sample close to the original one measured by $l_2$ distance. One difference between adversarial training and DRA is that DRA uses samples from the optimization trajectory while adversarial training only uses the sample of final step. They show that the proposed DRA has some benefits over baselines on some datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs to go through a major revision to address the concerns and answer the questions in the **weaknesses** section.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jn3mW2vPUe", "reviewer": "Reviewer_CjJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The experiments and analyses to investigate the adverse impact of CCD shift are comprehensive. Specifically,  the authors employ various methods under settings with or without removing shift sampling to show that CCD shift is a key factor that limits the performance. Then, the authors take a step further and reveal that CCD shift hinders re-balancing methods in feature learning and causes sub-optimality of Fisher-consistent post-hoc adjustment. Additionally,  the authors also demonstrate that CCD shift affects confidence calibration and feature deviation. Through these thorough experiments,  they effectively illustrate the adverse effects of CCD shift.\n- This paper offers numerous theoretical analyses to demonstrate DRA's rationality and solidity. Firstly, the authors convert Distributional Robust Optimization (DRO) into a min-max optimization problem and present the objective function.  The authors introduce a theorem to prove that balanced risk can be bounded by the objective. Furthermore, they propose DRA, using a sequence of examples to enhance DRO and make the optimization more stable. They further provide a theorem to explain the rationality. These theoretical analyses makes the proposition of DRA reliable. \n- Numerical results indicate that DRA is effective. Specifically, empirical results show that DRA can enhance different re-balancing methods and can cooperate with other data augmentation to further improve accuracy. What's more, the experiments also demonstrate that DRA can improve confidence calibration and alleviate the saddle-point issue.\n\n\n\nWeakness:\n- The authors solely conduct experiments comparing \"CE-DRS-SRA ($\\beta=0.30, k=M-1$)\" with \"WRM ($\\beta=0,k=M-1$)\" to investigate the effectiveness of class-aware radius, which might not be sufficient. Similarly, the investigation of the utilization of sequence-augmented data is also limited.\n- There are some typos. For example, \n  - In section 4.3, \"which is a derived issue of CCD shift, as found in our study (Sec 3.3)\" should be \"which is a derived issue of CCD shift, as found in our study (Sec 3.3)\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper first conducts a series of experiments to study the adverse impact of the Class-Conditional Distribution (CCD) shift on the long-tailed recognition problem. Then, the authors propose Distributionally Robust Augmentation (DRA) to make models robust to CCD shift. Meanwhile, some theoretical analyses are provided to prove its rationality. Finally, empirical results demonstrate the effectiveness of DRA on long-tailed classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Additional experiments to study the effectiveness of class-aware radius and sequence augmented data might strengthen the work.\n\n-  Compare and review a broader range of SOTA methods. \nFor example, they might consider methods like \n\n- \"Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition\" , \n- \"A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning\", \n- \"Class-Conditional Sharpness-Aware Minimization for Deep Long-Tailed Recognition\",\n- \"Learning with Multiclass AUC: Theory and Algorithms\"", "broader_impact_concerns": "There are no concerns about the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VYnytXPfov", "reviewer": "Reviewer_gVMi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Paper Strength**\n\n- The paper addresses an important and challenging problem of long-tailed recognition, and provides a comprehensive analysis of the CCD shift issue from both empirical and theoretical perspectives.\n- The paper conducts extensive experiments on five long-tailed classification benchmarks, and shows that DRA can significantly improve the performance of existing re-balancing and data augmentation methods, as well as alleviate some issues such as confidence calibration and saddle-point.\n\n**Paper Weakness** \n\n- How does the proposed method handle the uncertainty and noise in the data, especially for the tail classes that have scarce and unreliable samples? \n- **Lack of ablation study on the choice of cost function.** The paper uses a specific cost function $c$ for the DRO problem, but does not provide any justification or analysis for these choices. It is unclear how sensitive the DRA method is to different choices of $c$. A more thorough ablation study on the DRA method would be helpful.\n- **Lack of clarity and details in the theoretical analysis.** The paper provides a generalization bound for the DRA method, but does not explain the meaning and derivation of some terms and conditions in the bound, such as $C(N_j)$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a data augmentation method termed Distributionally Robust Augmentation (DRA) to enhance the robustness of models to class-conditional distribution (CCD) shift in long-tailed recognition. The paper conducts a thorough empirical study on the impact of CCD shift on existing re-balancing and data augmentation methods, and shows that CCD shift is a critical factor that limits the performance of long-tailed recognition, especially for tail classes. The paper then formulates a min-max optimization problem to train models that are robust to a class-aware uncertainty set of potential distributions. The paper also presents an efficient algorithm to implement DRA by generating a sequence of augmented examples in the inner optimization loop.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The primary questions for the rebuttal primarily arise from the \"weaknesses\" section.", "broader_impact_concerns": "What are the possible ethical and social implications of the proposed method for long-tailed recognition, especially for the underrepresented or marginalized groups that correspond to the tail classes? How can the method ensure fairness, accountability, and transparency in its decision making?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the soundness of the proposed DRA, as well as the extensive experiments. 
The revision has addressed all the concerns of the reviewers.
Decision: Accept (poster)