Beyond re-balancing: distributionally robust augmentation against class-conditional distribution shift in long-tailed recognition
Authors: Keliang Li
Hong Chang
Shiguang Shan
Xilin CHEN
Venue: Submitted to ICLR 2023 (2023)
Keywords: Long-tailed recognition
data augmentation
distributionally robust optimization
im-balance

Abstract: As a fundamental and practical problem, long-tailed recognition has drawn burning attention. In this paper, we investigate an essential but rarely noticed issue in long-tailed recognition, Class-Conditional Distribution (CCD) shift due to scarce instances, which exhibits a significant discrepancy between the empirical CCDs for training and test data, especially for tail classes. We observe empirical evidence that the shift is a key factor that limits the performance of existing long-tailed learning methods, and provide novel understanding of these methods in the
course of our analysis. Motivated by this, we propose an adaptive data augmentation method, Distributionally Robust Augmentation (DRA), to learn models more robust to CCD shift. The convergence and generalization of DRA are theoretically guaranteed. Experimental results verify that DRA outperforms related data augmentation methods without extra training cost and significantly improves the performance of some existing long-tailed recognition methods.

Under review as a conference paper at ICLR 2023
BEYOND RE-BALANCING:
DISTRIBUTIONALLY RO- BUST AUGMENTATION AGAINST CLASS-CONDITIONE DISTRIBUTION SHIFT IN LONG-TAILED RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
As a fundamental and practical problem, long- recognition has drawn burn- ing attention. In this paper, we investigate an essential but rarely noticed issue in long-tailed recognition, Class-Conditional Distribution (CCD) shift due to scarce instances, which exhibits significant discrepancy between the empirical CCDs for training and test data, especially for tail classes. We observe empirical evi- dence that the shift is key factor that limits the performance of existing long- tailed learning methods, and provide novel understanding of these methods in the course of our analysis. Motivated by this, we propose adaptive data augmenta- tion method, Distributionally Robust Augmentation (DRA), to learn models more robust CCD shift. A new generalization bound under mild conditions shows the objective of DRA bounds balanced risk on test distribubtion partially. Exper- imental results verify that DRA outperforms related data augmentation methods without extra training cost and significantly improves the performance of some existing long-tailed recognition methods.
INTRODUCTION
Recently, visual recognition has achieved significant progress, driven by the development of deep neural networks (He et 20166) as well large-scale datasets (Russakovsky et al. 2015). However, in contrast with manually balanced datasets, real-world data often has a long-tailed distribution over classes i.e. a few classes contain many instances (head classes), whereas most classes contain only few instances (tail classes) (Liu et al., 2019 Van Horn & Perona 2017). Training models on long-tailed datasets usually leads to degenerated results, including over preference to head classes,
undesired estimation bias and poor generalization (Zhou et al. 2020: Cao et al. 2019 Kang et al. 2019).
To solve above issues, various solutions have been proposed. Many of them focus addressing imbalanced label distribution for simulating class-balanced model training. Direct re- be-baanncing, like re-sampling and re- weighting, is the most intuitive (Huang al., 2016/ZZang al., 2021b) Recently, the two-stage methods, which apply re-balancing strategy in tuning classifier (Kang et al, 2019) or defer re-weighting after initialization (Cao al. 2019). have been verified effective. Logit adjustment uses margin-based loss or post-hoc adjustment to rectify the biased prediction caused by long-tailed distribution (Menon al. 2020 Ren al. 2020 Hong an input-label pair as (a:, y). classification or recognition models are trained to estimate the posterior probability Pzy|z) x Ply) In long-tailed recognition scenarios, most solutions actually obey the following assumption: the class distribution P(y) shifts from training to test (usually class- imbalanced training but class-balanced
202T). Formally, denoting
test), while the class-conditional distribution (CCD) P(xly) keeps consistent, i.c. Ptrain (y) * Py)ec(y) and Pirain (xly)) Prese( (xly) (Menon et al, 2020 Ren et al.) 2020). Under this assumption, a series methods including direct re-balancing and logit adjustment have been proved Fisher-consistee (Menon et al. 2020).
We argue that although the consistent CCD assumption et al. 2020) is reasonable if there is no sampling bias within each class, estimating P(rly) by empirical CCD is unreliable, especially for tail classes where the samples are extremely scarce. Therefore, to obtain a generalizable model, the shift between empirical CCD and the ideal CCD cannot be ignored. Our focus does not overlap

---

Under review as a conference paper at ICLR 2023
Aicuracy
0.0
0.5
1.0
Accuracyy
00-
0.-
0.5
0.4
Class index
Class index
Figure 1: Accuracy on CIFARIO-LT with or without removing CCD shift. All methods show sig- nificant improvement after removing shifts. And the improvement mainly appears in classes with fewer instances, which verifies the empirical CCD distributions of tail classes are more unreliable. Shaded regions show 95% CIs over 5 runs.
with existing methods that attend to scarce tail instances or inconsistent P(xly). Transfer learning and data augmentation have been proven effective from the motivation of increasing the diversity of tail classes (Kim et al.) 2020 Zhong et al., [021]; Zhou et al. (022). But they are still possibly biased due unreliable empirical distribution and usually lack theoretical guarantee. Some recent works focus on inconsistent class-conditional distribution caused by domain bias or attribute-wist imbalance (Gu et al., 2022; Tang et al. 2002), which is under shift from unreliable estimation as well. Nevertheless, the influence of CCD shift has not been thoroughly investigated and there has been no effective solution yet.
In this work, we perform an empirical study to quantify the effect that the shift of P(xlv) will have on long-tailed recognition, by using CCD from balanced datasets as an oracle to alleviate CCD shift. With this oracle, the performance of existing methods is significantly improved. shown in Figure I. which indicates CCD shift is a key factor in limiting the performance of long-tailed learning. From the CCD shift perspective, also give new insights to counter-ittuitive facts of previous methods e.g. why decoupling methods (Kang et al. (2019) work and how Fisher-consistent parameter in logit adjustment (Menon al. 2020 gets sub-optimal performance.
Motivated by our empirical study, to enhance robustness against CCD shift, we propose Distri- butionally Robust Augmentation (DRA) which assigns class-aware robustness, generalizing (Sinhal et al. (2017) and admits novel generalization bound. This bound verifies models robust to CCD shift benefit long-taile recognition. Our experiments show DRA improves various existing methods significantly, validating our theoretical insights. Our main contributions are highlighted as follows:
We identify a rarely noticed but essential issue long-tailed recognition, class-conditional distribution (CCD) shift, and provide new insights from the CCD shift view into some existing methods.
To train models robust to CCD shift, we propose DRA with theoretically sound modifi- cations over prior DRO methods Sinha et al. 2017), which admits novel generalization bound verifying that models robust to CCD shift benefit long-tailed recognition.
Extensive experiments on long-tailed recognition show effectiveness of DRA: it signifi- cantly improves existing re-balancing methods and achieves comparable performance to state-of-the-arts Moreover, DRA outperforms related data augmentation methods without additional training costs.
RELATED WORK
2.1
LONG-TAILED RECOGNITION
Re-balancing methods. In this section, review broader scope of re-balancing methods. We regard methods trying to solve the prior shift of Pg)) as re-balancing methods. Long-tailed learning can be considered special label shift problem whose testing label distribution is known while the
2

---

Under review as a conference paper at ICLR 2023
original label shift problem needs to detect and estimate the unknown label distribution (iitonn et al. 2018 Garg et al. 2020; Storkey 2009; Latinne et al. 200I]. Directly re-sampling or re- re-weighting by the inverse of class trequencies is intuitive but causes over-fitting to tail classes or optimization difficulties (Zhang et al. 2021b] Cui et al. 2019). Deferring use of re-balancing strategies (e.g. DRS/DRW (Cao et al. 2019| Zhou et al. (0200) and decoupled methods which only re-balance the classifier (e.g. CRT. LWS (Kang et al. 2019)) are proved more effective (Zhong et al. [022] Zhang al. 202 Beyond class frequencies, more efficient coefficients for re-balance are explored. Focal [Lin et al. 2017) and CB (Cui et al. 2019) use the predicted logits or the effective numbers to re-weight classes, while IB (Park et al. (022)) adopts influence function of every instance to revise weights. Another strand of re-balancing methods proposes margin-based loss to heuristically assign a larger margin to tail classes, e.g. LDAMM (Cao et al., 2019) and RoBal (Wu et al. 2021). Logit adjustment (Menon et al. 2020) utilizes margin-based loss to adjust logits, and the proposed logit adjustment loss and post-hoc adjustment are equivalent to PC softmax and Balanced softmax respectively (Hong et al., 2021 Ren et al.. 2020). As for the convergence of re-balancing methods, has been theoretically proved with re- re-balance weight matching the margin, the loss is Fisher- consistent under consistent class-conditional distribution assumption (Menon et al.] (200)). Recent works find logit adjustment leaves space to be improved by adding an instance-aware confidence function Zhong al. 2021) or regularizer on posterior probability (Hong et al. 2021).
Information transfer and data augmentation. Some methods use transfer learning from head to
tail classes to remedy the lack of data diversity. M2m (Kim et al. 2020) formulates an additional optimization to translate head instances to tail instances and GIT (Zhou et al. (022) uses GAN (Goodiellow et al., 2014) to learn nuisance transformations to transfer invariances for tail classes. A recent assumption is that features from head classes and tail classes share intra-class variances while observed variances positively correlate with class frequencies (Yin et al. 2019)). Under this assumption, some methods transfer variances by sharing classifier weights (Liu et al. 2021] or sampling features from estimated Gaussian distributions (Liu et al. 2020). On the other hand, data augmentation methods are also proved effective to increase the diversity of instances, e.g. ISDA [Wang et al., 2019), mixup (Zhang et al. (017)). To adapt to long-tailed settings, some methods make modifications by adjusting the probability of classes in mixup (Chou et al.) 2020 Xu et al., 2021 or adding label smoothing after mixup training (Zhong et al. 2021), while MetaSAug (Li al. 202T) uses meta-learning to estimate covariance in ISDA.
Challenging Ptrain (xly) = Ptest Prest(xll)). Very recent works challenge the shift of P(xly) as well and propose new perspectives for the long-tailed recognition problem. [Tang et al. (2022) discover that real-world data exhibit both class-wise imbalance and attribute-wise imbalance, but the latter has long been overlooked. So they propose invariant feature learning (IFL) framework using IRM (Ahuja et al., (020) to deal with this generalized long-tailed learning problem. Besides, Gu et al. (202)) considers the situation that the instances of a class are from different domains, and proposes united framework based on meta-learning under the multi-
setting. GIT (Zhou et al.) 2022) Pxest(xly) with the discovery that the model cannot give in- variant predictions for class-agnostic transformations tail classes because of the less diversity of tail classes.
implicitly challenges Pirain (xly)
Other related methods. DRO-LT (Samuel & Chechik 2021) proposes a margin-based loss for better feature extractor which is an upper bound of a Distributionally Robust Optimization problem. CDT (Ye et al. 2020) finds the feature deviation between the training and test data, especially for minor classes, and proposes aware temperature. MFW (Ye et al. (021)) further explains feature deviation is from larger gradients tail classes in the training process and proposes to mix features within mini-batches to balance the gradients.
2.2 DISTRIBUTONALLY ROBUST OPTIMIZATION
Distributionally Robust Optimization (DRO) Kuhn et al. 2019 Delage & Ye 2010) considers the worst-case risk minimization on a series of potential distributions, called uncertainty sets, which are usually determined based various computations, e.g. Wasserstein distance Kuhn al. (2019), f-divergence Namkoong & Duchi (2016) and moments constraints (Delage & Yo 2010). WRM 2017) gives a general solution to the below DRO problem for smooth loss functions:
(Sinha et al.
min sup Eyleo v),, (p (e((pPN) < r). PEP
3

---

Under review as a conference paper at ICLR 2023
where the uncertainty set is ball under Wasserstein distance We. WRM converts the Lagrange penalty above problem to a min-max optimization with a constant Lagrange multiplier and solves it by Algorithm
min sup E(a.g)>pwlla Mo(2's v/) >c((2', v'), y))l 2,0 (x'.0")
Our DRA generalizes WRM by using class-wise uncertainty set and generating a sequence of ex- amples in the inner-optimization, which shows superiority WRM theoretically and empirically. Zhang et al. (2020b) utilizes DRO to train models which are robust to label shift, orthogonal to our model which is robust to potential CCD shift. Lin al. (2022) surveys recent studies on DRO.
CLASS-CONDITIONAL DISTRIBUTION SHIFT IN LONG-TAILED RECOGNITION
In this section, we give empirical studies on unreliable class-conditional distribution (CCD) esti- mation in long-tailed recognition. We first introduce problem setup. Then we leverage balanced distribution to manually remove CCD shift in ablation experiments. Finally, we present theoretical and empirical analysis based on the experimental results.
3.1 PROBLEM SETUP
In clasifiicttion/rcogniitii setting, a labeled instance is a pair (I,, y), where takes value in [L] + (1.... L} with L the number of classes and x IR.. We consider a 0-parameterized classifier f. e.g. neural network, which outputs fx() E IR estimate P(y|z). Denote training set as (11,, (())
Ptrain, with Ptrain = JE(EL] P(us))(sllv) and N, the number of instances from class j. In the long-tail scenario, P(y) is imbalanced, and assume that P(y1) > ... P(yz)- In practice, the model is desired to perform well on all classes for the purpose of e.g, detecting rare species or making fair decisions (Van Horn & Perona 2017 Hinnefeld
2018). Hence, the Z P(z In other words, the
distribution to evaluate is usually class- balsceal i.e. Prest
goal is minimizing the balanced risk (Menon et al. 2020
Roal
1
Palys (vi + arg max ff jE(L]
Σ
(1)
L
vE[L]
1
Σ jE(E)
E flo/1 arg max (fo DE
In previous work, to estimate Riat. P(xly;) is approximated by a loss function e.g. cross- entropy and the empirical distribution (Menon et al. 2020 Vapnik 1991) PN, (æly,)
Ny Σ Einayy 5(x4,y6, yt), which unreliable under long-tailed setting due to scarce instances of tail classes. It can be explained more formally by the following proposition.
Proposition 1 For a L-Lipschitz loss function P(rly) is conditional distribution on R and (x is the empirical distribution of P(zly) estimated from N instances. Denote = Ep(xly) [|]=]|]] < oo for some ox > then Cı, C2. depending on A, a, we have with probility least () mimttm))))))
[Ep()))((( EpE(O
(xx( (1xx,))) <
It says with very few instances e.g. tail classes, training performance does not ensure generalization, iee. the shift between empirical distribution and true distribution cannot be ignored. That is the so-called CCD shift. The proof of this proposition can be found in Appendix A Unfortunately, unreliability of CCD cannot be directly estimated from the bound Proposition | due to unavailable real distribution.
3.2 IDENTIFYING CCD SHIFT BY LEVERAGING BALANCED DISTRIBUTION
Many benchmarks of long-tail recognition, such as CIFARI0/100-LT (Coo et al. 2019) and ImageNet-L.T (Liu et al. 2019). are down-sampled from balanced datasets. Formally, instances

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracy of decoupling method CRT on CIFARIO-LT under different settings without or with removing shift sampling. The underline means accuracy of imbalanced features and small font denotes difference between re-balanced and imbalanced features. More results are in Table 4
feature re-balance
classifier adjust
base
removing shift
CRT
77.72
87.07
DRW
CRT
75.97-175
88.42
DRW
78.06
89.01
from balanced datasets and long-tailed tailed dataset are sampled respectively from
PN,bal y) Σ (1, PN,,
(3)
jE(L]
PN,IT(I,U) = E PVT(U, (N, PN, Px,, jE(N]
(4)
Here Pat Noat.; Σ ((tggy; 8(2s; Me) and (NG.rr(alus)) NET. Ei ) 8 Vt). Besides, Phar(y) is uniform and PLT(3) is imbalanced. To conduct an ablation comparison with respect to CCD, we replace the CCD (4) with Bat,bat(a()))) which more reliable by Proposi- tion I as oracle for real CCD. While (x) may not approximate real CCD well, long as the approximation brings significant improvement on performance (as our experiments and see more in Appendix [B.I). can identify CCD shift according to Proposition That is, sampling from the following distribution
Premove shift = Σ Purlys)PNy Ba, (2ly;)
(5)
JELL]
named as removing shift sampling. In fact, the obtained samples come from a "fake" dataset as the class distribution is predetermined instead counted from instances. We put the assumption and implementation details of removing shift sampling in Appendix B.I
We perform ablation experiments CIFAR-LT without or with removing shift sampling inves- tigate the effect of CCD shift on vanilla ERM (Vapnik 1991> and some representative re-balancing methods: DRW Cao al. (201)), CRT (Kang et al.) 2019) and PC softmax (Menon et al. 2020 Hong et al. 2021). The main results and analysis are provided in the following subsections. We put
experimental details and more results Appendix B
3.3 DISCOVERING THE UPPER BOUND OF LONG-TAIL RECOGNITION
Figure and Figure (in Appendix) quantitatively show how much CCD shift affects the perfor- mance in long-tailed recognition. The accuracy curves of all methods (vanilla ERM and ra-balancing methods) exhibit sigmificant gaps in performance with or without removing CCD shifts. And the gaps mainly happen for tail classes, indicating that the CCD estimations of tail classes are more reliable. The results quantitatively verify that shift of CCD cannot be ignored and is the key factor to limit the performance in long-tail recognition.
Note that experiments with removing shift give an upper bound to the performance these methods, as the empirical CCD from balanced datasets is not available when training on long-tailed datasets. The results suggest that it is promising and reasonable to improve generalization under long-tail distribution from the perspective of enhancing or against the unreliable empirical distribution.
3.4 ANALYZING RE-BALANCING METHODS FROM THE PERSPECTIVE OF CCD SIIFT
Decoupling method avoids more severe CCD shift. An interesting phenomenon in decoupling methods is imbalanced features are better than re- bebalanced features (Kang al. 2019 Zhou et al. 2020). We explain that re- re-balancing in feature learning exacerbates unreliable empirical distributions because it assigns higher weights unreliable empirical distributions of tail classes expectations.
5

---

Under review as a conference paper at ICLR 2023
As result, simply re- re-balancing harms feature representation since the influence of unreliable pirical distribution is even strengthened.
Table 1 quantifies the effect of CCD shift on re- re-balancing method CRT (Kang al. 2019). Here, "feature re- re-balance" and "classifier adjust" indicate whether and what re-balancing method is ap- plied in feature learning and classidier adjustment, respectively. It shows re-balanced features worse than uniformly learned imbalanced features. But this phenomenon does not appear after re- moving shift of CCD, the re-balanced feature leads to better performance than the imbalanced feature. Hence, we have evidence that two-stage methods succeed partially because they avoid ag- gravating CCD shifts in feature learning. However, re-balancing in classifier adjustment may still suffer from CCD shift, leaving space to be improved, as do in (Zhang et al. [2021a) Wang et al. 2021.
Sub-optimality of Fisher-consistent parameter in post-hoe adjustment. Recall that post-hoc logit adjust- ment method adjusts logits to fo(x), +logg( ((;))) The parameter v is proved Fisher-consistent, however, it is sub-optimal in experiments (Menon et al. 2020). We demystify this contradiction from the empirical study of shows logit adjustment gets best per- formance with much bigger than We explain that rare instances from tail classes cannot provide enough infor- mation so the model regards them as "out of distribution" examples and gives low logits their class (Wang eet al. 202T). Therefore more adjustment to tail classes, L.e. T 1, gets better performance. The contradic- tion comes from the facts that logit adjustment assumes (aest(( y) (Menon et al., 2020 Hong et al. 2021) but the shift of P(x really happens. As shown in Figure 2 with removing shift, r = becomes optimal. Based above analysis, after rectifying biased logit ad-
Rinetune
sccurscy
CCD shift.
62
DoSe
remove shift
0.6
0.8
1.0
1.2
1.4
1.6
18
2.0
Figure 2: Accury on CIFARIO-LT with varying post-hoc adjustment. After removing shift t = gets optimal per- formance while it sub-optimal with shift of P(rly).
Ptrain (xl (olyy)
estimation on Ply(z), further improvement
justment lies in hardly recognizable samples due to CCD shift. Hence, it may be more reasonable to adjust logits by instance instead by class to compensate for hard samples.
4 DISTRIBUTIONALLY ROBUST AUGMENTATION AGAINST CCD SHIFT
We have seen that shift of CCDs limits the performance of re-balancing methods. For unknown real distribution, minimization of the balanced risk is unobtainable even for Fisher-consisten methods (Grreth et al. 2013). To this end, propose Distributionally Robust Augmentation (DRA) to learn a model more robust to the unreliable distribution estimation.
4.1 CONVERTING CLASS-AWARD ROBUSTNESS TO MIN-MA MAX OPTIMIZATION
With empirical distribution and a loss function, the balanced risk is approximated as:
1 Σ Erx.arclb (lo Rbal Σ
(6)
JE(L]
As have shown, empirical CCDs unreliable estimate real CCDs. Instead of minimizing the risk on empirical distribution, we utilize DRO (Kuhn et al.) 2019) to obtain more robust model against unknown CCD shift. In particular, we aim at minimizing the distributionally robust risk:
RDRO = Σ sup Ep, (o) JELL] PEP,
We choose DRW to get re- re-balanced features instead of RW (direct re- weighting)) for RW gets the same result whether with removing shift sampling not: it harms first-stage learning. Related explanation could be found in Appendix B.

---

Under review as a conference paper at ICLR 2023
The loop:
Vade( 36) An ((2,, Bch (x))
① loops
the last exampple
loop
k-th.... loops
3i "sweet pepper"
forward
la
outen-opiimization
inner-ptiinization
Figure 3: An illustration DRA. Augmentation examples 20g generated from original data æ; in M inner loops (with two choices of the last example or a sequence of examples) are used as new training instances 1r y/) train robust model against potential distribution shift. The order numbers indicate the process of DRA in an overall iteration.
in which P, is a probability measure on X x L and P, = (P.:W.(P, Wasserstein distance induced by cost function cl (1 V1 (22. | T2 20. 1 (x Different from previous methods (Sinha et al. 2017 Shafieezadeh Abadehhhhbbhhhhhh et al. 2019). we set radii of uncertainty sets P, dependent on classes to adapt to imbalanced instance numbers, which implies different reliability by Proposition [ Specifically, we set the radii decreased with the number of instances, i.e.r, Sspacity(Frognee et al. 2021 Zhang et al. 2020a).
<ryh. We
rt. make the model robust enough without ver-oocupatioo of model
7 defined on of distributions intractable, we convert it to a min-man min-max opti- mization problem on the training set, as presented in Theorem 4 (in the Appendix).
Since RDRO
RDRO inf
E Arry + E sup JELL]
lo( 3z) X, c(2, (x,))))). (EXx_.))Xx((
by 2020EE
Although the gradient of Xj can be computed by automatic differentiation, it brings another loop. Considering the radii of uncertainty sets need tuning as well, we relax X; (8) to simplify the optimization procedure without more hyperparameters. Then our objective changes from 8 to:
F(O) - Σ
sup Esg {1(x,1)3 - - WW(P,, PNeW)) jE(L] P,
(9)
1
Σ
sup (xxx1,
1o Vz) - X3 c(2, ()).
(10)
L
JEIL]
The final objective (10) actually equals to the Lagrange penalty p for the original problem 7 as in the above equation. Besides, generalization bound in Theorem 7 with mild conditions shows the risk on balanced label distribution with real CCD partially bounded by the objective (10). meaning
that our method provides principled solution to the CCD shift.
4.2 LEARNING WITH A SEQUENCE AUGMENTED
As illustrated in Figure 3 the learning process of DRA involves inner-optimization and outer- optimization, where the former generates augmentation examples from the original data, and the latter uses these augmented examples to train a robust model. In this way minimize the con- verted robustness objective (()) by DRA.
More specifically, as the orange curve in Figure 3 indicates, we can directly solve the inner opti- ( (Zg, X, c(2, (x. y)) by gradient descent. Here lø( Vv) can be seen as mining harder examples for the current model, with c(2, (xc 3)) constraining examples not too far away from the original instance and A, determining the magnitude of the constraint. There are different ways to augment examples during inner optimization. One can rely on Algorithm| P by just replacing À by our Aj. which only uses the last point augmentation example, annotated
mization

---

Under review as a conference paper at ICLR 2023
as "the last example" in Figure 3 This strategy causes unstable optimization and bad performance when A are relatively small, as shown in Figure To overcome these limitations, we propuse DRA Algorithm 11 which uses sequence of examples, i.e. the last == M k points, from the loops of inner-optimization, annotated as "a squence of examples" in Figure Empirically, method achieves better and stable performance small multipliers in Figure
Theoretically, we can prove a sequence of examples can make the optimization more stable as s decreases the following convergency bound under common conditions of Sinha al. (0117):
1-(C))'
B < FE
C2 + +OL Nhateh? T
(11)
8Nhateh
where C1 <1, C2 are constants and Naatch is the batch size. The bound is more formally presented in Theorem [ More theoretical analysis (generalization and optimality guarantees) and implemen- tation details of DRA could be seen in Appendix A C.2
5 EXPERIMENTS
We conduct experiments using DRA together with re-halancing methods e.g. delayed re-balancing, decoupling and logit adjustment on various long-tailed datasets. We also analyze the training cost of DRA and make comparison with other data augmentation methods. More complete experimental details and results are provided Appendix C
5.1 EXPERIMENTAL SETUP
Comparative methods. We compare our methods with (I) re-balancing methods including
Decou-
pling (Kang et al. 2019), Logit Adjustmen(MMenon et al. 2020), PC softmax (ong et al.
2021
DRS/DRW et al. 2019); ; (II) learned data augmentation including M2m (imm et al.
2020
GhT et al. 2022): (III) other related methods including LADE(Hong al.) 2021]. 1p.
DRO-
(Saaall & Chechik 2021). IB(Park et al. 2021). MFW (Ye et al. 2021). CDT Ye et al.)
2020).
Datasets.
We conduct experiments four real-world datasets including CIFAR 10F LT, Tiny-ImageNer-LI. (Cao et al. (2019) and ImageNet-LT (Liu et al., (201)). We set the imbalance evaluate the perfor-
ratio of CIFAR-LT and Siny-ImageNet-Le to 100. For all the benchmarks,
mance by top-1 accuracy on the validation set as previous works (Hong et al. 2021 Cao et al. 2019
Menon al. 2020).
5.2
MAIN RESULTS
Table shows DRA improves all baseline re-balancing methods on three benchmarks in long- tailed recognition, e.g. DRA improves logit adjustment by 0.68% and 0.54% on CIFARIO-LT and CIFAR100-LT respectively. Note that LADE, also based on logit adjustment, improves by 0.22% and 0.22% respectively. Our DRA even marginally outperforms GIT and M2m which use learned augmentation but need additional model (e.g. a GAN (Goodfellow et al. (0144) and training process apart from end-to-end training. We conduct more detailed comparison with these data augmenta- tion methods in ablation study.
Table 7 (in Appendix) shows that DRA is also effective the large-scale dataset ImageNet-LT. Sim- ilarly, DRA shows considerable improvement on belaalancing methods, building stronger baselines for long-tailed recognition. Moreover, the improvement gained from DRA is not only from few-shot but also from many-shot and medium-shie shot, which verifies distributionally robustness could enhance the generalization of head classes well.
5.3 ABLATION STUDY
Class-aware uncertainty set. As explained in Appendix C.2, set class-aware multiplier as X5 normalize NN,") 8 + 5 with hyperparameters 3 and Through I we assign different robustness to classes,3 = 0 gives every class the same robustness and DRA reduces to WRM. With different values of 3 and S, we can reveal the effects of class-aware radius and validate our hypothesis.
8

---

Under review as a conference paper at ICLR 2023
Table Comparison of top-1 accuracy (%) on different benchmarks. denotes results not reported, i means results from original papers, & means reproduced results from official codes, means re- quiring more training epochs. Best results are in bold and small red font denotes performance gain.
Methad
CIFARIO-IT
COFAR100-LT
Tiny-ImageNet-II1
ERM
73.24
40.28
34.20
CE-DRS
77.21
44.16
36.02
LDAM-DRW
78.67
45.05
37.43
LDAM-DRS
78.71
45.12
37.66
softmax
79.58
44.34
Logit Adjustment
79.65
45.17
LDAM-M2mf
79.1
43.4
CE-DRS-GIT
77.35t
43.021
17.43
LDAM-DRS-GIT
79.36
43.65t
21.991
MFW"
78.5
44.7
35.4
CDT²
79.4
44.3
37.9
IBB
78.86
43.18
40.40
LADE;
79.87
45.39
DRO-LT":
79.1
44.2
CE-DRA
75.08+1.88
41.37+1.09
3
CE-DRS-DRA
78.89+1.68
44.87+0.71
18.08
LDAM-DRS-DRA
79.75+1.04
45.54.0.42
39.07+1.41
PC softmax-DRA
80.39+0.81
44.84+0.50
Logit Adjustment-DRA
80.33+0.68
45.71+0.54
78.8
The number of augmentation examples. We perform experiments to verify the effect of us- ing sequence of points as augmentation exam- ples DRA to calculate outer gradient. Specif- ically, we set & in DRA to M/2 M 1 and obtain comparative results.
WRMI--G kusb-11
02-055
77.6
Figure 4 shows the effectiveness of DRA in two innovations of class-aware uncertainty set and the number of augmentation examples. As the bound IIT indicates, sequence of augmenta- tion examples is more stable than the last point. Class-aware radius exhibits significant superi- ority while class-consistent radius (in WRM) only slightly improves the performance.
Figure 4: Accuracy CIFAR10-LT with respect to different choices of hyperparameters S, and in inner optimization
More ablation study. We put more ablation study Appendix C.4 including comparison with augmentation methods, training cost, confidence calibration, visualization and discussion.
6 CONCLUSION
In this paper, we discover empirical evidence that unreliable distribution estimation in long-tailed recognition, exhibiting shift of P(zly), is key factor to limit performance even with re-balancing methods. It suggests that only regarding long-tailed learning as label distribution shift problem not reasonable enough. By our proposed DRA, we show training model robust to potential distribution shift is effective to improve long-tailed recognition with theoretical and empirical guarantees. Space to improve e.g. more precise uncertain set for potential shift is considered possible. Overall, some degree our work sheds light on the bottleneck long-tailed recognition and encourages further solutions under other settings from the perspective of unreliable empirical estimation.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
For our empirical study experiments, Appendix B.1 describes the implementation details such as the probability setting of removing shift sampler, model architecture, and hyperparameters. In our main comparative experiments, for methods used jointly with DRA, we implement them based on official codes and keep default hyperparameters for their modules. For other methods, we use results repro- duced from publicly available official codes or numbers reported in the original papers. Appendix providees detailed experiment settings such as model architecture, optimization hyperparameters, and the implementation of DRA.The pseudocode of DRA is in Algorithm [ Besides, we put all proof's of propositions and theorems presented in this paper in Appendix
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk min- imization games. In International Conference on Machine Learning, pp. 145-155, PMLR, 2020.
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, (4-5):155-196, 1993.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-an aware margin loss. Advances neural information processing systems, 32, 2019.
Hsin-Ping Chou, Shih-Chieh Chang, Jua-Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference on Computer Vision, pp. 95-110. Springer, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class- blass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVE conference on Computer Vision and Puttern Recognition, pp. 9268-9277, 2019.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595 2010.
Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields. 162(3):707-738, 2015.
Charlie Frogner, Sebastian Claici, Edward Chien, and Justin Solomon. Incorporating unlabeled data into distributionally robust learning. Journal of Machine Learning Research, 22(56): 2021.
James Gareth, Witten Daniela, Hastie Trevor, and Tibshirani Robert. An introduction to statistical learning: with applications in R. Spinger, 2013.
Saurabh Garg. Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
Ian J Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Xiao Gu, Yao Guo, Zeju Li, Jianing Qiu, Qi Dou, Yuxuan Liu, Benny Lo, and Guang-Zhong Yang. Tackling tailed category distribution under domain shifts. arXiv preprint arXiv:2207.101:: 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
J Henry Hinnefeld, Peter Cooman, Nat Mammo, and Rupert Deese. Evaluating fairness metrics in the presence of dataset bias. arXiv preprint arXiv: /eXXv:/000.09000 8809.09245, 2018.
10

---

Under review as a conference paper at ICLR 2023
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang, Dis- entangling label distribution for long-taie onigtailedd visual recognition. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 6626-6636, 2021.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 5375-5384, 2016.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 13896-13905, 2020.
Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Slafieezadeh-Abadee Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pp. 130-166. Informs. 2019.
Patrice Latinne, Marco Saerens, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities may significantly improve classification accuracy: evidence from multi-class problem in remote sensing. In ICML, volume 1, pp. 298-305, 2001.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng, Metasaug: Meta semantic augmentation for long-tailed visual recognition. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. $212-52211 2021.
Fengming Lin, Xiaolei Fang, and Zheming Gao. Distributionally robust optimization: A review theory and applications. Numerical Algebra, Control & Optimization, 12(1):159, 2022.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2980-2988, 2017.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Bo Liu, Haoxiang Li, Hao Kang, Gang Hua, and Nuno Vasconcelos. Gistnet: a geometric struc- ture transfer network for long-tailed recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8209-82118 8218,, 2021.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 2970-2979, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Loig-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. Advances in neural information processing systems, 29, 2016.
Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998.
11

---

Under review as a conference paper at ICLR 2023
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balane loss for im- balanced visual classification. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 735-744, 2021. 2021.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Bal- anced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33:4175 4186, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)2211225 2015.
Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9495-9504, 2021.
Soroosh Shafierzadeh-Abadehh Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1-68, 2019.
Alexander Shapiro. On duality theory of conic linear problems. In Semi-infintee programming. pp- 135-165. Springer, 2001.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571, 2017
Amos Storkey. When training and test sets are different: characterizing learning transfer. Dataset shift in machine learning, 30:3-28, 2009.
Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. Invariant feature learning for generalized long-tailed classification. arXiv preprint arXhv:2207.09504, 2022.
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild.
arXiv preprint
1709001400 2017.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural informa- fion processing systems, 1991.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for long-tailed visual recognition. arXiv preprint arXiv:21 2021.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances in Neural Information Processing Systems, 32, 2019.
Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long- tailed distribution. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 8659-8668, 2021.
Zhengzhuo Xu, Zenghao Chai, and Chun Yuan. Towards calibrated model for long-tailed visual recognition from prior perspective. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// openreview wwwpeeviewwweeeteeeee wwwforummidiiuull... id=vvzAfN Boo
Han-Jia Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001.0138) 2020.
Han-Jia Ye, De-Chuan Zhan, and Wei-Lun Chao. Procrustean training for imbalanced deep learn- ing. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 92-102, 2021.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer leam- ing for face recognition with under- redreseeede data. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pp. 5704-5713, 2019.
12

---

Under review as a conference paper at ICLR 2023
Shiran Zada, Itay Benou, and Michal Irani. Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images. In International Conference on Machine Learning, pp. 25817-25833. PMLR, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. arXiv preprint arXiv:0010.01766, 2020a.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint arXiv:2010. 12330 2020b.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming Hc, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 2361-2370, 2021a.
Yifan Zhang. Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-taile tailed learning: A survey. arXiv preprint arXiv:2 10.0459,, 2021b.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
Allan Zhou, Fahim Tajwar, Alexander Robey, Tom Knowles, George Pappas, Hamed Hassani, and Chelsea Finn. Do deep networks transfer invariances across classes? In International Confer- ence on Learning Representations, 2022. URL httpt::t///......t.. /wo//opeeevvveeeeeeeeee ew. net/ forum? iid=
httttps://pperevviiee
En7i r5rR0q.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF confer- ence on Computer Vision and Pattern Recognition, pp. 9719-9728, 2020.
A PROOFS AND MORE THEORETICAL ANALYSIS
Lemma R Theorem) (Villani 2009 5.16) For Wassernstein distance We between two distributions Q and Q', Lip(f) = sup,,,2
[()) ~2
it admits
Jemm (()) (dE).
We(Q
sup Lip(0)<1 m
((E)((EE)
Lemma (Measure concentration) (Fournier & Guillin 2015 Theorem 2) For a probability dis-
tribution P IR"" and PN is the empirical distribution of N instances i.idd sampled from P, A
Ep(zl) [||=||] < oo for some a > 0, then 3c1, C2 depending on A, a, (P, Px) < 1-11 holds.
[ogles/m) 1/m Nf 2 log(ci/n)
C2N
E(n) =
log(ta/1) c2N
1//1 ifN log(es/n)
C2
A.1 PROOF OF PROPOSITON 1
Er(at)) w)} Epwcn
= L (Ux, w)/L) ((r
SL. (T.:W))( (*ly))
The last equality from Lemma the result.
2 And directly substituting nj Lemma 3 by
4 par pm mm
gets
13

---

Under review as a conference paper at ICLR 2023
A.2 PROOF OF THEOREM 4
Theorem 4 For a continuous loss 1: : O (A' [L]) - R. we have
RDRO-
inf 2220,6
(Agry E
sup
10(x2.3)) Mg c(2, (1,))))).
SELL]
(12)
Considering Lagrange penalty with nonnegative multipliers {>g (1JJEEl)) we have the reformulation:
F(0) = Σ
sup Eg ((xx)))
A.W(P,, PN),,)
(13)
jE(M]
P
E
sup
10(22. - X3 c(2, (),)))
(14)
L
JELLI
Our proof generalizes the proof in [Sinha al. (2077) which is special case X C of ours.
Using the duality firstly, we have
Σ sup Ep, [o(x,))]
Rono L
jELL PEEp
sup inf PEP, X,20
1 T.
Σ jE[L]
L v)] -Ay (rler(olvs)) Agr3}
inf (Ayry + sup Pyep,
Σ jE(2]
[Eg [o(,,))) - W. (w,ur(
2,20
T.
The last equality is based strong duality which will be explained later. All we need to prove
sup PEED
1
Σ (E 1o(.
Wx,
L
JE[L]]
sup 1exx11
40(2,, - 2s c(2, (z, 8))]
JE(L]
Actually, we have
1 sup L PæEP; jE(L]
Σ
(op,vo())) - W,W,(P,, (N.1((())))))
1 Σ
/ (((),))) M(d(1,
sup L PyeP; .E(L))
1o(5.8)
>
inf Per(t).P...ttell))
sup P
T.
Σ
1
Σ
/
(o(E.v) (((
M((dE,)),
jE(T]]
sup MEn
1
Σ
sup ls Uz) Afc((EE)) vo), M(dEi,dEb) ==<,,,,=)xxx11
SE(L)
Σ ERM.r JE(A
lo 1o(x..))) yz) X3 c(z, (x,
sup =x(x....
The first equality is from the definition Wasserstein distance and the last equality from sup==(rx.::exxx1 1o Uz) - Xÿ <(2,E2) is independent of &1. We now explain that the in- equality is actually an equality. For every i, we find 2 (.%, ys) for every > 0, E N
14

---

Under review as a conference paper at ICLR 2023
satisfying
(0 vi) - Xj c(2,, (2), 81)) 2 sup
10 U2) - Mg c(2, (2,, -
==
if
sup 10 2 Mz) >3 clz, (e), < <00
(z) >3 c(2,, (r). 2 k
if
sup
(x V2) Aj c(2, (2i, (1);
Let E2) = - EIE(N] 5(2), (2t, 9:)), and for arbitrariness e or k; the equality established.
Finally, we prove the strong duality. Let P3 = PNETo (xlw), and W.(P,, PNET. (xlly) = 0. Therefore, {0...0} E Rt is the inner point of the set {b E R2yb, =
1
Σ
k z; (dE)), Q is probability measure on X), satisfying the slater's condition of
9. W
a standard conic programming duality result (Shapiro 2001 3.4), thus the strong duality estab- lished.
A.3 PROOF OF THEOREM 5
Theorem Under similar conditions to WRM (Assumption)) 223. let T be the number of iterations in outer optimization, Ar F(0o) nE(Ty F(O,), Nbatch the batchsize, with stepsize of outer
optimization a mint 1 NeLwL)) and inner stepsize Ciineer x : Lo = Log + Lecken 27.5 To
x-beee
with Loz, Loo; Tz20, Tozz, depending on l. For Algorithm 7 let M k is the number of exam- ples, À = minje(L) = max max(TT02.2 Loo, Tizo, Tozz]. then for {0,) in the outer optimization, we have
(1 A-La))
6Loz²
Ar
Σ & NEM
c + 4
(15)
T
&Nbatch
A-L=s
A- Luz
Nbateh
T
Actually, in the inequality (1I). considering that X is small but concavity is established (A, Lgz), we assume Ag-L.z & < 1, then the first term of the bound reduces with the increase of the number of examples thus the overall optimization becomes more stable. DRA provides principled solution to CCD shift as long as real CCDs are included by the candidates we consider.
With similar mild assumptions as WRM (Sinha al. 2017) below, one can prove that our Distribu- tionally Robust Augmentation Algorithm| Jenjoys a guarantee of convergence. It finds a e-stationary point of model parameters 0 with ( iterations (as fast SGD (Amari 1993
Assumption The loss 1 0 Z(= X [T]]
R satisfies the smoothness conditions:
2))(0,2) - Val(o", =))1_ - Lee ||0 11 v.((0, ve((0, 2') |1. < Les || 2²
Vo)(O,z) 2) Vol(O, 2 }|| < Laz 1|= =1 v=((0,=) V21(0', .²) < L20 10 0-
Il-Il. is the dual norm of the norm ||-I1 (as our settings, 11-112/
We also borrow lemma from WRM follows:
Lemma 6 Sinha et al.
2017 Lemma 1) Let f 0: 2 + R be differemtiable and 1-strongly con-
cave in z, define f(0) supzez [00,2). Let go (0,z) Vof(0, 2) and (= 2) V.f 2) and f
satisfies Assumption
7 with replacing I with f. Then f is differentiable and Vol Vol = (0))
where 2 20)) ssup f(0,22 2). And we have
Lz8 1|0 02
(0,) - =2((1)) S
17
15

---

Under review as a conference paper at ICLR 2023
and
V V/ s (Leo + 120k0z) 110 0211
Under Assumption 1 the primal of inner optimization 1(0, 2) ),c(2, 20) is (A3 Les - --strongly concave, so that 2j.0 arg sup- supp-(0,2 2) A,c(z, zo) is well-defined and satisfies the condition of
feehe. Lemma 6 Thus, we have Ve supsez 1(0, =) Ayc(z, 20) Volg(=j,o) and F() ,,-
Lipschitz. Then we make an assumption on the variance of the gradient in the outer-ppimization, which is common condition when analyzing the convergence SGD.
Assumption For any sampled training set {(z,, y:) bi)]EE(N holds
E
VoF(O)] S 0
where 2 = arg la(z) Ajc(z, (0i, v:)).
With the preparation above, we begin to prove Theorem 5 Ayo(z, (z). In Algorithm the gradient we use to update is actually
Let hy,0, (xx, ((0,2)
Vohy, (x,
(x. :.):: (2.,
9t
Nbatch VE[Noneen]
rE[s)
Since the gradient of F(0) is -mmoth,,
Lg 11++11 - -
(10+1)<<(0)) (VF().), 0c+1 - 01)
=F(0.) a(1 Loa )[VF(())]1 + a(1
LØa - (VF((),, -9.)
Lga²
+ 2
[IF((.) g
We need to estimate F(O,) 9: in the equation above. Let
gi = Nbatch
100,, (0))))
(E(Naatec]
With the help of Se. we can estimate F(O,) Se more easily. The difference between It and de comes from we use a series of points to compute the gradient to and the inner optimization cannot obtain optimal solution precisely. The difference between (F)),) and of is from just sampling a batch from all training instances.
We compute the difference between It and gi 5, go ge, then we have
||
Vol (x ( .v)) Vol(Oe, 2
Nbatch
se iENaatch
rE(s]
Σ 22 (x (.) rE(s)
Nbatch
ie Naatoh
Assumption The inner-optimization just reaches the linear convergence rate Nesterov 7998) of gradient descent in strongly-convex optimization i.e. [| .yr) Z <
X- ( - 20,1l 5.
We make this assumption under the motivation that the last example is enough when the convergence much faster than the convergence bound that the strongly-conve condition gives. However when the convergence of omer-optimizzatin is slow ie. just reaches the convergence bound, using a sequence of examples gain benefits.
16

---

Under review as a conference paper at ICLR 2023
We have ||< 16) = 11 < Au, Aw,- With another assumption that inner-optimization
reaches the linear convergence rate of gradient descent on strongly concave optimization (Nesterov) 1998), as the inner optimization at least X - concave, we have
[1 ktr v.) 20 11 - (1
A Lozz
1 Ay,
Luz E.
G
1-1- kaz)" Aho=²
As result, ||ōel12 &Nbatch
--g x X-Lezz
Substituing Si with 8,, we have
F(0++1) <F(0,) 7 IVF((0)12 2 _ 116:12
+ - L VF((,) - 9:) + Lo (11bel 1 - 11
For Lemma 6 9i unbiased estimation VF(),) so Elg (P(),) (7(00))0 0.]] O. Using Assumption
to control the invariance of the estimation and taking expectations of the above formula, we have
0 E[F(O++1) F(O.)) 2 -2 =(IVF
Loa 0² + 3 1
1- (1 2-6 )z
4L.a 2
Nbatch
sNbatch
A-hezz G
1
where we use a < 2h. to get LL S a
Summing by L, we get
T
AF + aT
2 Looa² +
6
(1
A-boa )²
4L
2
T
Nboteh
sNNatthh
A L
Substituting with the stepsize a, the conclusion obtained.
A.4 GENELIZATION BOUND AND OPTIMALITY OF AUGMENTATION EXAMPLES
In addition, give bound the estimated balanced risk real distribution and the optimality of examples generated by DRA under mild conditions.
Theorem (Genelization bound) Assuming I O (X (L) R continious and for every
class j. let the real flass-conditional distribution ((xly;) satisfies 3a > 0, (lee) oo, then 3c1, C2 only depending on A, a, for any multipier (}}bEEL) with probability 1 - it holds
1 (My maxi( log(cs/n) 1/m 1ogle1/m) L
Roat S
Σ
N2N₁
c2Nj
JELL)
sup =.
1(., Uz) - >3 c(2, (x, y))}},
in which Roal,1 Ejenn (1(C,, is the balanced risk on test distribution Ptest 1 Ejeto P(xly,) estimated by loss function 1(x y).
Remark The above bound shows new viewpoint beyond existing theoritical results of long-tailed recogniton in the community. We make some remarks for the bound as follows:
1. It is more flexible and tight than the results of prior DRO works (Sinha et al. 2017 Kuhn let al. (2019). It adjusts class-wise robustness by >5 for euch class j. while prior work only admits a special case of X; = for all.) j E [T.).
2. It is closer to practical training compared to the Fisher-consistency result by logit adjust- ment (Menon et al.) 2020). Fisher-come consisseen theory from Theorem in Menon al. (2020) only states that a Bayes- corres-pptmall classifier would be obtained by the minimization of logit adjustment loss under balanced label distribution with real CCDs.
17

---

Under review as a conference paper at ICLR 2023
3. It is longer dependent on the capacity of the hypothesis class, which is extremely large for modern neural networks, while the bound in Theorem of Caa et al. (2019) is.
This bound shows with high probability the balanced risk on real distribution could be bounded by our objective (10) plus constant depending on the multiplier > and the number of instances from each class, which serves as generalization gap. decreasing with the number of instances Nj. indicates Xy controls a trade-off between robustness and performance. If we do not enforce any robustness i.e. the multipiers Aj are extremely large, DRA actually equals to ERM. In this situation, large X leads to considerable generalization gap, especially for the tail classes. Hence, we conclude that models robust to CCD shift benefit long-tailed recognition. Conversely, making X; extremely small near zero, it admits a trivial bound meaning the model refuses to make predictions, which is well-known
pvver-pessimism" in DRO (Frogner al. 2021
Proposition 8 (Optimality of augmentation examples) Assuming 1 Q x (X [T.]) - R. is con- tinuous. Tx((x,)) y)) == arg max;= (..))==xx 1(zz. U2) - >y- c(z, (z. u)) the optimal solu- tion of inner optimization. M is the set of probability measure on X [L] and (r),r(z)))) NoT. ((en)=s) ((T)(( u.)) is the empirical distribution consisting of them. Then for any mul- tipier {Ai}jE(L), have
P ( arg
Ep, PJEM
v)) - ((),,,)) ~2 (x,e((s))))
which indicates the empirical distribution consisting ideal optimal solutions of inner optimization of class the optimal perturbed distribution of the Lagrange penalty problem.
A.5 PROOF OF THEOREM 7
Using Theorem 4 for any multiplier (AybEELL) we have
1
Σ sup F, (1xx,1)) < Σ Ajry
L
jE(Z] P₁EP,
jE(L]
+ EE (xxx,.)))x((1]
sup
(E,, Ve) Ag (2,
in which ry is the radius of P..
And with assumption, P(rlyj) satisfies conditions of Lemma
So let Tj
max mrax(( ((eolly//)) 1/m, Ioogee /m) 11/0) we have P(xlU;) E Py with the probability of 77. Thus,
establishes
Σ JELL]
1 Σ sup Ep, ((x,)))) < 1
L
JELL] PyEp,
and we get the conclusion.
A.6 PROOF OF PROPOSITION
sup Ep, 122 -11 - (,, PN.LT PyEM
=Ep (+l))
sup
Nÿ c(2, (a,)))
-Epy - >2 {( Baaiiis)PPPP...P (N.Lr(
Σ (TT;((Z,,)))) (z.,
< Ep w)]
The first equality is from Theorem 4 and the second is from the definition of P (* The inequality is from the definition of Wasserstein distance. For the inequality in the opposite direction trivial, establish the conclusion.
18

---

Under review as a conference paper at ICLR 2023
B DETAILED EMPIRICAL STUDY ON CCD SHIFT
B.I EXPERIMENTS SETTING
Assumption of removing shift. To conduct experimental comparisons on CCD, we propose remov- ing shift sampling in Section 3.2 An ideal situation for comparision on CCD is to obtain real CCD P(xly) and keep imbalanced label distribution P(y). In fact, due to no access to the real P(x| y), we make the empirical CCD P PN.bal (xly) serve as oracle to substitute real CCD P(xly). At the same time, we keep imbalance by using the same label distribution P Pt.(y) as long-tailed datasets so that the label distribution is imbalanced in every batch and it is in the whole training. In this way, more unique instances are seen by the model but the number of samples from each class, i.e. label distribution, keeps unchanged in all batches during training. So under our assumption, we make a fair ablation study on CCD with proper control of other factors. It is possible that bat (xll) can- not estimate real CCD well, but as long as the oracle brings improvement in performance, we can blame CCD shift correctly. The counterexample appears in extreme cases when the oracle does not make improvement. that case, we cannot tell whether the CCD shift does not appear thus should not be blamed or our oracle is too weak to identify the CCD shift.
Implementation of removing shift sampling. We do removing shift sampling on original balanced dataset CIFAR 10/100 (Krizhevsky) (009). We use a dataloader with a class-imbalanced sampler which samples with the same class probability PLT (y) as the long-tailed dataset and samples uni- formly within each class. The accumulated probabilities of instances from each class equals the predetermined class frequency and every instance within a class has the same sampling probability. To this end, we implement the sampler that used to generate CIFAR-LT in Cao et al.] (2019). Different from the generation of long-tailed dataset, sample from the whole balanced dataset and get class-imbalanced instances in every batch. In addition, We keep the same amount of data as the long-taile tailed dataset while only change the class-conditional distribution via removing shift in the ablation experiment.
multinomial distribution sampler in the same way as
More implementation details. We train ResNet (He et al. 2016) with batch size 256, optimized by SGD with momentum 0.9 and weight decay 2 x 10 with warm-up scheduler. The learning rate decays by a factor of 0.01 at epochs 160 and 180 with an initial rate 0.2. We evaluate Top-1 accuracy on the original validation set of the datasets, following the common protocol long-tailes recognition (Cao et al. 2019).
Methods ablation study. We select ERM as baseline and DRW, PC softmax (equals to post-hoc logit adjustment) and CRT as representative re-balancing methods.
1. ERM. ERM means empirical risk minimization (Vapnik 1991). We train model on long- tailed datasets with cross-entropy loss without any strategy.
2. Balance. For easy reference, we also train a model under our setting on the original CI- FAR10/100.
3. DRW. We implement DRW (Cao et al. regular way the early phase and re- re-eeight the loss by the inverse of the predetermined class frequencies only in the last phase during training. We typically re- re-weight the loss function starting at 160 epochs.
2019) following the original paper. We train in a
4. RW. RW means directly re- weighting the loss by the inverse of the predetermined class frequencies. We re-weight the loss in the whole training phase. In ablation experiments on decoupling methods, we utilize DRW and RW to obtain re-balanced features and ERM to obtain imbalanced features in the first stage of training.
5. PC softmax. We implement PC softmax following Hong et al. (222 by training the model by ERM for 200 epochs and applying adjustment at testing.
6. CRT. We implement CRT following Kang et al. 2019). We train the whole model for 200 epochs and re-train the classifier for 10 epochs with re-weighting loss. We restart the learning rate when re-training the classifier.
7. LWS. We implement LWS following Kang et al. 2019). We assign weights to classifiers with learnable scale factors i.c W; = f. W,, E [Z]. We train the whole model for 200 epochs and then train factors fa for 10 epochs by re-weighting the loss. We tune the
19

---

Under review as a conference paper at ICLR 2023
ERM
DRW
Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
0.2
base
remove shift
60
70
80
90
100
30
40
50
60
70
100
PC softmax
CRT
1.0
1.0
Accuracy
0.8
0.8
V
0.6
0.0
50
60
70
80
100
30
40
50
60
70
Class index
Class index
Figure 5: Accuracy on CIFARI00-LT without or with removing CCD shift. It shows similar results
that models trained with removing shift sampling get significant performance improvement.
Table 3: Accuracy of different methods on CIFAR-LT without or with removing CCD shift
CIFARI00-LT removing shift
CIFARIO-LT removing shift
base
base
ERM
50.7
40.28
80.37
73.24
DRW
63.97
44.41
89.01
78.04
PC softmax
58.61
44.34
87.90
79.58
CRT
52.82
43.48
87.07
77.72
Balance
66.73
91.53
learning rate on different datasets when learning scale factors as we found is somewhat sensitive.
B.2 MORE RESULTS AND ANALYSIS ON CIFARI00-LT
On a whole, results on CIFAR100 agree with those CIFARIO-LT. In Figure and Table remov- ing shift sampling improves the performance of re-balancing methods and ERM on CIFAR100-LT, which indicates again that shift of CCDs is the key to limit tailed recognition.
As for decoupling methods, fableAssoow re- balanced features outperforms those uniformly trained significantly on two decoupling methods CRT and LWS. It agrees with our explanation of why decoupling methods work: first-stage learning without re-balancing avoids more severe CCD shift while re- re-balancing with removing shift could benefit feature learning.
Results of logit adjustment on 100-R in shown Figure 6 With removing shift sampling is optimal, while without the best is much bigger than I. This result again agrees with our supposition that logit adjustment gets sub-optimal experiments and leaves space to be improved due to CCD shift.
B.3 CONFIDENEE CALIBRATION AND FEATURE DEVIATION UNDER SHIFT OF CCDS
As two recently studied issues in long-tailed recognition, confidence calibration (Guo et al. 2017) and feature deviation (Ye et al. 2020) have drawn an amount attention. We conduct experiments to investigate how the shift of CCDs affects these two issues.
20

---

Under review as a conference paper at ICLR 2023
Table 4: Accuracy of decoupling methods, CRT and LWS, on CIFAR-LT with different features, without or with removing shift. LWS shows similar results CRT: re-balance harms feature learning while benefits feature learning with removing shift.
CIFAR IT0-LT
CIFARIO-LT
feature re-balance
classifier adjust
removing shift
base
removing shift
base
CRT
52.82
43.48
87.07
77.72
DRW
CRT
58.39+5.57
42.36-112
88 88.42+1.35
75.97-1.75
LWS
57.67
44.05
87.55
76.21
DRW
LWS
61.95+4.28
43.63-0.42
88.95+1.4
75.36-n.84
DRW
63.97
44.41
89.01
78.04
RW
CRT LWS
48.1 11 11 4.71
28.48 15.0
79.51 7,56
71.43 6.29
RW
49.03 8.64
29.76 14.29
78.75 8.80
71.78 4.43
RW
49.92
30.48
80.34
72.78
Balance
66.73
91.53
57.5
55.0
accuracy
base remove
1.2
1.6
Figure 6: Accury on CIFAR100-LT with varying 7 in post-hoc adjustment. After removing shift
t gets optimal performance similar to that on CIFARIO-LT.
Table 5: ECE on CIFAR-LT different methods without or with removing shift sampling.
CIFARIO-LT
CIFAR 100-LT
removing shift
base
removing shift
base
ERM
6.4
15.5
15.91
29.75
DRW
2.00
10.77
2.19
22.16
PC softmax
2.55
9.25
2.73
19.43
CRT
4.40
14.54
19.5
26.63
Balance
1.77
3.89
21

---

Under review as a conference paper at ICLR 2023
IPM
DAA
Oethhremwwe
Acce
Accennt
GAB
Gea
Exparer
Figure 7: Reliability diagrams of different models trained on CIFAR1O-LT. "(remove)" means using removing shift sampling and PCE means PC softmax. The gap between model confidence and error probability is reduced by removing shift sampling significantly.
Confidence calibration. Confidence calibration is to make model prediction estimate true correct- ness likelihood well, which is important in many applications. has been proved that the calibration of neural networks is generally bad and long-tailed distributions make neural networks even more miscalibrated and over-confident (Guo et al. 2017 Zhong et al.) (022). Expected Calibration Error (ECE) is widely used to measure the calibration of model. With all N instances devided into B interval bins of equal size by their predictions, ECE is calculated as:
B
|S (ace(S.) - con (S:)).
ECE = Σ
in which S; is the set of instances whose predictions fall into the i-th bin. acel.) and con compute the the accuracy and estimated confidence on S: respectively. As shown in Table [ re- balancing methods improve ECEs except CRT on CIFAR-I00 and removing shift sampling improves the calibration of different models significantly. However, using both re-balancing methods and removing shift sampling is still worse (in ECE) than that on balanced dataset. We suppose the reason lies in that still cannot perfectly model real P(zly) even with our sampling method. Figure 7 shows the reliability diagrams of different models trained on CIFAR I0-LT. With removing shift sampling, the reliability of models is significantly improved.
Table 6: Average feature deviation distance on CIFAR-LT of different methods without or with removing shift sample
CIFARIO-LT
CIFARI00-LT
removing shift
base
removing shift
base
ERM
0.70
1.19
1.96
2.90
DRW
0.693
1.23
2.11
3.18
Balance
0.829
2.10
Feature deviation. Feature deviation phenomenon found long-tailed recognition by recent works (Ye et al., 2020) 2021). That is the average distance of features learned from long-tailed training dataset usually exhibits imbalance between training and test and the distances tail classes are larger than those of head classes evidently. Ye et al. (2020) proposes a feature deviation distance dis(j) to measure the deviation for class j:
R
dis(j:go)
Σ 1=1
mean(Sx((go0zz (f(((()))) - mean( ( (2
R
22

---

Under review as a conference paper at ICLR 2023
in which 1l-112 is L2- norm, Sk means sub- sub-samliing K instances from class j. R is the number of sampling rounds, and go is the feature extractor. And for convenient comparison, we also use a class-wise mean of feature deviation distance, called average feature deviation distance:
dis
Σ dis(j:ge).
(16)
JEIL]
Table 6 shows feature deviation is affected by shift of CCDs removing shift sampling improves this metric on the whole. Surprisingly, the re-balancing method (i.e. DRW) generally intensifies the feature deviation no matter with removing shift or not. Besides, training without re-balancing (i.e. ERM) but with removing shift seemingly even outperforms the balanced result, e.g. 0.70 vs 0.829 on CIFARIO-LT and 1.96 vs 2.10 on CIFARIOO-LT. One may conclude that imbalance does not make an obvious difference on feature deviation. In fact, this is because features of head classes benefit from imbalance and removing shift makes up deviated features of tail classes to some extent. As result, using average feature deviation distance as the measure, imbalance seems not critical for feature deviation.
Ceoaaba festue deviition
Sernett
Pesture decation destare
Feature decstion dalance
PPM
Class Index
Class index
(a) CIFARIO-LT
(b) CIFARI00-LT
Figure 8: Class-wise feature deviation on CIFARIO-LT/CIFAR CAA.LT///IAAARO1L1111 with removing shift sam- pling. Even with removing shift sampling and re-balancing it still exhibits imbalanced feature devi- ation on both CIFARIO-LT and 1OFARI
Figure shows class-wise feature deviation affected by imbalance as well. The feature deviation distances of different classes still exhibit imbalance with removing shift. A significant gap of feature deviation distance of tail classes appears between balanced training and re-balanced training with removing shift. We suppose that is because re- balancing methods still leave space to improve feature deviation, e.g. it has been discovered that the imbalanced logit distribution harms feature learning in the beginning stage of training (Ye et al. 2021
C FULL EXPERIMENT DETAILS AND RESULTS
C.1 EXPERIMENTAL DETAILS
CIFARI00-LT and CIFARIO-LT. We apply SGD with batch size 256 and the base learning rate 0.2 to train ResNet-32 (He et al. 2016) model for 200 epochs, following Hong et al. (2021). We employ the linear warm-up learning rate schedule for the first five epochs and reduce at epochs 160 and 180 by factor of 0.01 and use the same weight decay 0.0002 as previous works (Cao al.
2019 Zhou et al. 2022).
Following Park et al. 2021 we apply SGD with batch size 128 and weight decay 0.0002 to train a ResNet-18 model for 100 epochs. We set the base learning rate to 0.1 and reduce it at epochs 50 and 90 by factor of 0.1.
ImageNet-LT, Following Hong et al. (202I) we apply SGD with batch size 128 and weight decay 0.0005 train ResNet-50 model for 90/180 epochs. We perform cosine learning rate scheme with an initial learning rate of 0.05.
Reproducing baseline results. We combine DRA with various existing methods and compare their results with these baselines methods in long-tale recognization. For the baselines combined with DRA, we implement them based their official codes with default parameters in their modules
23

---

Under review as a conference paper at ICLR 2023
under training setting for fair comparison, such model architecture and optimization hyperpa- rameters. There are a few results disagreeing with original numbers reported in their papers, caused by their different training settings from ours, e.g. get 44.34 top-1 accuracy of PC softmax CIFAR 100-LT while the reported number is 45.3 in Hong et al. (02T), which because they use bigger weight decay parameters and only perform experiments under one random seed. For other baselines, we reproduce the results using publicly available official codes or direct use numbers re- ported in the original papers. For all results we implement reproduce based official code, we run five times with different seeds and report the average.
C.2 IMPLEMENTION OF DRA
Algorithm DRA: augmenting with sequence of examples
Input: {(,, M }. a batch of instances
Input: {Ay)sE(L), the class-awan aware multipliers
Output: Augbatch- = (1, .)))
Noates- (M-
1: Augbatch + 0
2: for -, ,..,,,,,NNtteer do
3: 2! <- Fi, j f- 0, - f- (2c,0.-* ..........
4: while M and 99 > c do
5:
9. V2yy, ((r) U1), (Z Ut)) (0(2), v);
Compute gradient of inner-optimization
6:
2. J+1 + x² ainner 9;
b Update augmentation examples by gradient
7:
if
j 2 l then
8:
Augbatch
(ar4, 9:)))
Save sequence of exampls
9:
j=j+1
10: return Augbatch
> Return augmented instances for outer-optimization
Algorithm WRM (Sinha et al. 2017): augmenting with the last example
Input: {(I,, y1) MMarch }. batch of instances
Input: {A}, the multiplier
Output: Augbatch {x: y:)) Noater Noatox-(MM)) (M-k)
1: Augbatch + {
2: for i I,...,, Nhatch do
3: x/ <- Vi. f- 0, Sº5t <- (2e, 0...0,0.000
4: while M and g e do
5:
g. = V>)c((>), (hc((1/,11)) vi), (I,, v:)) lo lo(2, v:);
Compute gradient inner-optimization
6:
xx ++1 - of Qinner g
b Update augmentation examples by gradient
7:
j j=j+1
8:
Augbatch comcate((Augbatehe (#) vx)))
Save the last exampl
9:
return Augbatch
> Return augmented instances for outer-optimization
The data augmentation process of DRA is summarized in Algorithm Noting that the class-ware multiplier Xy has negative correlation with the radius of uncertainty set positive correlation with the number of instances of class j. thus we set A, = normalize [N S with B, S 2 0 and normalize meaning scale the vector to unit vector. 3 determines robustness difference over classes and S determines overall robustness level. In this way, we could reduce the multipliers of DRA to only two hyperparameters. We set M 10 and Qinner 0.1 in DRA. As for the hyperparameter k, the number of iterations starting to be used to augment examples, we set as half of the whole number of iterations M heuristically the solution of our primal problem. We do not fine-tune to get an optimal number but only select from the above two choices, which, however, gets pretty good result.
M 1, which is equivalent to directly utilizing WRM as
Inspired by (Km et al. 2020; Samuel & Chechik 202I), we apply DRA in the later stage of train- ing since need initial model for DRA generate augmentation examples. Specifically, for
24

---

Under review as a conference paper at ICLR 2023
CIFARIO-LT and CIFARIOO-LT start to use DRA 160 epochs, for Tiny-ImageNet-LI start to use DRA on 90 epochs. For ImageNet-LT we start to use DRA on 80 epochs when training for 90 epochs while start
DRA 150 epochs when training for 180 epochs. And for CRT, we only use DRA in the training of the second stage.
Following Hong al. (2021) Kim al. 2020), tune the hyperparameters: of DRA by grid search on the validation set. WRM uses the following strategy to determine the hyperparameter multiplier: C- Ep (z) [||= while C 0.04 is a constant and (xp<(=>]]=rl)) is the average norm of all instances in training set. However, this strategy is not helpful in determining the hyperparameters 248.39 on CIFARIO-LT while 222.4 on C00ARI00-LT. The two numbers are similar but the optimal S found by our grid search on the two sets disagrees lot. Moreover, scaling the optimal S CIFAR by the proportion of (*)]lllll) does not work well either. How to determine the hyperparameters for DRA more elegantly could be a future research point further improve our method.
S. For example, { [
C.3 RESULTS ON IMAGENET-LT
Table 7: The performances ImageNet-LT. We report accuracy three splits of classes: Many- shot (more than 100), Medium-shot (20-100) and Few-shot (less than 20), following Hong et al. (2021) [Zong et al. 202II. The small red font denotes performance gain. means results from the original paper.
Method
Many
Medium
Few
All
90epochs
ERM
65.1
35.7
6.6
43.1
Decouple-CRT
61.22
47.52
26.41
49.52
Decouple-7-norm?
59.1
46.9
30.7
49.4
Decouple-LWSt
60.2
47.2
30.3
49.9
PC softmax
60.4
46.7
23.8
48.9
Logits Adjustment
60.62
47.33
27.53
49.25
LADE
60.34
47.37
27.82
49.20
Decouple-CRT-DDE
61.63
47.53
30.61
50.25
PC softmax-DRe -DR
60.38
46.81
24.23
49.11
Logits Adjustment-DB -RA
60.63
47.84
27.60
49.49+0.24
180epochs
ERM
66.84
40.89
11.54
46.05
Decouple-CRT
61.59
47.70
30.32
50.3
PC softmax
62.13
49.25
30.51
51.18
Logits Adjustment
62.67
49.96
31.67
$1.90
LADE
62.80
49.76
33.4
52.14
Crrouple-CT--DR..
61.81
49.43
31.57
51.37+1.07
PC softmax-DRA
62.41
49.72
31.89
51.64+0.46
Logits Adjustment-DRA
63.29
48.88
32.00
52.32+0.42
C.4 MORE ABLATION STUDY
C.4.1 COMPARISON WITH DATA AUGMENTATION METHODS
We can regard our DRA as generating examples for data augmentation during model training and so does WRM. Hence, it is natural to ask what is the relationship between DRA and other data augmentation methods long-tailed recognition. In our view, DRA focuses on assigning robustness over classes against unreliable CCD estimated from scare instances, while other data augmentation methods aim to improve the diversity of instances heuristically, e.g. through head-to-tail transfer or instance generation. Moreover, those data augmentation methods neither attend the key issue of CCD shift nor provide theoretical guarantee. In our experiments, we conduct a comparison with two recent works GIT and M2m (Zhou al. 2022 Kim al. 2020 with respect to performance
25

---

Under review as a conference paper at ICLR 2023
and training cost. Besides, for augmentation methods that adopted in balanced learning, suppose they are complementary to DRA. So we also conduct an experiment of combining DRA with representative augmentation method Mixup (Zhang al. 2017).
Performance comparison with learned augmentation methods. GIT and M2m, we conduct experiments under the same settings of GIT and M2m. As the official code GIT and M2m cannot work well with our larger batchsize setting, we keep all settings except changing the batchsize 128 and initial learning rate 0.1.As shown in Table generally, DRA gets comparable performance to M2m and GIT with a small gap (less than 0.2%) with CE-DRS on CIFARIO-LT and outperforms them on all other settings. It is surprising WRM does not get any significant improvement on all baselines, which shows assigning the same robustness to all classes is not appropriate as much robustness is unnecessary for head classes and may cause over-pessimism while little robustness cannot improve generalization for tail classes evidently. In contrast, DRA more theoretically sound and more stable than these heuristic augmentation strategies. We hope this could encourage future methods to solve the shift of P(xly) in long-tailed recognition more effectively.
To make a fair comparison with
Table 8: Comparision of other augmentation methods with DRA. means reproduced result from official code. Best results are marked in bold.
Method CE-DRS
Aug. strategy
CIFARIO-LT
CIFAR100-LT
75.87
41,21
GIT;
77.06
41.86
M2m
77.5
42.2
WRM
75.92
41.31
DRA
77.38
42.61
LDAM-DRS
77.47
42.78
GITt
78.49
43,49
M2mł
78.68
43.12
WRM
77.34
43.11
DRA
78.76
43.53
Training cost of DRA compared with learned augmentation methods. We compare DRA with learned data augmentations e.g. GIT and M2m with respect the training cost Table 9 All the methods are measured on NVidia card (GTX 2080Ti). The inner optimization of DRA increases the training cost exactly as it needs to make the model forward and backward to compute gradients on the input to generate examples. Considering all these methods are only used in the later phase of training c.g. the last 40 epochs all 200 epochs on CIFAR-LT, they are not quite time-conssue cone-conu in the end-to-end training part. However, GIT and M2m need much additional time to train model as preparation e.g. a GAN for GIT and a network classifier trained by ERM for M2m. DRA avoids the additional time and just slightly increases the training time per epoch, where the gap between DRA and GIT or M2m could be ignored considering their additional time cost for preparation. Besides, it seems using sequence of examples i.e. M/2 does not cost more training time than using the last example.
Combination with Mixup. We also explore whether DRA is compatible with existing general data augmentation methods or not. We use the simple, efficient and widely-used Mixup(Zhang et al.) 2017) as an example. Specifically, we only use Mixup to pre-train the model before using DRA. Specifically, on CIFAR-LT, we use Mixup in the early 160 epochs and DRA in the later 40 epochs of 200 epochs in training. We also compare ours with state-of-the-an Mislas Zhong et al. 202 based on Mixup. Table 10 shows DRA is suitable to train with Mixup as DRA even further improves the performance which has already been enhanced by Mixup and outperforms Mislas, showing compatibility between the two methods. It implies that even if data augmentation methods would improve the diversity of data, they are still possibly biased due to unreliable empirical distribution and thus could enjoy "correction" from DRA.
26

---

Under review as a conference paper at ICLR 2023
Table Comparison of training cost of CE-DRS with different augmentations on CIFARI00. The training time is counted in seconds. GIT and M2m need additional time to train another model, e.g. a GAN and model trained by ERM respectively. Whereas, DRA just slightly increases training time.
Aug. strategy
Time per epoch(s)
Additional time(s)
CE-DRS
3.72
GIT
25.84
33710.41
M2m
29.25
669.2
DRA(K M 1)
27.60
DRA(k M/2)
32.26
Table 10: Ablation results on combining Mixup and DRA, $ means reproduced result from official
code. Best results are marked in bold. The small red font denotes performance gain from our DRA.
Method
Aug. strategy
CIFARIO-LT
CIFARIO0-LT
CE-DRS
Mixup
79.70
47.08
Mixup+DRA
80.59+0.89
47.39
PC softmax
Mixup
81.29
46.12
Mixup+DRA
82.41
46.35+0.23
Mislas
82.10
47.19
C.4.2 DRA CONFIDENCE CALIBRATION
Inspired by the discovery in our empirical study that confidence calibration is effected by both imbalance and shift of CCDs, we expect DRA would relieve over-confidence of models. From our experiments, it kind of surprising that DRA not only improves calibration but also improves the well-calibrated models by Mixup most of the time. We use ECE (Guo et al. 2017) as the measure.
In Figure 9 seems just with re-balancing methods model cannot be calibrated well and DRA actually gets smaller ECE these methods, which validates again that confidence calibration is affected by shift CCDs and DRA improves it by making the model more robust to the shift.
Besides, considering the discovery in recent works (Zhong et al. 2021 Xu et al. 2021 that Mixup has significant positive effect on calibration, we conduct experiments to measure ECE with both Mixup and DRA. Figure 10 shows DRA further improves calibration which DRA weakens calibration but
the basis of Mixup most
of time. The only exception is CE-DRS on CIFARIO-LT, boosts its accuracy shown in Table TO It seems with both Mixup and DRA the regularization on logits too strong to obtain good calibration as the model gives confidence even lower than the accuracy instead of usual over-confidence on neural network (Guo al.) 2017). These results also raise an interesting question: Do calibration and accuracy agree in long-tailed recognition? Or more specifically, does good calibration mean a good model in long-taile recognition (Xu al. 2021 Actually, our experiment shows that it is possible to boost performance while weakening calibration.
27

---

Under review as a conference paper at ICLR 2023
CE-015
CE DTA
CE-015
DRC
Gop
GuB
GuB
Gea
3.
33
Lupeze
02
3.2
9.32
600
Ourfocece
CIFARIO-LT
(b) CIFAR100-LT
Figure 9: ECE (%) and reliability diagram on CIFARIO-LTYCIFARR000111 DRA gets better cali- bration performance marginally. PCE means PC softmax.
One
Dra
32
RC5- 2.66
4.48
EC5
802+1.22
Accumzy
Acounacy
Aecumay
323
(a) CIFAR10-IT
(b) CIFAR100-LT
Figure 10: ECE (%) and reliability diagram CIFARIO-LT/CIFAR LT--T with Mixup. DRA even improves the calibration models with Mixup, which have obtained small ECE and been calibrated well. PCE means PC softmax.
28

---

Under review as a conference paper at ICLR 2023
C.4.3 VISUALIZATION OF AND DISCUSSION ON EXAMPLES GENERATED BY DRA
j=1
3
8
10
Figure 11: Examples generated by DRA on CIFAR-LT. Left most column are original examples and j is the number of iterations. DRA tends to add pixel-level transformations to obtain harder examples while some of these transformations agree with semantic information.
29

---

Under review as a conference paper at ICLR 2023
To get more precise understanding, visualize the examples generated DRA, in Figure 11 exhibits that DRA adds pixel-wise transformation to instances in order to generate harder examples as data augmentation. Some of these transformations tend erase conspicuous features in an instance by adding "noise". this way. model is encouraged to extract broader features instead of overfitting features it has learned. For example, DRA seems to add "noise" erase the ladder from "fire engine" in the first row. As ladder is a salient feature for fire engine, and this augmentation makes model attend more recognizable features e.g. water tank and red color. Similarly, in the second row, DRA adds "noise" on the beaks and claws of the bird and encourages the model to take in more features help recognition instead of dependenn on the two features.
In addition, more surprisingly, DRA can catch semantic information while discard nuisance infor- mation by itself. As the fourth row, the yellow sweet peppers happen to be transformed to green while pixels out of the peppers almost kept unchanged. In other words, DRA makes a transforma- tion of color, and the transformed instance still belongs to "sweet pepper" class. In this way, DRA performs transformations that keep the semantics unchanged. As shown in the bottom two rows, DRA could give transformations on the background (nuisance information) and keep semantic in- formation: almost all of the pixel-wise transformations in the last but one row lie in the sky, and the semantic object "road" isn't changed. In the last row, the pixels of sofa are not transformed while the background is changed.
From these results, we infer that seeking harder examples for a robust model sometimes agrees with transformations that keep the semantics of instances. For example, background intensity could serve as examples to obtain distributional robustness under a few situations, and this can be an explana- tion of why GIT improves performance by adding transformations keeping semantics of instances. However, when transformations that keep semantics are not that effective to gain distributionally robustness, DRA tends to seek other examples with obvious features are erased.
From the observation that examples from DRA do not have to keep semantics and seem like in- stances with "noise", we hope DRA could suggest new motivation for data augmentation: making more robust model for unknown distribution shift instead of utilizing information to reduce the shift, e.g. GIT, ISDA (Zhou et al. 2022 Wang et al.] 2019 Li et al.) 2021). This motivation, making model robust to shift, could give hints to explain the rationality of existing data augmentations that
don't have to keep semantics, e.g. adding pure noise to instances as data augmentation (Zada et al,
2022 and those based on Mixup (Zhang et al. 2017 Chou et al. 2020 Zhong et al. 2021
Xu al.
2021
30


Reviews:
[{"review_id": "rD4sKPAQp4", "reviewer": "Reviewer_Lnfc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the Class-Conditional Distribution (CCD) shift issue in long-tailed recognition due to scarce instances, which exhibits a significant discrepancy between the empirical CCDs for training and test data, especially for tail classes. To alleviate the issue, this paper presents a data augmentation approach to generate more virtual samples.", "summary_of_the_review": "This paper studies a relatively new problem in long-tailed learning and a simple strategy is proposed. However, the proposed method seems not novel enough and has several hyperparameters. In addition, experimental show that the advantages of the method is not significant.\n", "strength_and_weaknesses": "# Strength\n1. The studied problem is interesting and not well-addressed in existing literature;\n2. The proposed method is well-motivated and equipped with theoretical analyses;\n3. Extensive experiments show the advantages of the proposed method;\n4. This writing is clear and easy to follow.\n\n# Weakness\n1. Novelty: the proposed Distributionally Robust Augmentation (DRA) approach is borrowed from previous work DRO and no significant improvement has been made. The used min-max optimization is very commonly used in related works;\n2. Empirical improvement is weak. As reported in Table 2 and 7, the proposed DRA does not consistently outperforms existing ones, such as Logit Adjustment, PC softmax, LADE;\n3. Rationale behind the approach: There is a gap between the DRA and CCD shift. Why DRA can remove CCD shift? How accurate can DRA approximate real CCD? Those questions seem not answered in the theorems.\n4. Hyperparameters: the method involves several hyperparameters, e.g., $C, S, \\beta, M, \\alpha_{inner}$ and some of them require a validation set for tuning which is not applicable in some cases. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W_c9IzzhYSJ", "reviewer": "Reviewer_qfH1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied the phenomenon that inconsistent CCD causes bad performance in long-tailed recognition. It proposed a new approach DRA, for data augmentation. The theoretical properties of the proposed method were proven, and experimental results revealed the improvement of applying DRA in long-tailed problems. ", "summary_of_the_review": "The motivation of this paper is good, and the method seems reasonable. However, the theoretical analysis does not provide much insight into the field, and the performance improvement is marginal. Furthermore, the paper's organization is not good, making the paper very hard to read. ", "strength_and_weaknesses": "Strength\n\n* Good motivation and theoretically proved algorithm;\n\nWeakness \n\n* The theory about the algorithm is trivial and does not provide much insight into the progress of the field;\n* The high-level framework of DRA is missing. The paper quickly dived into details, making it very hard to understand the big picture;\n* The connection between each part is not clear. For example, Eq.5 seems to express the paper's central idea but is never referred to/used in the next parts. The connection between Eq.5 and the theorems is also not clearly stated. \n* The performance improvement seems marginal. \n* The authors only conducted experiments on CIFAR and Imagenet-LT, which are relatively small datasets. Would the authors like to run experiments on larger datasets, such as iNaturelist, to confirm the effectiveness of the authors' data augmentation scheme?\n* The paper seems to use a sampling method to remove the CCD shift (Eq. 5), so the problem becomes a traditional label shift problem. Then it applies the DRO to solve the label shift problem. The second part is quite similar to the work \"Coping with Label Shift via Distributionally Robust Optimisation, ICLR 2021\". Could the author tell the difference between the second part of this paper and that paper?\n* The paper proposed the sampling method (Eq.5) and then quickly moved to the optimization method. I'd like to see a more thorough analysis of the sampling method, including in what circumstances the sampling method is guaranteed to remove the CCD shift. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ML4UkkbWdT", "reviewer": "Reviewer_9ohR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "(1) This paper aims to solve the long-tailed recognition task.     \n(I) This paper proposes that the class-conditional distribution shift can not be ignored in practice with empirical analysis.    \n(II) Based on this observation, this paper proposes to utilize DRO to minimize the worst risk and It can be understood as an adaptive data augmentation method.     \n(2) Experiments on CIFAR-LT and Tiny-ImageNet-LT are conducted. Some improvements are observed compared with baselines. \n", "summary_of_the_review": "The paper aims to address the long-tailed recognition problem. Analysis of the CCD issue is interesting.\nHowever, the evaluation is unconvincing. Comparisons with state-of-the-art methods are missed. Also, only small datasets are evaluated.   \n\nIf concerns are well addressed, I'm glad to raise my score.\n\n\n", "strength_and_weaknesses": "Strength:    \n\n(1) The paper gives a sound theoretical analysis.   \n\n\nWeakness:    \n\n(1) Experiments are conducted only on CIFAR-LT or TinyImageNet-LT, which makes the experimental results unconvincing. In the long-tailed community, we usually validate the effectiveness of methods with ImageNet-LT, iNaturalist 2018, and Places-LT.     \n\n(2) The authors claim that \"decoupling methods avoid more severe CCD shift\". Does it mean that it is necessary to address long-tailed recognition with 2 stages? However, recent state-of-the-art methods, like PaCo [1], train representation and classifier jointly. Can the proposed method improve performance when it is applied to such a strong baseline?     \n\n[1] Parametric Contrastive Learning. ICCV 2021.      \n\n(3) How can we remove the effect of shifted CCD?      \n     As analyzed in Sec 3.2, function (5) is used to sample images. From my understanding, with function (5), all images can be viewed by the trained model, which is the same as that of training models on balanced data. The only difference is that the sample probability for tail-class images will be smaller.   \n\n(4) Analysis of the computational cost for the proposed method is missing.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "BOAAevfnrtI", "reviewer": "Reviewer_7ExN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors tackle the problem of long-tailed recognition. The authors highlight the issue of Class-Conditional Distribution (CCD) shift due to scarce instances. The authors propose an adaptive data augmentation method, Distributionally Robust Augmentation (DRA) to improve performance for long-tailed recognition tasks. Experiments across several semi-synthetic datasets illustrate the efficacy of DRA. ", "summary_of_the_review": "Overall, I think while this paper tackles an important problem of long-tailed recognition, the paper is hard to follow in its current stage and several arguments are presented without concrete theoretical and empirical evidence. In my understanding, the motivation for the CCD shift can be presented in a simpler way. A clear description of the proposed DRA algorithm is also missing in the paper. While the proposed DRA taking looks promising from empirical results, the connection with the proposed motivation is a bit thin. \n\nI encourage authors to participate in the discussion to clarify if I share any misunderstanding. I will be open to changing my score if they think I misunderstood any of the key arguments made in the paper.  ", "strength_and_weaknesses": "**Strengths**: \n- The paper tackles an important problem of long-tailed recognition. \n- Empirical results with the proposed DRA technique in Table 2 are promising. \n\n\n**Weaknesses**: \n- Discussion on Proposition 1 is a bit misleading. Authors argue that ` with enough instances from each class, models working well on training set could perform well in the test'. However, it is well accepted that usually even for balanced benchmark datasets (i.e., CIFAR100 or Imagenet), with overparameterized networks, we can not approximate $p(x|y)$ or the generalization performance. In general, for high-dimensional vision datasets typically used, the sample size is small enough that we would see high drift as evaluated by Proposition 1. If this understanding is incorrect, I would encourage authors to provide examples in balanced datasets like CIFAR10, CIFAR100, or Imagenet where bounds like in Proposition 1 are tight. \n- To put it differently, the issue here is that authors are trying to argue that it is not possible to estimate $p(x|y)$ from finite samples in LT classes. While this is understandable, it is unclear if (i) any method actually explicitly computes $p(x|y)$ to handle long tail classes; and (ii) $p(x|y)$ can be estimated reasonably even for balanced benchmark datasets like CIFAR-10, CIFAR-100, etc.  \n- In light of the above comment, I guess a more appropriate argument would be to discuss the relative sample sizes in tail classes of long-tailed datasets versus sample sizes in balanced datasets like CIFAR100. However, I am not sure if such an argument could be theoretically made with bounds in Proposition 1. \n- It is unclear if equation (4) is actually used to sample datasets in prior work. To the best of my understanding equation (4) represents the empirical distribution of a dataset sampled to satisfy LT phenomena. The core issue here is if we use sampling with replacement from distribution in (4) versus distribution in equation (5), we may see a lot of repeated examples from long tail classes in case of (4) than in case of (5). While this is intuitive, it is unclear if prior work has really used sampling with replacement from a model as in (4). Can authors add references to prominent prior works which used this strategy for benchmarking?\n- The paper is hard to follow starting from Section 4. In particular, I could not find a clear description of the proposed DRA method in the main paper. In Algorithm 1, it is unclear why the loop in Line 4 will be executed with $g_i^*$ initialized the way it is in Line 3. \n- Figure 3 is also a bit unclear. It is hard to follow the description of the DRA method from this figure. For example, it is unclear what \"From the k-th,...,M-th loops\" mean? What is the difference between blue and orange arrows? Why there are two input images to the networks (one in blue and one in orange)?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject