DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge
Authors: Jie Shao
Ke Zhu
Hanxiao Zhang
Jianxin Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Long-tail learning; long-tail classification
diffusion model
Primary area: diffusion_based_models
TL;DR: We introduce a pipeline for long-tail recognition that uses a diffusion model trained solely on the long-tailed dataset to generate a balanced proxy.

Abstract: This paper introduces a novel pipeline for long-tail (LT) recognition that diverges from conventional strategies. Instead, it leverages the long-tailed dataset itself to generate a balanced proxy dataset without utilizing external data or model. We deploy a diffusion model trained from scratch on only the long-tailed dataset to create this proxy and verify the effectiveness of the data produced. Our analysis identifies approximately-in-distribution (AID) samples, which slightly deviate from the real data distribution and incorporate a blend of class information, as the crucial samples for enhancing the generative model's performance in long-tail classification. We promote the generation of AID samples during the training of a generative model by utilizing a feature extractor to guide the process and filter out detrimental samples during generation. Our approach, termed Diffusion model for Long-Tail recognition (DiffuLT), represents a pioneer application of generative models in long-tail recognition. DiffuLT achieves state-of-the-art results on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, surpassing leading competitors by significant margins. Comprehensive ablations enhance the interpretability of our pipeline. Notably, the entire generative process is conducted without relying on external data or pre-trained model weights, which leads to its generalizability to real-world long-tailed scenarios.

DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge
Jie Shao
Ke Zhu
Hanxiao Zhang
Jianxin Wu"
National Key Laboratory for Novel Software Technology. Nanjing University, China School of Artificial Intelligence, Nanjing University, China {shaoj, zhuk, zhanghx) (anghx))@lammde ammda.. nju. ajj.edu..cee
edu. en, wujx2001@nju. edu.
Abstract
This paper introduces novel pipeline for long-tail (LT) recognition that diverges from conventional strategies. Instead, it leverages the long-tailed dataset itself to generate a balanced proxy dataset without utilizing external data or model. We deploy diffusion model trained from scratch on only the long-tailed dataset create this proxy and verify the effectiveness of the data produced. Our analysis identifies poproximmttelyiin-dfitbbbiiiiiii from the real data distribution and incorporate blend of class information, as the crucial samples for enhancing the generative model's 's performance in long-tail classification. We promote the generation of AID samples during the training generative model by utilizing feature extractor guide the process and filter out detrimental samples during generation. Our approach, termed Diffusion model for Long-Tail recognition (DiffuLT), represents a pioneer application of generative models in long-tu tail recognition. DiffuLT achieves state-of-the-aa art results on CIFARIO-LT, CIFAR and ImageNet-LT, surpassing leading competitors by significant margins. Comprehensive ablations enhance the interpretability of our pipeline. Notably, the entire generative process conducted without relying external data or pre-trained model weights, which leads to its generalizability real-world long-tailed scenarios.
(AID) samples, which slightly deviate
Introduction
Deep learning has exhibited remarkable success across spectrum of computer vision tasks, especially in image classification, e.g., as exhibited by He et al. [2016], Dosovitskiy et al. [2021), Liu et al. (202T). These models, however, encounter obstacles when faced with real-world long-tailes (LT) data, where the majority classes have abundant samples but the minority ones are sparsely represented. The intrinsic bias of deep learning architectures towards more populous classes exacerbates this issue, leading to sub-optimal recognition of minority classes despite their critical importance in practical applications.
[ao Conventional long-tailed learning strategies such as re-weighting (Lin et al. [2017), Cao et al. [2019a] re-sampling (Zhou al. (2020a] Zhang al. [2021a]). and structural adjustments (Wang et al. [2020] Cui et al. 2022)), share commonality: they acknowledge the data's imbalance and focus on the training of models. They demand meticulous design and are challenging generalize. Recently, shift towards studying the dataset itself and involving more training samples through external knowledge to mitigate long-tailed challenges has emerged (Zhang et al.) (202b) (024a)). Yet, many real-world scenarios, access to specialized data is limited and closely guarded, such in military or medical contexts. This predicament prompts a critical inquiry: Is it feasible to balance long-tailed datasets without depending external resources or models?
"J. Wu is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

50
50
OOD ID
DDPM CBDM
Deneity
CBDM
48
Oarr
Ours
41
0.2
46
30
Distance Fesluee Spoct
+4
PNO
20
10
Raseline
38
7.5
7.0
6.5
6.0
5.5
Figure 1: The samples generated by diffusion models improve long-tail tail classification on CIFAR100- LT, showing correlation between FID and accuracy and a stronger correlation between the proportion of AID samples and accuracy. Our method significantly boosts classifier accuracy compared with others (left). Feature space visualization reveals that different diffusion models generate samples with varying distributions, and our model biases the generative process toward AID samples (right).
Our answer is yes. Recent advances in diffusion models have demonstrated their significant potential in generating high-quality images (Ho et al. (2220] [Song et al. 2020)), Rombach et al. (022 Assuming that diffusion models are proficient learning distributions, we develop a diffusion model trained from scratch only the long-tail distributed dataset. This model creates new samples for underrepresented classes, which are then used to train classifier on the re-balanced dataset, leading to improved accuracy. We are the first to demonstrate the effectiveness of using generated samples, without relying on external data or models, in improving long-tail classification. We observe a notable pattern: enhancing the performance of the generative model with loss modification called CBDM (Qin et al. (2023)) also enhances the classifier's s accuracy, as illustrated in fig. [I] This phenomenon implies that generative model with better performance tends to generate samples that more beneficial for the classification task. This observation raises an important question: What are the most valuable generated samples for classification, and how are they generated? This question is critical, as it determines whether a diffusion model is going to be beneficial or detrimental for LT recognition.
We answer this question by analyzing features of generated samples, visualized in fig. D using I-SNE (Van der Maaten and Hinton (008)). We categorize the generated samples into three groups: in-distribution (ID), approximately in-distribution (AID), and out-of-distribution (OOD). Our research indicates that AID samples are pivotal in enhancing classifier performance. Through experiments, we conclude that diffusion model can assimilate patterns from the head classes and integrates them into the tail ones to produce AID samples. These samples significantly enhance the quantity and diversity of the tail classes, thereby substantially improving their performance. Then the important question to solve is: How can we generate AID samples efficiently?
To encourage the model to predominantly generate AID samples, we introduce a novel type of loss. and OOD samples. Such
This loss employs a feature extractor to penalize the generation strategy not only elevates the performance of the generative model on long-tail datasets but also renders more effective and efficient enhancing classifier performance.
In general, we introduce a new pipeline, DiffuLT (Diffusion model for Long- Tail recogmition), for long-tail datasets. It has three steps: initial training, sample generation, and retraining. Initially, we train a feature extractor and a diffusion model incorporating a supervision term to encourage the generation of AID samples. Subsequently. this generative model employed to augment the dataset towards balance. The final step involves training a new classifier on the enriched dataset, with minor adjustment to reduce the impact of synthetic samples. is crucial to underscore the importance of training the diffusion model without external data or knowledge, to maintain fairness in comparison.
Our contributions are summarized follows:
We pioneer in addressing long-tail recognition by synthesizing images using diffusion
models without relying external data.

---

Our research delves into the mechanisms underlying our approach, highlighting the signifi- cance of the generated AID samples. These samples emerge from a fusion of information from both head and tail classes, playing crucial role in enhancing classifier performance.
We introduce a novel loss function that enhances the performance of diffusion models on long-tailed datasets and biases them towards generating AID samples, thereby making the generation process more effective and efficient for classification.
Extensive experimental validation across CIFARIO-LII CIFAR 100-LT, and ImageNet-LTT datasets demonstrates the superior performance of our method over existing approaches.
2 Related Work
Long-tailed recognition Long-tailed recognition is a challenging and practical task CCui al. 2019). /Zhou et al. [2020b], Cao et al. (2019b). (Zang et al. [2023a] [Zhu et al.]
(2024), since natural
data often constitute a squeezed and imbalanced distribution. The majority of traditional long-tailed
learning methods can be viewed as (or special cases) of re-weighting (Cao et al.
[001b]] Kang et al.
2020 Zhong al. [2021a] Wang et al. [0244a) and re-sampling (Cui al. 2019 with more
emphasis on the deferred tail class to seek an optimization trade-off. There are variants of them that adopt self-supervised learning (Zhu et al. [2023], Li et al. [202T]. theoretical analysis (Li et al.
[2022], Menon et al. [2021], Yang et al. [2024]] ) and decoupling pipeline (Kang et al. [2020), Zhou
al. 2020bl) to tackle long-taie tailed learning from various aspects, and they all achieve seemingly
decent performance in downstream tasks.
One of the core difficulties in long-tailed learning is the insufficiency of tail samples. And recently, quite some works start to focus on this aspect by involving more training samples through external knowledge (Zhang al. [2021b), Ramanathan et al. [2020). Dong al. [2022], [Shi et al.) [2023a]). Nevertheless, the most distinct drawback of these works is that they either rely on external data source or strong model weights. This condition can seldomly hold true in practical scenarios where only a handful of specialized data are available and are secretly kept (consider some important military or medical data). We thus raise natural question about long-tailed learning: can we utilize the advantage of generating tail samples without resorting to any external data or model? That is, the whole process is done in an in-domain (also called held-in) manner. In this paper, we propose to adopt the off-the-shelf diffusion model to learn and generate samples from the data at hand.
Diffusion models and synthetic data Diffusion models have been highly competitive in recent
years (Ho et al. [2020]. Song et al. (2020), producing promising image quality in both unconditional and conditional settings (Dhariwal and Nichol [202T]. Rombach al.] [2022). |aamesh et al. (202T)) Despite the predominant use in creating digital art, the application of diffusion models in scenarios of
limited data remains under-explored. This paper affirms the utility of diffusion models in enhancing
representation learning, particularly within the long-tailed learning framework, offering novel
insight into their application beyond conventional generative tasks.
The integration of synthetic data into deep learning, generated through methods like GANs Goodfel
low et al. 2014), Isola al. [2017] and diffusion models (Dhariwal and Nichol [202I], Rombach et al.
[2022] ), has been explored to enhance performance in image classification (Kong et al.
[0019] Azizi
ct al. [2023] Zhang et al. 2024a Trabucco al. (2023). D, object detection (Zhang et al. (2023b) ), and
semantic segmentation (Zhagg et al. [202cc]]] 2023b)) These approaches often depend on substantial volumes of training data or leverage pre-trained models, such as Stable Diffusion, for high- qugl-tualtty data generation. Yet, the efficacy of generative models and synthetic data under the constraint of limited available data and in addressing imbalanced data distributions remains an unresolved inquiry. This paper specifically addresses this question, evaluating the viability of generative models and synthetic data in scenarios where data is searce and imbalanced.
Method
3.1 Preliminaries
For image classification, we have a long-tail dataset D 1.) E C with each an; representing an input image and ys representing its corresponding label from the set of all classes
3

---

Table 1: FID of different generation models and their corresponding classifiers' accuracy.
Table 2: Percentage of different types of gener- ated samples for each model.
Model
FID
Acc. (%)
Model
PtD
PAID
PooD
Baseline
38,3
DDPM
21.2
39.7
DDPM
7.76
43.8
CBDM (T
38.6
29.1
32.3
CBDM (T 3)
7.42
44.8
CBDM (T 2)
6.82
46.0
CBDM (T
40.2
33.5
26.3
CBDM 1)
5.86
46.6
CBDM (T
44.8
36.3
18.9
C. In the long-tail setting, a few classes dominate with many samples, while most classes have very few images, leading significant class imbalance. The classes in C ordered by sample count with |cl 2 (22 Z 2 |casl. where lcll denotes the number of training samples in class ej and Leal is defined the long-tail ratio. The goal of long-tail classification y capable of effectively handling the tail classes.
Jeil |eall. The ratio Jewa
learn a classifier fep X
The naive idea is to train a generative model on the long-tail dataset D and use the trained model generate new samples and supplement the tail classes. Inspired by its superior performance, select diffusion models as the generative model in our pipeline. In our approach, we follow the Denoising Diffusion Probabilistic Model (DDPM by Ho et al. (2020) framework. Given a dataset D { Vi > we train diffusion model to maximize the likelihood of the dataset. At every training step, we sample mini-batch
batch of images 2g from the dataset and add noise obtain Zt.
q( ; xp) (Vārao (1-aa)))
where a; = II, 3.) is calculated through pre-defined variance schedule {3e E (0, 1) After training diffusion model 9 to get po(x;;1 æt, t), we reverse the above process step by step to recover the original image To from pure noise 2T ~ _N I). The training objective is to reduce the gap between the added noise in forward process and the estimated noise in reverse process:
LDDPM = E...,,,)) []le: ca(vā + vI - ā,Et, t)113)
(2)
where 6, ~ NO I) is the noise added to original images and 6g is the noise estimated by the trainable model with parameters 6. DDPM can be conditional by transforming y into trainable class embedding and incorporating the label directly as am input, similar to time f. To improve the performance of DDPM on long-tailed dataset, several works (Qin al. [2023] Zhang al. (2024b) have been proposed to adjust the distribution of generated samples. CBDM adds distribution adjustment regularizer the loss term. This term is designed promote the generation of samples for tail classes, which is defined as (where sg means stop gradient):
LCBDM
TL
Σ (1leo( (,9, sg(eo( 1/11) +>ll wotlllllssgeeaa PEY
(,,,,,,,11)) (3)
y
3.2 DiffuLT: Diffusion model for Long-Taa Tail recognition
Diffusion model helps long-tail classification. In this phase, a randomly initialized diffusion model 0 is trained to enrich the dataset. Preliminary experiments involve training DDPM on a tailed dataset and using it to generate additional data. threshold N₂ is set, and for classes Cj with fewer generate the images to meet this threshold. This augmentation results in Nesn, where Ngen = Eeyec max(0, N₁ - (cşl) represents the total number generated samples. These generated samples are then integrated with the original dataset, forming augmented dataset D J D gen. which a classifier is trained to enhance classification performance.
than N, samples, collection of synthetic samples, Dzen = { yı )
We conducted experiments on CIFARI00-LT with an imbalance ratio of 100 and set N₂ = 500 to supplement the data. The results, detailed the second line of table [ show 5.5% accuracy increase for the classifier trained on D D gen compared to the baseline. This improvement underscores the effectiveness our straightforward method in boosting overall performance. Considering the generated samples (especially for tail classes) may be of lower quality due to limited data availability,

---

DONM
CUUM T-
COLM y=2
COOM
Figure Visualization of generated samples for class 90 in feature space using t-SNE. The associated model is indicated in the upper-left corner.
Table 3: Quantities, overall classifier enhancement, and average improvement per sample for different groups of data generated by diffusion model.
Table Diffusion trained with varying proportions of head class data and the corresponding results for tail classes.
Group
[2 ||
Acc. (%)
AAce/||Dass
Ph
PAID
Acc; (%)
Baseline
38.3
25.0
0
25.8
26.0
ID
21,511
44.2
2.75 x 10
40
33.2
29.7
AID
11,886
45.2
5.78 x 10
80
35.7
32.5
OOD
5,756
36.2
-3.61 x 10
100
39.1
32.8
Diffusion Models (CBDM) is employed improve generation quality in long stilet settings. By integrating LDDPM and LCBDM in training the model 0 on D, the dataset is enhanced, and classifier trained described previously. Subsequent testing CIFAR 100 reveals that the classifier achieves an accuracy of 46.6%, marking an 8.3% increase over the baseline, as noted in the final line of [able
What samples are helpful? AID samples! We adjusted the hyper-parament T in LCBDM and evaluate models with varying FID scores. Results presented in @bble]show that accuracy improves as FID decreases. Lower FID scores indicate that generated samples more closely resemble the real data distribution. Notably, some generated samples clearly fail, while others correctly resemble their intended class. This observation motivates further investigation into the efficacy of samples.
Class 90 (truck) is selected randomly as representative example in (FFAR10-LLT A baseline classifier (ff)) trained exclusively on the original dataset D, is used to analyze the generated data. This classifier extracts features which are then visualized using t-SNE, as shown in fig. 2 The visualization reveals that samples generated via CBDM tend to be more centralized. For deeper analysis, we define the center fo of a class's 's features as the average of the real data in feature space, and set the maximum Euclidean distance between two real samples' features as a threshold dy. We then define 3 types of the generated samples based on their distance to fo:
{ de S dy, d < 2d d > 2dd,,
ID AID
d; foll2=
OOD
where ID denotes in in-distribution samples, which closely match the patterns of the original data. We define and name approximately in-distribution (AID) samples, which exhibit slight deviations. OOD stands for out-of-distribution ones, which are significantly differing from the center. We summarize the composition of samples generated by each model in table 2 Notably, the CBDM model generates a lower proportion of OOD samples, consistent with its FID score. For evaluating the impact cach type, train classifiers using only the ID, AID, and OOD samples generated by CBDM with 1 respectively as 2 genn combined with D, and present the results in table Surprisingly, classifiers trained with AID samples achieve the highest accuracy and show the greatest average improvement per sample. Based on this finding, our hypothesis is that AID samples are the most beneficial in enhancing classifier performance.
Mechanisms behind the AID samples. We conducted experiments to explore how AID samples enhance classifier performance and where their new and useful information originates. A diffusion model (CBDM with - = 1) trained using images from tail classes (fewer than 100 samples), supplemented by variable proportion ps of head class images. This model generates samples
5

---

ID
AID
OOD
ID
AID
OOD
ID
AID
OOD
Snake
Tiger
Wardrobe
Skyscraper
Train
Whale
Figure 3: Examples of three groups of generated samples.
specifically for tail classes with the proportion of AID samples PArD. and gets the performance of the corresponding classifier denoted as Ace, The results, presented in table show that at ps 0%. relying solely on tail class images, PAID is 25.8%, and Acc, improves marginally to 26.0%. only 1% above the baseline. As Ps increases, both PAID and Ace, rise, peaking ph. 100%. This trend illustrates the diffusion model's 's ability to transfer information from populous to underrepresented classes, effectively blending data across different classes into AID samples. Examples of the sample groups are displayed in fig. 3 where ID) samples closely resemble real images, AID samples blend patterns from multiple classes, and OOD samples typically exhibit anomalies.
Generation of AID samples. How can we efficiently generate AID samples? While a filtering strategy can be used collect AID samples, it is not the most efficient method. more effective approach involves encouraging the generation model to specifically produce AID samples. Given that AID samples are defined by their distance from the center of real images feature space, we can utilize the baseline classifier
feature extractor to guide the generation of AID samples. Our goal is to encourage controlled deviation within feature space. After T denoising steps, the deviation should idcally be within the range of d, to 2d,. Assuming that the deviation in each step is proportional the noise strength, we introduce an additional term in the loss function to encourage small, stepwise deviations. We define the deviation at each step in feature space as
VI āT II di
de
(x)) - PP (xo + - - f)))
(5)
where so + (x t, v)) represents the de-noised images' feature. The new AID loss is then
3 2 d1
LaI = alB4 |d
where a is a LCBDM to train the generation model. After training, the generation process, we employ Po to filter out harmful OOD samples, resulting in Pgen- We then train the classifier using the combined dataset D U D gen- Recognizing that generated data less crucial than real images, we introduce a weighting term to the cross-entropy loss to adjust the influence of the generated samples:
and defaulted to 0.1. We incorporate this term into both LDDPM and this model to generate data. During
Lels = - (D....))DDD.... (Wyg + (1 - (n)) log exp(fen(x))
(7)
EM exp(fe.e.x
where controls the weight of generated samples and is set 0.3 by default. yy is additional label assigned to each image v, which distinguishes between generated and original samples. Specifically, Vg is used for generated samples, while yy 0 marks the original ones.
3.3 Overall Pipeline and Discussion
Now we are ready propose a new pipeline called DiffuLT to address long-tail recognition. The pipeline is shown in fig. 4 with four steps:
Training: Initially, we train feature extractor yo and conditional, AlD-biased di diffusion model 0 using the original tailed dataset D alone.

---

Imbal anced
Training
Aets
Forward only
Po
fp
Feature Extractor
Final Classifier
stspps
@ Aeis
Pr
Generate
Diffusion Ø
gen
Figure 4: The overall pipeline of our method DiffuLT.
Generating: We establish threshold N, and employ the trained diffusion model 0 to generate and supplement samples. Using Po, we filter out OOD samples, resulting refined dataset D gen.
Training: We then train new classifier fo on the augmented dataset D U D gen using weighted cross-entropy, forming our final model.
Compared traditional methods that focus primarily training, ours not only enhances performance but is also reusable for model updates. Our method requires more training time, typically four times longer, train the generation model and produce samples. However, our methods prove valuable when performance improvement is critical. Unlike typical data expansion methods, our approach offers both practical and theoretical benefits because don't rely on any external dataset or model. For detailed analysis, please refer to the appendix
Experiment
4.1 Experimental setup
Datasets. Our research evaluate three long-tailed datasets: CIFARIO-LT (Cao et al. [019a]] 100ART (Cao et al. (2019a]), and ImageNet-I.T (Liu et al. (2019a]). Following the methodology described (Cao et al. (019aa)) we construct long-tailed versions of the first two datasets by adjusting the long-tail ratio to 100, 50, and 10 to test our method against various levels of imbalance.
Baselines. In our comparative analysis, we benchmark against a broad spectrum of classical and contemporary long-tailed learning strategies. The methods compared can be classified into multiple genres like re-weighting and re-sampling techniques, head-to-tail knowledge transfer approaches, data- augmentation, and on. Some methods have issues such as unfair comparisons or implementation problems. We document both their results and our implementation outcomes in the appendix A
Implementation. We set a = 0.1, and w = 0.3. The generation thresholds N, for CIFAR 10-LT and CIFAR1O0-LT were fixed at 5000 and 500, respectively. We employ ResNet-32 the classifier backhone. For ImageNet-LT experiments, we set a generation threshold N₂ 300. The classifiers were based on ResNet- 10 and ResNet-50 architectures with w 0.5.
More details about the experimental setup are available the appendix
4.2 Experimental Results
We assess the efficacy of our specially designed loss function, detailed in table 5 This function improves the FID by reducing OOD samples, while also increasing the number of AID samples and the classifier's accuracy. Further experiments in table highlight the necessity of our training loss. For benchmarking, we use basic filtering strategy for CBDM, with all models generating samples to meet the threshold N, for each class. The terms "Kept" and "G-Num" denote
Generative Results.
7

---

Table 5: FID of diffusion model, proportion of samples, and corresponding classifier accuracy
Table 6: Methods and types of retained samples, pre-filtering counts, and classification accuracy.
Method
FID
PID
PAID
PooD
Acc. (%)
Method
Kept
G-Num
Ace. (%)
DDPM
7.76
39.1
21.2
39.7
43.8
CBDM CBDM CBDM
39.153
46.6
AID ID & AID All
108,684
48.1
CRDM
5.86
44.8
36.3
18.9
46.6
48,414
47.1
Ours
5.37
40.7
50.1
9.2
49.7
Ours
39,153
49.7
Table 7: Results on CIFARI00-LT and CIFARIO-LT datasets. The imbalance ratio r set to 100, 50
and 10. The highest- results are in bold, with the second-best in underline. Additionally, we present the results for different groups (many, medium, and few) in CIFARI00-L7 with 100.
CIFARI00 LT
CIFARIO-LT
Statistics Med.
Method
100
50
10
100
50
10
Many
Few
38.3
43.9
55.7
70.4
74.8
86.4
65.2
37.1
9.1
Focal Loss Lin et al.
2017
38.4
44.3
55.8
70.4
76.7
86.7
65.3
38.4
8.1
LDAM-DRW Cao al. 2019a
42.0
46.6
58.7
77.0
81.0
88.2
61.5
41.7
20.2
cRT Kang et al.
2019
42.3
46.8
58.1
75.7
80.4
88.3
64.0
44.8
18.1
BBN Zhou et al.
2020a
42.6
47.0
59.1
79.8
82.2
88.3
RIDE experts)
Wang ct
2020
48.0
68.1
49.2
23.9
CAM-BS Zhang et al
2021
41.7
46.0
75.4
81.4
MisLAS Zhong
2021b
47.0
52.3
63.2
82.1
85.7
90.0
DiVE He el al. 202T
45.4
51.1
62.0
CMO Park el
2022
47.2
51.7
58.4
70.4
42.5
14.4
SAM Rangwant et
2022
45.4
81.9
64.4
46.2
20.8
CUDA Ahn et al. 20233
47.6
51.1
58.4
67.3
50.4
21.4
CSA Shi 202.b
46.6
51.9
62.6
82.5
86.0
90.8
64.3
49.7
18.2
ADRW Wang 2024b
46.4
61.9
83.6
90.3
Li
2023
48.9
53.8
DiffuLT
51.5
63.8
84.7
86.9
90.7
69.0
29.7
DiffuLT BBN
51.9
56.7
64.0
85.0
87.2
90.9
69.5
51.9
30,2
DiffuLT RIDE (3 experts)
52.4
56.9
64.2
85.3
87.3
90.9
70.3
52.1
30.7
the types of samples retained and the total number samples generated before filtering, respectively. Our methods enhance the generation process's efficiency and achieve the highest accuracy.
CIFAR100-LT & CIFARIO-LT. We benchmark our approach against a range of methods on the CIFAR LOO-LT and CIFAR 10-LT datasets, with results detailed in table 7 The results not shown in the original papers are indicated as "_" in the table. On CIFARIOO-LT, our method surpasses competing models, achieving accuracy improvements of 13.2%, 12.4%. and 8.1% compared with the baseline for r = 100, 50, and 10, respectively. On CIFAR 10-LT, our model also demonstrates strong competitiveness, enhancing accuracy by 14.3%, 12.1%, and 4.3% across the long-tail ratios, further validating the effectiivness of our method. Since our methods solely modify the training data, they be easily integrated with other methods to achieve better results.
For CIFAR 100-LT with an imbalanced ratio of 100, performance also assessed across three categories: many (classes with over 100 samples), medium (classes with 20 to 100 samples), and few (classes with fewer than 20 samples). While our approach does not lead in the "Many" category, excels in "Med." and "Few". significantly outperforming others in the "Few" group with 29.7% accuracy - 8.3% above the nearest competitor and 20.6% beyond the baseline.
Imt.eNet-LT. On the ImageNet-LT dataset, our methodology is evaluated against existing ap- proaches, with results summarized in table Utilizing a ResNet- 10 backbone, our method registers a 50.4% accuracy, outperforming the nearest competitor by 4.5%. With ResNet-50, the accuracy further escalates to 56.4%, marking substantial 14.8% enhancement over the baseline. Despite slight decline in the "Many" category relative to the baseline, our approach excels in "Med." and 'Few", with the latter witnessing a remarkable 33.6% improvement over the baseline. Our method can be combined with others achieve enhanced results.

---

Table 8: Results ImageNet-LT. We deploy ResNet-10 and ResNet-50 classifier backbones. Top-performing results are highlighted in bold, with second-best outcomes underlined.
ResNet-10 All
Re
All
Many
Med.
Few
CE
34.8
41.6
64.0
33.8
Focal Loss
Lin et al.
2017
30.5
OLTR Liu ct al.
20
35.6
cRT Kang al.
2019
41.8
47.3
58.8
44.0
26.1
RIDE (3 experts)
Wang et al.
2020
45.9
54.9
66.2
51.7
34.9
MisLAS Zhong al.
2021b
52.7
CMO Park al.
2022
49.1
67.0
42.3
20.5
SAM Rangwani
2022
53.1
62.0
52.1
34.8
CUDA Ahn et al. 20233
51.4
63.1
48.0
31.1
CSA Shi et
2023b
42.7
49.1
62.5
46.6
24.1
ADRW Wang et al.
2024b)
54.1
62.9
52.6
37.1
DiffuLT
50.4
56.4
63.3
55.6
39.4
DiffuLT + RIDE (3 experts)
51.1
56.9
64.1
55.8
39.9
Table 9: Ablation experiments to verify the effect of each module.
Table 10: Performance with different weights and hyper-parameter
Gen.
LAID
Weight
Ace. (%)
Acc. (%)
Acc. (%)
38.3
0
38.3
0
38.3
0.1
49.2
0.1
49.7
46.6
0.3
51.5
0.5
49.5
49.7
0.5
50.1
1.0
48.3
V
50.3
0.7
50.3
2.0
45.1
51.5
1.0
50.3
4.0
43.3
4.3 Ablation Study
Different modules in our pipeline. Our methodology comprises several critical components: generated samples (using CBDM), AlD-biased loss, filtering, and weighted cross-entropy. We conduct ablation experiments on CIFAR100-LT with 100. The results, presented table 9 highlight the crucial role each component plays in enhancing the overall performance. Notably, the generated samples and AlD-biased loss are the most influential factors.
Sxpeerparraeterss We adjust the parameters in the weighted cross-entropy and a in LAID on CIFAR 100-LT with r 100 and evaluate the classification results. These results are summarized in table I0, Through iterative adjustments, we find that the optimal performance, 51.5% classification accuracy, is achieved when w 0.3. Similarly, the best setting for is determined to be 0.1. Consequently, we establish 0.3 and a = 0.1 as the default settings for our method.
5
Conclusion
In this research, we proposed a novel, data-centric approach designed address the challenges of long-tail classification. We defined and identified AID (approximately in-distribution) samples the important ones. We then revised a diffusion model trained with an AlD-biased loss term on only the original dataset for the purpose of generating more AID samples, thereby significantly enriching the dataset. Following sample generation, we trained a classifier on this enhanced dataset and employed a weighted cross-entropy loss. Our method has shown deliver competitive performance, highlighting its efficacy in real-world applications. The experiments conducted as part of this study notably emphasize the critical role played by AID samples and their significant impact.
We propose that this approach introduces a new paradigm tackling long-tail classification chal- lenges, offering a substantial complement to existing methodologies. It provides robust framework

---

that can be udapted to various scenarios where performance is a critical factor. Despite its advantages, the training of the diffusion model and the generation of samples are time-consuming. The need for optimization in training and generation speeds represents limitation of our current method. We will leave this point as future work further improve the effectiveness and efficiency of our method.
Acknowledgements
We ucknowledge the funding provided by the National Natural Science Foundation of China under Grant 62276123 and Grant 61921006. Wu is the corresponding author.
References
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with distriidittii aware margin loss. Advances in neural information processing systems, 32, 2019a.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9719-9728, 2020a.
Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3447-3455, 2021a.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by rout- ing diverse distribution aware experts. In International Conference on Learning Representations, 2020.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3695-3706, 2022.
Cheng Zhang, Tai- Yu Pan, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo. Boqing Gong, and Wei-Lun Chao. Mosaicos: a simple and effective use of object-centric images for long-tailed object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 417-427, 2021b.
Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. Advances in Neural Information Processing Systems, 36, 2024a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-665 6851, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
10

---

Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Class-balancing diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18434-18443, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Blass-balanced loss based on effective number of samples. In CVPR, pages 9268-996999 0178-997,,.11 2019.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR, pages 9716-9725 2020b.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced margin loss. In NeurlPS, pages 1565-1576, 2019b.
datasets with
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(99:10995-10816 2023a.
Ke Zhu, Minghao Fu, Jie Shao, Tianyu Liu, and Jianxin Wu, Rectify the regression bias in long-tailed object detection. arXiv preprint arXiv:2401. 15885, 2024.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia, Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16499, 2021a.
Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and adjustmettt for imbalanced learning. Advances in Neural Information Processing Systems, 36, 2024a.
Ke Zhu, Minghao Fu, and Jianxin Wu. Multi-label self-supervised learning with scene images. In ICCV, pages 6694-6703, 2023.
Tianhao Li, Limin Wang, and Gangshan Wu. Self supervision to distillation for long-tailed visual recognition. In ICCV, pages 630-639, 2021.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In CVPR, pages 6929-6938, 2022.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
Zhiyong Yang, Qianqian Xu, Zitai Wang, Sicong Li. Boyu Han, Shilong Bao, Xiaochun Cao. and Qingming Huang. Harnessing hierarchical label distribution variations in test agnostic long-tail recognition. arXiv preprint 24ti:::2005077888 2024.
Vignesh Ramanathan, Rui Wang, and Dhruv Mahajan. Diwl: Improving detection for lowshot classes with weakly labelled data. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 9342-9352, 2020.
Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classification. arXiv preprint arXiv:22 10.01033, 2022.
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Xin-Yan Han, Jie-Jing Shao, and Yu-Feng Li. Parameter-effiienn Long-tailed recognition. arXiv preprint 2rXiv:2309.1000 10019, 2023a.

---

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8770-87994 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to tome translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1125-1134, 2017.
Quan Kong, Bin Tong, Martin Klinkigt, Yuki Watanabe, Naoto Akira, and Tomokazu Murakami. Active generative adversarial network for image classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4090-4097, 2019.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmenta- tion with diffusion models. arXiv preprint arXiv:2332.00994. 2023.
Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang. Min Zheng, and Andy J Ma. Diffusionengine: Diffusion model is scalable data engine for object detection. arXiv preprint arXiv:2309.0382
2023b.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Francorac Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021c
Tianjiao Zhang, Huangjie Zheng, Jiangchao Yao, Xiangfeng Wang, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed diffusion models with oriented calibration. The Twelfth International Conference on Learning Representations, 2024b.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in open world. In CVPR, pages 2537-2546, 2019a.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.092117 2019.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for tailed recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489-16498, 2021b.
Yin- Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 235-244, 2021.
recognition.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6887-6896, 2022.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on class-imbalanced data. Advances Neural Information Processing Systems, 35: 22791-201 1-2805, 2022.
12

---

Sumyeong Ahn, Jongwoo Ko. and Se- Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. arXiv preprint XXiv:2302.00499 2023.
Jiang-Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? Advances in Neural Information Processing Systems, 36, 2023b.
Zitai Wang. Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. unified generalization analysis of re-weighting and adjit-adjustment for imbalanced learning. Advances in Neural Information Processing Systems, 36, 2024b.
Mengke Li, Zhikai Hu, Yang Lu, Weichao Lan, Yiu-ming Cheung, and Hui Huang, Feature fusion from head to tail: an extreme augmenting strategy for long-tailed visual recognition. arXiv preprint arXiv:2306. 06963, 2023.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019b.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 13:12204--2114 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-25 2015.
Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In NeurIPS, pages 4 4175-4186, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiyy Jia. Parametric contrastive learning. In
Proceedings of the IEEE/CVF international conference computer vision, pages 715-724, 2021.
Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Shiran Zada, Itay Benou, and Michal Irani. Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images. In International Conference on Machine Learning, pages 25817-25833. PMLR, 2022.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.0950/ 2018.
13

---

A Experiment
A.1 Experimental settings
Due to space constraints in the main paper, we only include essential information about the experi- mental setup. Additional details provided here.
CIFARIO0-LT & CIFARIO-LT. The original CIFARIO0 and CIFARIO datasets each consist of training set with 50,000 images evenly distributed across 100 10 classes, respectively. The CIFARIOOA 100-LT and CIFARIO-LT datasets, derived from CIFARI00 Krizhevsky [2009] and CI- FARIO Krizhevsky [2009], feature long-tail distribution where the class frequency decreases exponentially from class 0 to the last class. Commonly used long-tail ratios are 100, 50, and 10. Specifically, the CIFARI00-LI subsets contain 10.847, 12.608, and 19,573 images, with the largest class containing 500 samples and the smallest having 5, 10, and 50 samples, respectively. Similarly, the CIFAR 10-LT subsets consist of 12,406, 13,996, and 20,431 images, with the largest classes containing 5,000 samples and the smallest 50, 100, and 500 samples, respectively.
Experiments on CIFARI00-LT and CIFARIO-LT utilize the framework and training methodologies from Qin et al. 2023]] incorporating the CBDM Qin et al. [2023] loss function and Adaptive Augmentation Karras et al. 2020). We set the training duration to 500,000 steps, with hyper- parameters T and y fixed at and 0.25, as per the cited study. The batch size is maintained at 128,
the diffusion process runs for 1,000 time steps, and the learning rate is set at 0.0002 using an Adam optimizer.
For classifier training, we follow the code and protocols from Zhou et al. 2020aa. which prescribe a 200-epoch training regimen. The classifier training also employs a batch size of 128, utilizing an SGD optimizer with a learning rate of 0.1. The feature extractor yo is trained using this setup without any additional methods or data, The final classifier is trained similarly but incorporates both generated and original samples. All training tasks are conducted on 8x NVIDIA GeForce RTX 3090 GPUs, with further discussed in appendix B
Let.eN LT. ImageNet-LT, comprising 115,846 images across 1,000 classes with a maximum of 1,280 images per class and a minimum of 5, follows the specifications set in (Liu al. (2019a]). This dataset is derived from ImageNet Ruusskkvvky et al. (2015) by sampling a subset according to Pareto distribution with a power value of a = 6. For perspective, the original ImageNet training set contains 1,281,166 images, making ImageNet-LT less than 10% the size of the original dataset. The test set of ImageNet-LT mirrors that of ImageNet, containing 100,000 images.
Due to its considerable size and image resolution, ImageNet-LT necessitates modifications from the standard CBDM framework address inefficiencies. We adapt the codebase from Dhariwal and Nichol [021] for this purpose, extending the training to 1,980,000 iterations. This setup uses the Adam W optimizer with a learning rate of 3e-4 and batch size of 64, adjusted from the original 256 due to GPU memory constraints. Moreover, the model is trained at an image resolution of 256, and the total diffusion time step set to 1,000.
Classifier training for ImageNet-LT employs the framework from Zhang et al. (2021a). Specifically, the model is trained over 100 epochs with batch size of 512 using the SGD optimizer at learning rate of 0.2. All training tasks are carried out on 8x NVIDIA GeForce RTX 3090 GPUs.
A.2 Generated Images
We synthesize images for CIFAR- 10 and ImageNet-LT to demonstrate their utility in enhancing long-tail -ail recognition. Randonly selected examples of the generated samples are displayed, particularly for CIFAR100-LT in fig. 5 where images have resolution of 32 x 32 and focus on "few-shot"
with fewer than 20 instances. Despite the limited examples available in these classes, our diffusion model effectively utilizes the entire dataset to produce high-quality samples. While these synthesized images maintain some similarities with their original counterparts, the notable variations make them valuable for model training. However, because the generated images may sometimes distort essential features or introduce inaccuracies, it becomes crucial to filter out the OOD samples maintain their usefulness.
In fig. 6 we showease the generated images for CIFAR 10-LT, which demonstrate superior quality compared to those from CIFARIO0-LT, 100-LT, due to a more abundant training dataset. In CIFARIO-LT,
14

---

Figure 5: Generated images for CIFAR100-LT
Figure 6: Generated images for CIFAR10-LT
Table 11: Repeated experiments CIFAR100-LT and CIFAR 10-LT to test the robustness of our methods.
CIFAR100-LT
CIFARIO-LT
Method
100
10
100
50
10
(4)
51.5
56.3
63.8
84.7
86.9
90.7
(2
51.7
56.3
63.7
84.9
86.8
90.7
(3
51.5
56.3
63.3
84.9
86.3
90.6
Table 12: Repeated experiments on ImageNet-LT to test the robustness of methods.
ResNet-10
ResNet-50
(1
50.4
56.4
(2
50.4
56.5
(3
50.5
56.5
each class generally contains times more images than in CIFARIOO-LT. The smallest class in CIFAR10-LT has 50 images, greatly exceeding the minimum of 5 in CIFAR100-13 This increase in sample size, however, requires more stringent filtering process to prevent potential information loss due to the larger volume of images generated.
For ImageNet-LT, the generated images, displayed in fig. feature resolution of 224 224, which is significantly clearer than those from CIFAR-10 and CIFAR-100. While some finer details, such as text within the image or textures like fur, may not fully distinct, the generated images effectively capture the essential patterns of the classes. These images can significantly aid the long- sail-ail recognition task, particularly for classes with fewer original samples.
A.3 Robustness anlysis
Since CIFAR 100-LT and CIFARIO-LT are typically sampled randomly from their original datasets, we tested our methods across various sampled sets to assess their effectiveness and robustness. The results, displayed in table TT demonstrate that our method's performance stable, exhibiting only minimal variations. For ImageNet-LT, which is fixed dataset, generated samples three times to
15

---

Figure Generated images for ImageNet-LT
examine consistency. The variations among different sample sets are minimal, as shown in table I2 Therefore, our method proves to be robust in enhancing long-tail classification performance.
A.4 Details of Baseline Methods.
In the main paper, we outline various approaches to long-tail classification and benchmark our methods against these strategics. Here, we provide a concise overview the methods within each category. For re-weighting and re-sampling techniques, we examine Cross-Entron
(CE), Focal Loss
(Lin et (007)). LDAM-DRW (Cao et al. (2019a)). cRT (Kang al. (019]). BBN (Zhou al. [020a CSA (hii et al. (2233)). ). and ADRW (Wang et al. [2024b] ). In the realm of head-to-tar knowledge transfer, we include methods such as OLTR (Liu et al. ((01bb] and H2T (Li et (2023)
Label-smoothing strategies are represented by MisLAS (Zhong et al. ((221b]) and DiVE (He et al.
2021), while data augmentation, we compare our approach with CAM-BS (Zhang et 202 )
CMO (Park et al. (2022) ) and CUDA (Ahn et al. (2023)) Lastly, SAM (Rangwani et al. 2022
exemplifies an advanced optimization technique, and RIDE (Wang al. (220 showcases a mixture of expert technique in comparison.
16

---

Table 13: Results CIFARIO0-LT using alternative pipeline based the implementation guidelines from BSCE Ren et al. 2020 with imbalance ratios r set at 100, 50, and 10.
CIFARI00- LT
Method
100
50
10
Baseline
38.3
43.9
55.7
Baseline"
45.3
50.3
61.9
BSCE Ren et al.
2020
50.8
63.0
PaCo Cui al.
2021
52.0
56.0
64.2
GPaCo Cui
2023
52.3
56.4
65.4
DiffuLT
54.7
58.9
66.1
DiffuLT + GPaCo
55.4
59.5
66.4
A.5 Other Methods.
For various reasons, some methods are not included in our experimental comparisons. This decision primarily stems from two factors. Firstly, several methods, despite demonstrating impressive results, do not have publicly available code (Zada et al. [2022), limiting our ability to perform direct comparisons. Secondly, methods such those in Cui et al. [202T) and Cui al. [2023] achieve commendable results and can be replicated. However, their comparisons may be considered unfair. These methods utilize AutoAugment (Cubuk al. (2018). ). with parameters optimized across the entire dataset, rather than specifically for the long-tailed segment. This approach significantly boosts their baseline performance, as reported in Ren al.) [222]. For instance, on CIFARI00-E with an imbalance ratio of 0.1, baseline accuracy improves from 38.3 to 45.3, as shown in table 13 The first "Baseline" line reflects the standard settings, while entries marked with use the enhanced settings, demonstrating a substantial improvement. Comparing these results with those obtained under standard conditions would be unfair. Our methodology could also adapted to such settings and combined with these techniques to achieve excellent results, as illustrated in table [3 However, these results are omitted from the main paper to maintain fair comparison.
B Discussion
B.1
Comparison of our method with other type of methods.
The comparison is divided into two parts: comparing our methods with traditional long-tail recognition approaches and contrasting them with data synthesis methods, shown in fig. &
Compared to traditional long-tail classification methods that predominantly focus on training, our approach offers novel perspective. Rather than designing intricate methods to facilitate training long-tailed datasets, we straightforwardly enhance the dataset using a generative model. This approach is not only innovative but also compatible with existing training methodologies and demonstrates improved performance. However, the main limitation is the training of the generative model, which time-consuming and challenging to scale. These points will be further discussed in the subsequent subsection.
When comparing with data synthesis methods, it is clear that our approach does not surpass those employing technologies like Stable Diffusion or CLIP on long-tailed datasets. Nevertheless, the use of such large models trained on extensive data eliminates the core challenge of long-tail problems-daa scarcity. For instance, the class "train" in CIFAR CIFAR100-LT 100-LT has only ten images, whereas Stable Diffusion has been trained thousands of train images. In practical settings, accessing such expansive models tailored to specific datasets is unrealistic. These methods mainly benefit from data leakage. Our approach, in contrast, is designed for real long-tail tail scenarios without reliance external data or models, providing practical and theoretical value. We address several previously unanswered questions:
* Is diffusion model trained from scratch beneficial for long-tail recognition? Yes.
17

---

Train
40
200
Stable Difiusion
Classfier
Sec mll Eafrs
(a) Previous long-tail recognition methods.
(b) Data synthesis methods.
T steps
500
Train
Generate
Diffusion
390
(c) Ours
Figure 8: Main caption describing all images
Which generated samples useful for long-tail -ail recognition? AID samples.
Why does diffusion work for long-tail recognition? It blends class information to generate beneficial and novel samples.
B.2 Limitation
The primary limitation of our methods is the extensive training time required for the generative model. For instance, training diffusion model on CIFARI00-LY takes 24 hours, while ImageNet-LT requires approximately six days. As the quality and quantity of data increase, the training costs scale up significantly, making challenging to apply our methods larger datasets such iNaturalist and Places-LT LT due to resource and time constraints.
Despite these challenges, problem. To improve training efficiency, we are exploring two potential solutions. The first involves adopting techniques that accelerate the training and inference processes of diffusion models. The second strategy considers the use of pre-trained generative models in real long-tail scenarios. This approach does not contradict our previous assertion that using large, pre-trained models on familiar long-tailed data is unfair and not truly representative of long-tail challenges. Instead, we advocate for the use of pre-trained models on long-tail datasets they have not previously encountered, ensuring fairness and practical applicability. Employing a pre-trained model could significantly expedite our pipeline by eliminating the need train from scratch. This topic extends beyond the scope of this paper and will be addressed in future research.
methods are highly effective in addressing the long-tail recognition
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The conclusions and methods in the abstract and introduction accurately encapsulate the contributions of our paper, detailed in section The experimental results discussed are fully documented section 4 confirming the claims' validity.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper transparently acknowledges the main limitations in section 3.3 and section 5 Additionally, potential limitations are explored in the appendix B ensuring comprehensive discussion of the constraints and challenges encountered.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The focus of paper is primarily empirical; thus, our conclusions and methods
are derived from and validated by experimental results rather than theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have developed a straightforward and easily replicable pipeline, fully detailed in section All necessary hyper-panne spperr-rrmmeters and experimental settings are outlined Additional information required for reproduction is provided in the ap-
in section 4.1
pendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The code related this paper will be released open-source after it is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ inips.cccppbllic//
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All essential experimental settings, including dataset, hyperparameters, selec- tion criteria for these parameters thoroughly detailed in section 4 Additional experimen- tal details are provided the appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed information on the statistical significance of the experiments. These details are available in the appendix appendix
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed information about the computational resources required, including the type of GPU and execution times, is provided in the appendix appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://aarrppp..ccpppp///a/ttt
Answer: [Yes]
Justification: We have thoroughly reviewed and adhered to the NeurIPS Code of Ethics throughout our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [NA]
Justification: Our research concentrates on the long-tail recognition issue, a technical challenge within the field of computer vision that typically has minimal societal impact. We utilize open-source datasets and ensure that no harmful information is generated.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our research doesn't It involve models capable of generating high-risk content nor does it collect data from the internet, thus eliminating the need for such safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the data, code, and methods employed in this paper are open-source. We ensure proper citation of these resources and adherence their licenses and terms of use.
Guidelines:
23

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttsspperssaaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
This paper does not introduce any new assets, hence there is no associated documentation.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments or research with human subjects, therefore this question is not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer: [NA]
This paper does not engage in research with human subjects or crowdsoureing; thus, there are study participants, potential risks, requirements for IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
25


Reviews:
[{"review_id": "ofvLHWSE8C", "reviewer": "Reviewer_PgsY", "summary": "This paper introduces a diffusion based long-tail recognition method termed DiffuLT. DiffuLT uses diffusion model to generate supplement samples in order to balance the long-tailed dataset. The authors first discover that approximately-in-distribution (AID) samples are crucial in enhancing the generative model’s performance in long-tail classification, as it blends the useful information from other classes into the tail class. Inspired by such discovery, the authors propose to first train a feature extractor $\\varphi_0$ merely from the long-tail dataset, then utilize  $\\varphi_0$ to guide the generation of AID samples, and finally train the desired classification model using the original dataset and newly generated AID samples together. Experiments validate the enhanced performance of DiffuLT.", "questions": "Could you explain how your method differs from other data augmentation methods?", "limitations": "See *Weaknesses*.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written in the following dimensions:\n\n1. **Originality**: The originality of this paper comes mainly in two aspects. First, the authors discover that AID samples are crucial in enriching tail class samples and enhancing model performance. Second, the authors incorporate revised diffusion model to generate AID samples. This method is original and effective.\n2. **Quality**: This paper is in good quality. The proposed method is both convincing and effective.\n3. **Clarity**: From introduction to discovery of the effectiveness of AID samples, then the pipeline of DiffuLT and finally the experiments and ablation studies, the structure is clear and logical.\n4. **Significance**: The discovery that AID samples are crucial in long-tail recognition performance is quite useful.", "weaknesses": "1. It would be better if you show how “centralized” figure 2 is, for example, add support data of the proportion of ID, AID and OOD data. This figure is not intuitive enough. \n2. Citation suggestions. I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study. Specifically, i recommend citing the following articles:\n  -  A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning (NeurIPS 23)\n  - Deep long-tailed learning: A survey (TPAMI 23)\n  - Effective Data Augmentation With Diffusion Models (ICLR 24)\n  - Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition (ICML 24)\n3. Minor typos. Line 137 “as *an* input”. Line 227 “because it *doesn’t* rely on any *external* dataset or model”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 \n>It would be better if you show how “centralized” figure 2 is, for example, add support data of the proportion of ID, AID and OOD data. This figure is not intuitive enough.\n\nThank you for your feedback. We agree that the centralization of data is not immediately apparent in Figure 2. It becomes clear through the statistics presented in Table 2. We believe that adding annotations to indicate each data type's region in the figure would help readers understand it more effectively.\n\n### W2\n> Citation suggestions. I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study.\n\nThank you. We will include the first and last papers in our comparison results in the experiment, and we will incorporate the middle two works into our related works section in the future version.\n\n### W3\n> Minor typos. Line 137 “as an input”. Line 227 “because it doesn’t rely on any external dataset or model”.\n\nThanks for your patient review. We’ll address those issues accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJrlGW6LV9", "reviewer": "Reviewer_vaue", "summary": "This paper introduces a novel pipeline for long-tail recognition that differs from conventional strategies by focusing on dataset composition. The method focuses on the dataset and introduces a diffusion model, namely DDPM to generate data for tail classes. The analysis reveals that approximately-in-distribution (AID) samples, which slightly deviate from the real data distribution and blend class information, are essential for enhancing the generative model’s performance in long-tail classification. Experiments conducted on CIFAR 10-LT, CIFAR 100-LT, and ImageNet-LT demonstrate significant performance improvements.", "questions": "please refer to the weakness part.", "limitations": "The limitations are discussed in this paper. No potential negative social impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-organized and easy to understand.\n2.\tThe paper explores a novel perspective to tackle imbalance issue, by generating data for tail classes before training the classifier, which shows promising results compared to baseline results.\n3.\tThe paper explores the data composition, namely ID, AID and OOD, and gives a detailed analysis of the correlation between generated data composition and classification performance.", "weaknesses": "1.\tSome typos: line 137 ‘am input’\n2.\tThe baseline setup in Table 1,2,3 requires further explanation.\n3.\tThe paper mentions the time cost is four times longer than baseline, indicating that this method may be impractical for datasets with a significantly large imbalance ratio due to the extended generation and retraining times.\n4.\tSome baseline results in Tables 8 and 9 are indicated as ‘-’ and ‘ ’; an explanation for these forms of absence is missing.\n5.\tThis method focuses on generating data for tail classes, resulting in significant improvements for these classes compared to other baseline methods. A fairer comparison would be with the same pipeline using different generative models.\n6.\tA long-tailed diffusion model baseline[1] is missing.\n7.\tThe experiment are mainly conducted on manually-selected long-tailed datasets, it would improve the robustness of results by including real-world long-tailed datasets such as iNaturalist.\n\n[1] Long-tailed Diffusion Models with Oriented Calibration. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1\n> Some typos: line 137 ‘am input’\n\nThank you for your feedback; we'll fix it in the next version.\n\n### W2\n> The baseline setup in Table 1,2,3 requires further explanation.\n\nWe are sorry for the confusion. Actually, the baseline setup adheres to the code and protocols from [1], with a 200-epoch training schedule. Training is conducted with a batch size of 128 using an SGD optimizer at a learning rate of 0.1. These detailed settings can also be found in our appendix.\n\n### W3\n> The paper mentions the time cost is four times longer than baseline, indicating that this method may be impractical for datasets with a significantly large imbalance ratio due to the extended generation and retraining times.\n\nThanks for your review. We argue that since DiffuLT doesn’t rely on external knowledge like pretrained models or additional data, it is highly practical in scenarios where data is limited and that acquiring more training data is costly or impossible. In such cases, training time is not the primary concern, while acquiring satisfactory accuracy is. \n\nTo make our arguments valid, we showcase two quantitative examples: medical imaging using the KVASIR dataset and remote sensing with the EuroSAT dataset. We train the classifier on long-tail versions of these datasets with an imbalance ratio of 0.01, where models typically struggle due to the skewed distributions. \n\n| Model | Long-tailed | KVASIR Acc (%) | EuroSAT Acc (%) |\n|-----|-------------|----------------|-----------------|\n| Baseline | × | 83.8           | 93.2            |\n| Baseline | √ | 52.1           | 88.9            |\n| DiffuLT | √ | 66.8           | 92.0              |\n\nAs shown in the table, our method significantly mitigates the long-tail issue in these important scenarios, showing our great generalization ability for practical usage. We would provide more such discussions in the final version.\n\n### W4\n\n> Some baseline results in Tables 8 and 9 are indicated as ‘-’ and ‘ ’; an explanation for these forms of absence is missing.\n\nThanks for your careful review. Since the comparison results are based on the reported values in their respective papers, any results not shown in the original papers are indicated as “-”.\n\n### W5 & W6\n\n> This method focuses on generating data for tail classes, resulting in significant improvements for these classes compared to other baseline methods. A fairer comparison would be with the same pipeline using different generative models.\n\n> A long-tailed diffusion model baseline is missing.\n\nFollowing your suggestions, we compare our method with different generation models using the same pipeline in Table 5 of the paper. While T2H, as you mentioned, shows slightly better performance than CBDM, it falls short of our proposed model because it isn't primarily designed for discriminative tasks. \n\n| Method   | CIFAR100-LT Acc (%) |\n|----------|---------------------|\n| Baseline | 38.3                |\n| DDPM     | 43.8                |\n| CBDM     | 46.6                |\n| T2H      | 46.8                |\n| DiffuLT  | 51.5                |\n\nAdditionally, using pretrained diffusion models like Stable Diffusion could lead to data leakage, resulting in an unfair comparison, and they are not well-suited for data beyond natural images. We will add these comparisons in our final version.\n\n### W7\n\n> The experiment are mainly conducted on manually-selected long-tailed datasets, it would improve the robustness of results by including real-world long-tailed datasets such as iNaturalist.\n\nYou have raised an interesting point. We have tried our best to train a generation model with iNaturalist dataset, but we found that this dataset is too large that it consumes all of our GPU computing power. We are running DiffuLT pipeline with a manually sampled subset of iNaturalist, but due to time constraint, experiments are not finished at present. We promise to showcase analysis and experiment results in our final version.\n\nIn the meanwhile, since iNaturalist is similar to ImageNet-LT, both being natural image datasets, we believe the effectiveness of our model on this type of data is validated. Additionally, the results on KVASIR and EuroSAT, which are real-world long-tailed datasets that have entirely different distributions, further demonstrate the robustness of our model.\n\n[1] Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. CVPR 2020.\n\n[2] KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. MMsys 2017.\n\n[3] Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMlVgTXXyt", "reviewer": "Reviewer_VPct", "summary": "This article uses a Diffusion model to generate tail class samples, achieving balance among classes at the data level and thereby addressing the long-tail distribution problem. Additionally, the authors believe that AID samples are the most beneficial in enhancing classifier performance; therefore, they design an L_AID loss to constrain the diffusion model to generate richer AID samples. Qualitative and quantitative comparisons have been made on three benchmark datasets, showing the efficiency of their method.", "questions": "Please refer to weaknesses for corresponding questions.", "limitations": "The authors have addressed the limitation of this method.  This method will significantly increase training time.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clearly organized and well written.\n2. The experimental results look good.\n3. The motivation is reasonable.", "weaknesses": "1. The innovation is limited. Essentially, the authors' method designs a reconstruction loss to constrain the generative model to learn a distribution more consistent with real distributions, and many existing studies have proposed similar ideas.\n2. The authors emphasize that this method has generalizability in real-world applications; however, I hold a negative view on this point. I believe that the performance improvement observed in this paper is fundamentally due to training data and test data having identical distributions. Therefore, constraining the diffusion model to learn a distribution closer to training data benefits classifier learning. However, in real-world applications where limited training data may not share identical distributions with potentially infinite test data, using this method might lead to negative impacts.\n3. On what principle was Equation 4 determined? Why define [df, 2df] as AID? What impact would modifying this equation? For instance, would defining [df, \\sqrt(2)df] as AID affect performance?\n4. In Table 6, why does combining ID and AID result in worse performance than using only AID for CBDM?\n5. Authors should provide ablation studies for Nt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1\n\n> The innovation is limited. Essentially, the authors' method designs a reconstruction loss to constrain the generative model to learn a distribution more consistent with real distributions, and many existing studies have proposed similar ideas.\n\nThe focus of our approach is *not* on training a generative model to learn a distribution that closely matches real-world data. Instead, we emphasize the relationship between generative models and discriminative models, addressing two questions that are yet to be fully explored: Can a generative model *without external knowledge* improve a classification model? And *what type of generated data* is most beneficial? \n\nPrevious research has primarily relied on *pretrained* diffusion models, which we believe limits practical applicability and may lead to data leakage. We would appreciate it if you could provide examples of existing studies with similar concepts. To the best of our knowledge, no such prior art exists.\n\n### W2\n>The authors emphasize that this method has generalizability in real-world applications; however, I hold a negative view on this point. I believe that the performance improvement observed in this paper is fundamentally due to training data and test data having identical distributions. Therefore, constraining the diffusion model to learn a distribution closer to training data benefits classifier learning. However, in real-world applications where limited training data may not share identical distributions with potentially infinite test data, using this method might lead to negative impacts.\n\nAs mentioned in previous publicity in CVPR [1], the long-tailed setting assumes that the distribution of the label $p(y)$ differs between the training and test sets, but the conditional distribution $p(x|y)$ remains unchanged. These default settings hold true in current long-tailed recognition settings, and also can serve as *the potential theoretical foundation* for many long-tailed literatures [2][3]. The scenario you mentioned, where the training and test samples do not share the same distribution, might be different from the definition of long-tailed learning.\n\nTo address concerns about the potential negative impact of distribution shifts, we conducted a firmly experiments: we trained a ResNet-50 backbone on both the ImageNet-LT dataset and ImageNet-LT augmented with data $\\mathcal{D}_{gen}$ generated by our DiffuLT method. We then performed *linear probing* (the transfer learning settings) on the CIFAR100-LT with an imbalance ratio of 0.01. \n| DiffuLT | CIFAR100-LT Acc (%) |\n|---------|---------------------|\n| ×       | 62.5                |\n| √       | 70.9                |\n\nDespite the distribution differences between ImageNet-LT and CIFAR100-LT, our method still showed improvement. Therefore, we believe our method does not negatively impact model performance when distributions change.\n\nRegarding real-world applications, our paper demonstrates that our methods can benefit medical imaging, remote sensing, and general image classification under long-tailed settings. We recognize that the understanding of \"real-world application\" may vary, and we are open to further discussion if you could specify your interpretation of this term.\n\n### W3\n> On what principle was Equation 4 determined? Why define [df, 2df] as AID? What impact would modifying this equation? For instance, would defining [df, \\sqrt(2)df] as AID affect performance?\n\nThe definition of AID samples in the paper can be seen as `somewhat arbitrary'. However, we prioritized efficiency, performance, and simplicity in arriving at this definition. An ablation study shows that this approach achieves the best results with the highest efficiency.\n\n| AID range       | $‖\\mathcal{D}_{gen}‖$ | Acc (%) | $\\Delta \\text{Acc}/‖\\mathcal{D}_{gen}‖$          |\n|-----------------|--------|---------|----------------------|\n| $[d_f, \\sqrt{2}d_f]$ | 5,212   | 41.2    | $5.51\\times 10^{-4}$           |\n| $[d_f, 2d_f]$     | 11,886  | 45.2    | $5.78\\times 10^{-4} $          |\n| $[d_f, 3d_f]$     | 15,384  | 41.7    | $2.23\\times 10^{-4}  $         |\n\nWe opt not to use the first definition because the range should be as wide as possible to ensure efficient generation, while avoiding the production of too many unused samples. Although a more precise search might yield better results, we prioritize simplicity and avoid complex parameter tuning, leading us to set the region as defined.\n\n### W4\n> In Table 6, why does combining ID and AID result in worse performance than using only AID for CBDM?\n\nActually, this is one of the key ideas of our paper. In this setting, the number of generated samples is consistent, although the generation frequency varies due to the different distributions of data types. Our key observation, as shown in Table 3 of paper, is that AID samples contribute more to classification performance compared to ID samples. Since the \"ID & AID\" setting includes fewer AID samples than the \"AID\" setting, it results in a worse outcome.\n\n### W5\n> Authors should provide ablation studies for Nt.\n\nWe conducted an ablation experiment for $N_t$ under the CIFAR100-LT setting with an imbalance ratio of 0.01. \n| $N_t$  | CIFAR100-LT Acc (%) |\n|-----|---------------------|\n| 300 | 48.5                |\n| 400 | 50.4                |\n| 500 | 51.5                |\n| 600 | 51.8                |\n| 700 | 49.2                |\n\nThe performance shows only a slight improvement after $N_t > 500 $ and begins to decrease after $N_t > 600$. We believe the main reason for this is that the class with the most samples has only 500 instances, so choosing $N_t > 500$ inevitably introduces noise. When $N_t$ is too large, the impact of this noise outweighs the benefits of the newly added information.\n\n[1] Disentangling Label Distribution for Long-tailed Visual Recognition. CVPR 2021.\n\n[2] Long-tail learning via logit adjustment. ICLR 2021\n\n[3] Balanced Meta-Softmax for Long-Tailed Visual Recognition. NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work designs a diffusion based long-tail recognition method - DiffuLT, leveraging diffusion models to generate samples to balance the long-tailed dataset. Authors found that approximately-in-distribution (AID) samples are important in enriching tail class samples , and then they proposed to incorporate revised diffusion model to generate such samples. The idea is new and the performance gain is good. The authors' rebuttals have addressed most of the concerns of the reviewers.
Decision: Accept (poster)