Feedback-guided Data Synthesis for Imbalanced Classification
Authors: Reyhane Askari Hemmat
Mohammad Pezeshki
Florian Bordes
Michal Drozdzal
Adriana Romero-Soriano
Venue: Accepted by TMLR ()

Abstract: Current status quo in machine learning is to use static datasets of real images for training, which often come from long-tailed distributions. With the recent advances in generative models, researchers have started augmenting these static datasets with synthetic data, reporting moderate performance improvements on classification tasks. We hypothesize that these performance gains are limited by the lack of feedback from the classifier to the generative model, which would promote the usefulness of the generated samples to improve the classifier's performance. In this work, we introduce a framework for augmenting static datasets with useful synthetic samples, which leverages one-shot feedback from the classifier to drive the sampling of the generative model. In order for the framework to be effective, we find that the samples must be close to the support of the real data of the task at hand, and be sufficiently diverse. We validate three feedback criteria on a long-tailed dataset (ImageNet-LT, Places-LT) as well as a group-imbalanced dataset (NICO++). On ImageNet-LT, we achieve state-of-the-art results, with over $4\%$ improvement on underrepresented classes while being twice efficient in terms of the number of generated synthetic samples. Similarly, on Places-LT we achieve state-of-the-art results as well as nearly $4\%$ improvement on underrepresented classes. NICO++ also enjoys marked boosts of over $5\%$ in worst group accuracy. With these results, our framework paves the path towards effectively leveraging state-of-the-art text-to-image models as data sources that can be queried to improve downstream applications.

Reviewed on Published in Transactions on Machine Learning Research (09/2024)
Feedback-guided Data Synthesis for Imbalanced Classification
Reyhane Askari-Hemmat 1.2,3.1
Mohammad Pezeshki
Florian Bordes 1,2,3
Michal Drozdzal'
Adriana Romero-Soriano
'FAIR at Meta Mila Université de Montréal
McGill University, Canada CIFAR AI chair
Abstract
Current status quo machine learning is to use static datasets of real images for training, which often come from long-tailed distributions. With the recent advances in generative models, researchers have started augmenting these static datasets with synthetic data, re- porting moderate performance improvements on classification tasks. We hypothesize that these performance gains are limited by the lack of feedback from the classifier to the genera- tive model, which would promote the usefulness of the generated samples to improve the clas- sificr's performance. In this work, we introduce framework for augmenting static datasots with useful synthetic samples, which leverages one-shot feedback from the classifier to drive the sampling of the generative model. In order for the framework to be effective, we find that the samples must be close to the support of the real data of the task at hand, and be suf- ficiently diverse. We validate three feedback criteria on a long-tailed tailed dataset (ImageNet-LTT Places-T.T) as well as a group-imbalanced dataset (NTCO++)) ) On ImageNet-I.T, we achieve state-of-the-art results, with over 4% improvement on underrepresented classes while being twice efficient in terms of the number of generated synthetic samples. Similarly, on Places- LT we achieve state-of-ther art results as well as nearly 4% improvement on underrepresented classes. NICO++ also enjoys marked boosts of over 5% in worst group accuracy. With these results, our framework paves the path towards effectively leveraging state-of-the-arl text-to- image models as data sources that can be queried to improve downstream applications
1 Introduction
In the recent year, we have witnessed unprecedented progress in image generative models (Ho et al., 2020; Nichol al., 2022; Ramesh et al., 2021; Rombach al., 2022; Ramesh et al., 2022; Saharia al., 2022; Balaji et al., 2022; Kang et al., 2023). The photo-realistic results achieved by these models has propelled arms race towards their widespread use in content creation applications, and as byproduct, the research community has focused developing models and techniques to improve image realism (Kang et al., 2023) and conditioning generratinnggeeetinnnnnno consistency (Hu et al., 2023; Yarom et al., 2023; Xu et al., 2023). Yet, the potential for those models to become sources of data to train machine learning models is still under debate, raising intriguing questions about the qualilies that the synthetic data must possess to be effective training downstream representation learning models.
Several recent works have proposed using generative models as either data augmentation sole source of data to train machine learning models (He et al., 2023; Sariyildiz et al., 2023; Shipard et al., 2023; Bansal & Grover, 2023; Dunlap et al., 2023; Gu et al., 2023; Astolfi et al., 2023; Tian et al., 2023), reporting moderate model performance gains. These works operate in a static scenario, where the models being trained do not provide any feedback to the synthetic data collection process that would ensure the usefulness of the generated
'Equal contribution, Corresponding author: reyhsneaskari Smeta. com.
CCdee: httpa://gitie http:///ww...cbbooeeeerereeeedde www.eeddaaccck-iddda-d-dtatae Synthesi

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 1: Exemplary samples from different distributions. Subfigures show random samples for Jack- o-lanlern class coming from: (a) ImageNet-LTT; (b) Latent Diffusion Model (LDMMunclip v2-1), conditioned the text prompt Jack-o-lanterm; These exhibit a noticeable distribution mismatch with real data. (c) our pipeline, the generated samples more closcly align with the real data distribution and present higher diversily, including rare samples such as blue or white pumpkins.
samples. Instead, to achieve performance gains, the proposed approaches often rely laborious 'prompt engineering' (Gu et al., 2023) to promote synthetic data to be close to the support of the real data distribution on which the downstream representation learning model is to be deployed (Shin et al., 2023). Moreover, recent studies have highlighted the limited conditional diversily in the samples generated by state-of-hhe-rt image generative models (Hall et al., 2023; Cho et al., 2022b; Luccioni et al., 2023; Bianchi et al., 2022), which may hinder the promisc of leveraging synthetic data at scale. From these perspectives, synthetic data still falls short of real data.
Yet, the generative model literature has implicitly encouraged generating synthetic samples that are close to the support of the real data distribution by developing methods to increase the controllability of the generation process (Vendrow et al., 2023). For example, researchers have explored image generative models conditioned on images instead of only text (Casanova et al., 2021; Blattmann et al., 2022; Bordes et al., 2022). These approaches inherently offer more control over the generation process, by providing the models with rich information from a real image without relying on 'prompt engineering' (Wei al., 2022; Zhang et al., 2022; Lester et al., 2021). Similarly, the generative models literature has aimed to increase sample diversity by devising strategies to encourage models to sample from the tails of their distribution (Schwag et al., 2022; Um & Ye, 2023). However, the promise of the above-described strategies to improve representation learning is yet to be shown.
In this work, we propose to leverage the recent advances in the generative models to address the shortcomings of synthetic data representation learning, and introduce feedback from the rownst classifier model to guide the data generation process. In particular, we devise a framework which leverages pre-trained image generative model to provide useful, and diverse synthetic samples that are close to the support of the real data distribution, to improve on representation learning tasks. Since real world data is most often characterized by long tail and open-ended distributions, we focus on imbalanced dlasifiiatioon-scnariio in which different classes or groups are unequally represented, to demonstrate the effectiveness of our framework. More precisely, we conduct experiments on ImageNet Long-Tailed (ImageNet-LTT) (Liu et al., 2019), Places Long-Tailed Tailed (Places-I.T) and NICO++ (Zhang et al., 2023) and show consistent. performance gains w.r.l. prior art. Our contributions can be summarized as:
We devise diffusion model sampling strategy which leverages feedback from pretrained classifier in order to generate samples that are useful to improve its performance.
We find that for the classifier's feedback to be effective, the synthetic data must lic close to the support of the downsttream lask data distribution, and be sufficiently diverse.
thate-of-the-art results (1) on ImageNet-LII with an improvement 4% on underrepre- We report state-of the-art
sented classes while using half the amount of synthetic data than the previous stato-of-the-art; and (2) we report state-of thettrr-rrtttt results on Places-LT, LT, with an improvement of nearly 4% on underrep- resented classes; and (3) on NICO++, with improvements of over 5% in worst group accuracy.
2

---

Published in Transactions on Machine Learning Research (09/2024)
Imbalanced Data
Balanced
Classifier
C(fé)
Diffusion Model
train
Classifier
Data
real
fo
PDecal
Go
D U2 syn
S Jaam
Dual conditioning: A random sample text prompt
Figure/ 2: Overview of our framework. Given an imbalanced dataset, Dreat classifier fa(x) is initially trained. Knowing that the validation and test sets are balanced, the goal to create balanced training set using synthetic data. The Diffusion Model, Gø, is conditioned label-containing text prompt. The model's generation is also guided by feedback, C(fa), from the classifier to increase usefulness of the synthetic samples. Subsequently, (z) is retrained on the combined real and synthetic samples.
randomly selected real image and a
Through experiments, we highlight how our proposed approach can be effectively implemented to enhance the utility of synthetic data. See Figure 1 for samples from framework.
2 Background
Diffusion models. Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019) learn data distri- butions p(x)) or p(xly) by simulating the diffusion process in forward and reverse directions. In particular, Denoising Diffusson Probabilistic Models (DDPM) (Ho et al., 2020) add noise to data points in the forward process and remove it in the reverse process. The contimuous time reverse process in DDPM is given by, dx; = [f(x,, t) - g2 (t) log''(x;)) dt + g())dw, where t indexes time, and f(x;,)) t) and g(t) are drift and volatility coefficients. A neural network c (xx) is trained to predict noise in DDPM, aligning with the score function V loo (x+). Given trained model (4) (ax), Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020), more generic form of diffusion models, can generate an image To from pure noise æy by repcatedly removing noise, getting _T-1 given 2G (Song et al., 2020):
(1) arEa (xx)
It-1 Vac-1
Ei
a:-1 - of Eg (1)
(xt)
(1)
Vas
"direction pointing to æ.
prodicted mo
with a₂ and dr time-dependent coefficients and 6, 10, being standard Gaussian noise.
bussiffier-guidance in diffusion models. Guidance in latent diffusion models involves leveraging addi- tional information, such as class labels or text prompts, to condition the generated samples on. This modifies the score function as follows:
V (zgp) V logp(w) + V logp(y/>),
(2)
where is a scaling factor. The term Va log p(ylp)))) is generally modeled either as classifier-uiddacc (Dhariwal & Nichol, 2021) or classifier-free guidance (Ho & Salimans, 2022). In the Latent Diffusion Model (LDM) used in this paper, Va p(ylz) is modeled in classifier-free approach and controls its guidance strength.
3 Methodology
Figure presents an overview of our proposed approach. We assume access to a pre-tained diffusion model, which takes as input an image and text prompt, and produces an image consistent with the inputs. We train classifier fg on an imbalanced dataset of real images, Dreat This initial classifier serves as founda- tion for the subsequent generation of synthetic samples. We then collect dataset. of synthetic data, D
3

---

Published in Transactions on Machine Learning Research (09/2024)
by conditioning the pre-trained diffusion model on text prompts formatted as class-label and random images from the corresponding class. We leverage feedback signals from the pre-trained classifier to guide the sampling process of the pre-trained diffusion model, promoting useful samples for fa. Finally, we train the classifier from scratch on the union of the original real data and the generated synthotic data, Dreal U Dayn.
3.1 Increasing the Usefulness of Synthetic Data: bredback-guided Synthesis
Fcedback-guided synthesis. We propose feedback-guidance, a modification of the diffusion models' stan- dard sampling process for generating useful samples for training a classifier by getting feedback from the classifier itself. While standard classifier-guidande in diffusion models (Dhariwal & Nichol, 2021; Ho et al., 2020) focuses on high-density regions, our method prioritizes the generation of useful samples, by leveraging feedback from our pre-trained classifier fø- Leveraging classifier feedback allows for systematic approach for generating useful samples that provide gradient for the classification task at hand.
Our proposed feedback-guidance might be reminiscent of classifer--guidance in diffusion models (Dhariwal & Nichol, 2021; Ho et al., 2020), which drives the sampling process of the generative model to produce images that are close to the distribution modes. The proposed boedback-guidanne literature aiming to increase sample diversity in diffusion models (Schwag et al., 2022; Um & Ye, 2023), whose goal is to drive the sampling process of the generative model Lowards low density regions of the learned distribution. Instead, the goal of our proposed geedback-guidance is to synthesize samples which are useful for a classifier to improve its performance. Furthermore, our framework is inspired by the literature in active learning (Wang et al., 2016; Wu et al., 2022), where useful samples are selected from a pool of samples, whereas we propose to generate useful samples.
is also related to the
Formally, let Dreat be a training dataset of real data, So a classifier, and Gg a state-of-the-art pre-trained diffusion model. We start by training fo on Dreat, and define h {0, 1) as a binary variable that describes whether a sample is useful for the classifier So not. Our goal is to generate samples from specific class that are informative, i.e. from the distribution of p(z((,,)) v). To generate samples using the reverse sampling process defined in section 2, we need to compute V, log p(z/h, v). Following Eq. 2, we have:
V log (zlh,, v) V. (og + v log po + ~ C(,,
(3)
where C(2, y, fo) is criterion function approximating the sample usefulness (h), and w is a scaling factor that controls the strength of the signal from our criterion function. Note that by using a pre-trained diffusion model, we have access to the estimated class conditional score function V, log po(rly) as well as the estimated unconditional score function Va log (x)). The derivation of Eq. 3 is presented in Appendix A.1.
3.1.1 Feedback Criteria C(x, fo)
We explore criteria functions that promote generating samples which are informative and challenging for the classifier. We consider three feedback criteria: (1) the classifier's loss on the generated samples; (2) the classiffer's prediction entropy on the generated samples; and (3) the hardness score (Sehwag et al., 2022).
Classifier Loss. To focus on generating samples that pose challenge for the classifier for we use the classifier's loss the criterion function for the feedback guided sampling. Formally, we define C(x,, y, fo) in terms of the loss function C as:
Cla, fo) = L(fo(z).
(4)
Since L is the negative log- ligeliiel,hood, and following Eq. 3, we have:
V (og y) V log (o()) + V (og - w. log (v(la)).
(5)
Note that Po is the probability distribution modeled by the generative model and Po is the probability distribution modeled by the classifier fo. We are effectively moving towards space where under the classifier Ja the samples have lower probability", but simultaneously the term Vz log (o))ax))))) which is modeled by the generative model, ensures that the samples belong to class y.
2This is in contrast classifier guidance, which directs the sampling process towards examples that have high probability under class y-
4

---

Published in Transactions on Machine Learning Research (09/2024)
Increase in
Increase in y
Increase in
Inerease in a
Inerease in
Increase
(a) C = Loss
(b) C Entropy
(c) C Hardness
Figure 3: A grid of images generated for the class "African Chamcloon' using different criterion for feedback guidance. Along the x-axis, from left to right conditional guidance scale or the y in Eq. 3 is increased, while along the y-axis, from top to bottom, the classifier-feed random seed consistent across all images, ensuring that any observed variations are only due to changes in the guidance scales. Moving from left to right, it is evident that increasing " results in samples that are more faithful to the Affricnn Chameleon' class. However, this comes at the cost of generating very typical, easily classifiable images. Conversely, as we move from top to bottom, increasing the classffer-feedback guidance results in the generation of more 'challenging' or atypical images of African chameleon. However, this may come at the cost of moving away from the class "African Chameleon", thus is important to tune and w generate useful samples.
guidance scale or the w in Eq.
increased. The
Entropy. Another measure for the usefulness of the generated samples is the entropy (Shannon, 2001) of the output class distributions for x prodicted by fo(x). Entropy is a common mcasure that quantifics the uncertainty of the classifier on a sample x (Wang el al., 2016; Sorscher et al., 2022; Simsek et al., 2022). We adopt entropy as a criterion, H(f((x)), as higher entropy leads to generating more informative samples. Following Eq. 3, have,
V (x = Vz log + v log + HV)(((()))).
This sampling method encourages the generation of samples for which the classifier fo has low confidence in its predictions.
Hardness Score. The Hardness score (Schwag et al., 2022) quantifics how difficult or informative a sample (z, y) for given classifier fø. Tt is defined as:
1 (1) My) + In(dot(E,)) In(2r))
HS,
where My and Ey are the sample mean and sample covariance for embeddings of class y and ls is the dimension of embedding space. We directly adopt the Hardness score as criterion;
Vz (olh, v) = V. (og + vz (og + WV WV HS, fo).
This sampling procedure promotes generating samples that challenging for the classifier fa.
To showcase the interplay between and w and how changing the critcrion function C affects the sampling process, we plot grids of synthetically generated images in Figure 3. All samples are conditioned to generate an "African Chameleon', with variations in (tedback-guidande scales based on three different criteria: en- tropy, loss, and hardness. ? serves the role of ensuring that the generated images are faithful the visual characteristies of an 'African Chameleon', such as color patterns, skin details, or posture. On the other hand, relies the uncertainties in the classifier's predictions to guide the generative model.
5

---

Published in Transactions on Machine Learning Research (09/2024)
(a) sorrel
(b) carpenter's plane
(c) toucan
Figure 4: In each figure: first row depicts samples from Imagenct-LT, second row shows samples from LDM conditioned on text prompt (class label), and third row displays samples generated by conditioning the LDM on both text prompt and the image embedding of a random real sample from the class. (a) Homonym ambiguity: the text prompt sorrel refers to either the sorrel horse or the sorrel plant. However, in the Imagenet-LT dataset class sorrel refers to the horse while LDM generates the plant. (b) Text misinterprotation: LDM may interpret the text prompt incorrectly. For example the class carpenter's 's plan refers to carpentering tool, while LDM generates wooden planes. (c) Stylistic Bias: When prompted with Toucan, LDM mostly generates drawings of the bird toucan. However, in the training distribution, we mostly observe real birds. Conditioning on random training image allows the LDM to generate samples with the correct style. See Section 3.2 and also Figures 7, 8.
3.1.2 Feedback-guided Synthesis in Latent Diffusion Models
To apply feudback-guided synthesis latent diffusion models (LDMss) we need to compute the criteria function (z)) at each step of the reverse sampling process with the minor change that the diffusion is applied on the latent variables z. However, the classifier fe operates naive implementation of feedback-guided sampling would require a full reverse chain to find 2o, which would then be dccoded to find zo to finally compute (((())). Therefore, to reduce the computational cost, instcad of applying the full reverse chain, we use the DDIM predicted (or equivalently predicted un in Eq. 1) at each step of the reverse process. We find that this approach is computationally much cheaper and is highly effective.
the pixel space Consequently, a
3.2 Towards Synthetic Generations Lying within the Distribution of Real Data
We identify three scenarios where using only text prompt results in synthetic samples that are not close to the real data used to train machine learning downstream models:
Homonym ambiguity. A single text prompt can have multiple meanings. For example, consider generating data for the class sorrel that could either refer to the sorrel horse or the sorrel plant. See Figure 4 (a) for further illustrations.
Text misinterpretation. The text-to-image generative model can produce images which are se- mantically inconsistent partially consistent with the input prompt. An example of that is the class carpenter's plane from Lm. When prompted with this term, the diffusion model generated images of wooden planes instcad of the intended carpentry tools. See Figure 4 (b) for further illustrations.
Stylistic Bias. The generative model can produce images with a particular style for some prompts, which does not match the style of the real data. For instance, the Toucan images in the ImageNet-I. dataset are mostly real photographs, but the generative model frequently outputs drawings of this species of bird. See Figure (c) for further illustrations. Also more samples in Figures 7, 8.

---

Published in Transactions on Machine Learning Research (09/2024)
Conditioning on: text prompt: "Perssia cat"
Image embedding a real sample
(a) Without random dropout (p=0)
(b) With random dropout p=0.2
(c) With random dropout p=0.4
(d) With random dropout p-0.6
(e) With random dropout p-0.8
(1) With random dropout p-1.0
Figure 5: Synthetic sample generation using text prompt and image embedding. We plot different levels of dropout on the image embedding. We condition on a single random real sample (plotted on the left most). As observed, using only image conditioning and text (a), we observe very low diversity in the generations. As we increase the dropout probability, we observe more diversity. If we only condition on the text prompt (f), we also observe low diversity.
To alleviate the above-described issues, we borrow from the generative models' literature a dual-conditioning technique (Mishra et al., 2023). In this approach, the generator conditioned on both a text descriptor containing the class label and a randomly selected real image from the same class in the real training dataset. This additional layer of conditioning steers the diffusion model to generate samples which are more similar to those in the real training data; see Figure 4, to contrast samples from text-conditional models with those of text--nnd-nngg--ooniiiiie
models. Using the unCLIP model (Rombach et al., 2022), the noise prediction network (xx) in Equation extended to be a function of the conditioning image's embedding, denoted (t) Co (x,, Zeoud), where Zeond is the CLIP embedding of the conditioning image.
3.3 Increasing the Conditional Diversity of Synthetic Data
As discussed in Section 3.2 leveraging conditioning from real images to synthesize data results in generating samples that are closer to the real data distribution. However, this comes at the cost of limiting the generative model's ability to produce diverse images. Yet, such diversity essential to train rewnstream classification models. We propose to apply random dropout on image embedding. Dropout a tochnique used for preventing over-fitting by randomly setting fraction of input units 0 each update during training time (Srivastava et al., 2014). In this setup, the application of dropout serves different yet equally crucial purpose: enhancing the diversity of generated images.
By applying random dropout to the embedding of the conditioning image, we effectively introduce variability into the information that guides the generative model. This stochasticity breaks the deterministic link between the conditioning image and the generated sample, thereby promoting diversity in the generated images. For instance, if the conditioning image contains a Persian Cat with specific set of features (e.g., shape, color, background), dropout might millify some of these features in the embedding, leading
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Classification accuracy on ImageNet-LLT, All models are trained using ResNext-50 backbone, unless marked with ** Models marked with use ViT-B16 architature and are additionally initialized with CLIP (Radford al., 2021) weights.
Method
# Syn. data
ImageNet-LT
Overall
Many
Medium
Few
ERM (Park et al., 2022)
X
43.43
65.31
36.46
8.15
Decouple-cRT (Kang et al., 2020)
X
47.3
58.8
44.0
26.1
Decouple-LWS (Kang al., 2020)
X
47.7
57.1
29.3
Remix (Chou et al., 2020)
X
48.6
60.4
46.9
30.7
Balanced Softmax (Ren et al., 2020)
X
51.0
60.9
32.1
CMO (3 experts) (Park et al., 2022)
X 56.2
66.4
53.9
35.6
Mix-Up GLMC (Du et al., 2023)
x 57.21
64.76
55.67
42.19
VL-LTR"" (Tian al., 2022)
X
77.2
84.5
74.6
59.3
FillUUp (Shin et al., 2023)
2.6M
63.7
69.0
62.3
54.6
LDM (txt)
1.3M
57.90
64.77
54.62
50.30
LDM (txt and img)
1.3M
58.92
56.81
64.46
51.10
LDM-FG (Loss)
1.3M
60.41
66.14
57.68
54.1
LDM-FG (Hardness)
1.3M
56.70
58.07
55.38
57.32
LDM-FG (Entropy)
1.3M
64.7
69.8
62.3
59.1
LDM-FG (Entropy) VL-LTR"
1.3M
77.27
83.59
75.57
64.31
the generative model to explore other plausible variations of Persian Cat. Intuitively, this diverse set of generated samples, which now contain both the core characteristics of the class and various incidental features, better prepares the downstream classification models for real-world scenarios where data can be highly heterogeneous. See Figure
4 Experiments
We explore two classification scenarios: (1) class-imbalanced classification, where we generate synthetic samples create balancod representation across classes; and (2) group-imbalanced classification, where leverage synthetic data. achieve balanced distribution across groups.
4.1 ImageNet-LT: ilaalaimbll Classification
Dataset. The ImageNett-ongg" (ImageNet-LTT) dataset (Liu et al., 2019) is subset of the original ImageNet (Deng et al., 2009) consisting of 115.8K images distributed non-uniformly across 1,000 classes. In this dataset, the number of images per class ranges from a minimum of to a maximum of 1,280. However, the test and validation sets are balanced. In line with related literature (Shin et al., 2023), our goal is synthesize missing data points in way that, when combined with the real data, results a uniform distribution of examples across all classes.
Experimental setup. We leverage the pre-trained state-of-the-art image-and-text conditional L.DM v2-1- unclip (Rombach et al., 2022) to sample from. We adopt the widely used ResNext50 architecture well as the ViT-B16 model as the classifiers for our experiments on ImageNet-LT, For ResNext50, our classifier is trained for 150 epochs, for ViT-B models, the classifier is trained with real data for a total of 100 epochs and then tunet using real and synthetic data for another 10 epochs. To improve model scaling with synthetic data, we modify the training process include 50% real and 50% synthetic samples in each mini-batch. We apply balanced mini- batch approach when training all LDM methods. We also use the balanced Softmax (Ren al., 2020) loss when training the classifier. For additional experimental details Appendix F.
3rhii change boosts the performance by nearly points on ResNext50.

---

Published in Transactions on Machine Learning Research (09/2024)
Metrics. We follow the standard protocol and report the overall average accuracy across classes, and the stratified accuracy across classes Many (any class with over 100 samples), Medium (any class with 100-20 samples), and Few (any class with less than 20 samples).
Baselines. We compare the proposed approach with prior art which does not leverage synthetic data from pre-trained generative models and with recent literature which does. Furthermore, we compare against methods that leverage pretrained CLIP weights (Radford et al., 2021). We also compare the proposed approach with vanilla sampling LDM that uses only the text prompts* results of our proposed framework for the three feedback guidance techniques introdueed in section 3, namely, Loss, Hardness and Entropy. When leveraging synthetic data, we balance ImageNet-LT by generating as many samples as required to obtain 1,300 examples per class.
conditioning. We report the
Discussion. Table 1 presents the results, see further analysis on the performances of the model as the number of synthetic data scales in Appendix D. We observe that approaches which do not leverage generated data exhibit the lowest, accuracies across all groups, and overall, with Mix-Up obtaining the best results. It is worth noting that Mix- Up leverages advanced data augmentation strategies the real data, which result in new samples. When comparing frameworks which use generated data from state of the art generative models, our framework surpasses the LDM baseline. Notably, the LDM with Feedback-Guidance (LDM-FG) based on the entropy criteria increases the LDM baseline performance ~ 5 points overall and, perhaps more interestingly, these improvements translate into a ~ 9 points boost on classes Few. Our best LDM-FG FG also surpasses the most recent competitor, Fill-Up (Shin et al., 2023), by 1% accuracy point overall while using a half the amount of synthetic images. This highlights the importance of generating samples that are close to the real data distribution-an as Fill-Up does but also improving their diversity and usefulness. Furthermore, we showcase that since our method is data-driven, it can be combined with any existing algorithmic or architectural approaches. Specifically, we compare against recent benchmarks that leverage gesual-inguuiste long-tailed recognition along with initializing from CLIP weights (Tian et al,, 2022). We observe that additional synthotic data with Feodback Guidance using entropy outperforms the bascline VL- LTR specifically we improve the performance on classes few by points and on classes medium by point.
4.2 Places-LT: Class-imbalanced Classification
Dataset. To further study the effect of feed-back guidance on different datasets, we study the Places- Long tailed dataset (Liu et al., 2019). This dataset consists of 365 classes where the minimum number of examples in class is 5 and the maximum is 4980. However, test and validation sets are balanced across classes. Similar to ImageNet-LT, we aim to synthesize data points in way that, when combined with the real data, results in a uniform distribution across all classes.
Experimental setup. We leverage the pre-trained state-of-the-art image-and-text conditional LDM v2-1- unclip (Rombach et al., 2022) to sample from. We follow the common practice in the literature (Kang et al., 2020; Liu et al., 2019) and use a pretrained ResNet-152. We apply two stages of training (Ren et al., 2020). Stage one where the model is trained for 30 epochs. Once the base model is obtained, we further fine-tune the last layer of the model for 10 epochs using Meta Sampler and Balanced Softmax (Ren et al., 2020).
Metrics. Similar to ImageNet-LL,, we report the overall average accuracy as well as the stratified accuracy across classes Many (any class with over 100 samples), Medium (any class with 100-20 samples), and Few (any class with less than 20 samples).
Baselines. We compare against previous methods that do not use synthetic data as well as the recently proposed Fill-Up (Shin et al., 2023) that uses synthetic data. We achieve state of the art results overall as well as on classes medium, and few. When leveraging synthetic data, we balance Places-LT by generating as many samples as required to obtain 4980 examples per class.
Discussion. Table 2 summarizes the results. We observe that using syuthotic data increases the overall performance. Comparing Fill-up and Feedback Guidance, we see that using any type of feedback guidance greatly boosts the performance on classes Few with Entropy out-performing our feedback criteria.
4Text prompts are the format class-label.
9

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Classification accuracy on Places-LT using ResNet-152.
Method
# Syn. data
Places-Ie LT
Overall
Many
Medium
Few
ERM (Cui et al., 2021)
x
30.2
45.7
27.3
8.2
Decouple-TWS (Kang et al.; 2020)
X
37.6
40.6
39.1
28.6
Balanced Softmax (Ren et al., 2020)
X
38.6
42.0
39.3
30.5
ResL. (Cui et al., 2022)
X
39.8
39.8
43.6
31.4
MISLAS (Zhong et al., 2021)
X
40.4
39.6
43.3
36,1
Fill-Up (Shin et al., 2023)
1.8M
42.6
45.7
43.7
35.1
LDM-FG (Loss)
1.8M
42.3
41.2
44.4
39.8
LDM-FG (Hardness)
1.8
41.8
40.7
44.2
38,5
LDM-FG (Entropy)
1.8M
42.8
41.7
44.9
40.0
4.3 NICO++: Group-imbalanced Classification
Dataset. We follow the sub-population shift setup of NICO++ et al., 2023; Yang et al., 2023) which contains 62,657 training examples, 8,726 validation and 17,483 test examples. This dataset contains 60 classes animals and objects within 6 different contexts (autumn, dim, grass, outdoor, rock, water). The pair of class-context is called a group, and the dataset imbalanced accross groups. the training set, the maximum number of examples in group is 811 and the minimum O. For synthetic samples generated using framework see Figure 11.
Experimental setup. We again leverage the pre-trained state-of-the-art image-and-text conditional LDM v2-1-unclip (Rombach et al., 2022) as high performant generative model to sample from. Since some groups in the dataset do not contain any real examples, we cannot condition the LDM model on random images from group, and so instead, we condition the LDM on random in-class examples. We adopt the ResNet50 (He al., 2016) architecture as the classifier, given its ubiquitous use in prior literature. For each baseline, we train the classifier with five different. random seeds.
Metrics. Following prior work on sub-population shift (Yang et al., 2023; Sagawa et al., 2019), report worst-group accuracy (WGA) as the benchmark metric. We also report overall accuracy.
Baselincs. We compare our framework against the NICO++ benchmarks (Zhang et al., 2023; Yang et al., 2023). These stale-of-the-art methods may leverage data augmentation but do not rely on syuthetic data from generative models. We also consider vanilla LDM baselines conditioned on text prompt, and report results for all three criteria. We generate the text prompts class-label in context. We balance the NICO+ dataset such that each group has 811 samples, overall we generate 229k samples.
Discussion. Table 3 presents the average performance across five random seeds of our method in contrast with previous works. As shown in the table, our method achieves remarkable improvements over prior art which does not leverage synthetic data from generative models. More precisely, we observe notable WGA improvements of ~ 6% over the best previously reported results on the ResNet architecture. When comparing against basclines which do leverage synthetic data from statc-of-the-art generative models, our framework remains competitive in terms of average accuracy, and importantly, surpasses the ERM baseline in terms of WGA by over 14%. The improvements over LDM further emphasize the benefit of generating samples that are close to the real data distribution, are diverse and useful when leveraging generated data for representation learning. Notably, the improvements achieved by our framework are observed for all the explored criteria, although entropy consistently yields the best results.
Also note that our framework tackles the group-imbalanced problem from the data perspective. Any algo- rithmic approach such Bsoftmax (Ren al., 2020), IRM (Arjovsky et al., 2019), or GroupDRO (Sagawa al., 2019) can be applied on top of approach.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Classification average and worst group accuracy on NICO+ dataset using ResNet55 pretrained on Imagenet.
Algorithm
# Syn. data
Avg. Accuracy
Worst Group Accuracy
ERM (Yang et al., 2023)
X
85.3 + 0.3
35.0 4.1
Mixup (Zhang et al., 2017)
X
84.0 + 0.6
42.7 + 1.4
GroupDRO (Sagawa
al., 2019)
X
82.2 + 0.4
37.8 1.8
IRM (Arjovsky et al., 2019)
X
84.4 + 0.7
40.0 0.0
LISA (Yao et al., 2022)
X
84.7 + 0.3
42.7 2.2
BSoftmax (Ren et al., 2020)
X
84.0 + 0.5
40.4 0.3
CRT (Kang et al., 2019)
X
85.2 + 0.3
43.3 1 2.7
LDM
229k
86.02 + 1.14
32.66 1.33
LDM FG (Loss)
229k
84.55 + 0.20
45.60 0.54
LDM FG (Hardness)
229k
84.66 1 0.34
40.80 + 0.97
LDM FG (Entropy)
229k
85.31 + 0.30
49.20 = 0.97
Table 4: Ablation of our framework based on LDM. Results are computed w.r.t. the real balanced validation set of the ImageNetLT. All hyper-paramcters for cach sctup tuned.
Text
Img
dropout
FG
FID4
Density
Coverage
Avg. / Few validation acc.
x
18.46
0.962
0.690
59.52 / 50.74
/
X
14.24
1.019
0.676
59.95 / 49.5
/
/
13.63
1.063
0.722
60.16
54.79
/
V
Loss
18.48
0.867
0.639
62.19
54.78
/
/
Hardness
10.84
1.070
0.820
57.7
56.57
/
/
Entopy
21.36
0.821
0.614
65.7 57.7
4.4 Ablations
To validate the effect of dual image-text conditioning, dropout on the image conditioning embodding, and feedback-guidance, we perform an ablation study and report Fréchet Inception Distance (FID) (Heusel et al., 2017), density and coverage (Naeem et al., 2020), and average accuracy overall and on the classes Few. FID and density serve proxy to measure how close the generated samples are to the real data distribution. Coverage serves as proxy for diversity, and accuracy improvement for usefulness. FID, density and coverage are computed by generating 20 samples per class and using the ImageNet-T.T validation set (20,000 samples) as reference, The accuracies are computed on the ImageNet-LT validation set. As shown in the Table 4, leveraging the vanilla sampling strategy of an LDM conditioned on text only (row 1) results in the worse performance across metrics. By leveraging image and text conditioning simultaneously (row 2). we improve both FID and density, suggesting that generated samples closer to the ImageNet-I.T validation set. When applying dropout to the image embedding (row 3), we observe positive effect both FID and coverage, indicating higher diversity of the generated samples. Finally, when adding feedback signals (rows 4-6), we notice the highest accuracy improvements (comparing to the model trained only on real data) both on average (except for hardncss) and on the classes Fow, highlighting the importance of leveraging fredback-guidand to improve the usefulness of the samples for representation learning downstream tasks. is important note that quality and diversity metrics such as FID, density and coverage may be reflective of the usefulness of the generated synthetic samples (compare Hardness row with the Entropy row in Table 4).
5 Related work
Synthetic Data for Training. Synthetic data serves as a readily accessible source of data the absence of real data. One line previous works have used graphic-based synthetic datasets that are usually crafted through pre-defined pipelines using specific data sources such as 3D models game engines (Peng et al.,
11

---

Published in Transactions on Machine Learning Research (09/2024)
2017: Richter et al., 2016; Bordes et al., 2023; Richter et al., 2016; Peng et al., 2017) as synthetic data to train downstream models. Despite their utility, these methods suffer from multiple limitations, such as a quality gap with real-world data. Another set of prior work has considered using pre-trained GANs as data generators to yield augmentations to improve representation learning (Astolfi et al., 2023; Chai et al., 2021); with only moderate success as the generated samples still exhibited limited diversity (Jahanian et al., 2021). More recent studies have begun to leverage pre-trained text-to-image diffusion models for improved classification (He et al., 2023; Sariyildiz et al., 2023; Shipard et al., 2023; Bansal & Grover, 2023; Dunlap et al., 2023) other downstream tasks (Wu et al., 2023; You et al., 2023; Lin et al., 2023; Tian et al., 2023). For example, Imagen model (Saharia et al., 2022) was fine-tuned by (Azizi et al., 2023) to improve Imagenet (Deng et al., 2009) per-class accuracy. Diffusion classifier (Li et al., 2023), pre-trained diffusion model adapled for the lask of classification. Other work (He el al., 2023) adapted a diffusion model to generate images for fine-tuning CLIP (Radford et al., 2021). Furthermore, it is shown that prompt- enginocring can lead to a training data from which transferable representations can be learnod(Sariyiliizz et al., 2023). These studies suggest that. it is possible to use features extracted from synthetic data for several downstream tasks. However, consistent with our findings, other work (He et al., 2023; Shin et al., 2023) argue that these methods may produce noisy, non-representative samples when applied to data-limited scenarios, such as those with long-tailed and open-ended distributions. Use of textual inversion (Dhariwal & Nichol, 2021) in an image synthesis pipeline for long-tailed scenarios for improved domain alignment and classification accuracy was also proposed (Shin et al., 2023).
Balancing Methods for Imbalanced Datasets. An effective strategy for mitigating class imbalance include balancing the dataset (Shelke et al., 2017; Idrissi et al., 2022; Shen et al., 2016; Park et al., 2022; Liu et al., 2019). Dataset balancing can either involve up-sampling the minority classes to bring about a uniform class/ distribution or sub-sampling the majority classes to match the size of the smallest class/group. Traditional up-sampling methods usually involve either replicating minority samples or through simple methods such as linearly interpolating between them. However, such simple up-sampling techniques have been found to be less effective in scenarios with limited data (Hu et al., 2015). Sub-sampling is generally more effective, but it carries the risk of overfitting due to reduced dataset size. Another line of research focuses re- re-weighting lechniques (Sagawa el al., 2019; Samuel & Chechik, 2021; Cao et al., 2019; Ren et al., 2020; Cui et al., 2019). These methods scale the importance of underrepresented classes or groups according to specific criterion, such as their count in the dataset or the loss incurred during training (Liu et al., 2021). Some approaches adapt the loss function itself or introduce a regularization technique to achieve a more balanced classífication performance (Ryou et al., 2019; Lin et al., 2017; Pezeshki et al., 2021). Another effective method is the Balanced Softmax (Ren al., 2020) that adjusts the biases in the softmax layer of the classifier to counteract imbalances in class distribution. Furthermore, to enhance the minority over- sampling, one can augment a variety of minority samples using the extensive context provided by majority class images as a backdrop (Park et al., 2022).
6 Conclusion and Discussion
We introduced a framework that leverages pre-trained classifier together with a state-of-the-art text-and- image generative model to extend challenging long-tailed datasets with useful, diverse synthetic samples that are close to the real data distribution, with the goal of improving on downstream classification tasks. We achieved usefulness by incorporating feedback signals from the downstream classifier into the generative model; we employed dual image-text conditioning to generate samples that are close to the real data manifold and we improved the diversily of the generated samples by applying dropout to the image conditioning embedding. We substantiated the effect of each of the components in our framework through ablation studies. We validated the proposed framework on ImageNet-LT, Places-LT and NICO+ consistently surpassing prior art with notable improvements.
Overall, our framework provides a data approach for imbalanced classification tasks, where accessing real data is expensive or infeasible. Using an off-tho-shelf pre-trained diffusion model can be viewed as accessing a compressed form of large-scale data. Additionally, our feedback-guided sampling technique enables the extraction of useful information for the task at hand. It is important to note, that our approach is com-
12

---

Published in Transactions on Machine Learning Research (09/2024)
plementary to any existing algorithmic approaches that target imbalanced classification. One can combine both data and algorithmic solutions to a given problem to further improve the results.
Limitations. Our approach inherits the limitations of the state-of-the-art generative models, and the image realism and representation diversity are determined by the abilities of the generative model - our guidance mechanism can only explore the data manifold already captured by the model.
Future work. In our work, we only consider one feedback cycle from a fully trained classifier. We leave for future exploration setup where the feedback from the classifier is being sent to the generator online while the representation learning model is being trained. Furthermore, any differentiable criterion function can be applied in the sampling. For example one can consider an active learning acquisition function as a leedback criterion function (Gal et al., 2017; Cho et al., 2022a).
7 Broader Impact Statement
Generative models are known come with various risks and biases. We highlight that the negative impacts associated with these large scale diffusion models underscore the need for robust ethical guidelines, regula- tory oversight, and technological safeguards. In this work, we do not directly address or focus the biases that may exist in the generations of the diffusion model such as ethical concerns, or societal biases. However, we leverage large-scale pretrained generative models and propose an inference-time intervention to generate useful synthetic examples for imbalanced classification problems. From the classification perspective, we proposed method aimed at decreasing biases due to long-tail data. distribution with a positive impact on model fairness. It is important to note that while our approach contributes mitigating certain biases, it does not eliminate all potential biases of generalive process. We advocate for the continued development of comprehensive ethical frameworks and regulatory measures to address the broader impact of using gener- ative models in different applications. Our work is a step towards improving fairness in machine learning, understanding that it is part of an ongoing elfort to build more responsible AI.
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 2019.
Pictro Astolfi, Arantxa Casanova, Jakob Verbeek, Pascal Vincent, Adriana Romero-Soriano, and Michal Drozdzal. Instance-conditioned gan data augmentation for representation learning, 2023.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08 084666 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-inage diffusion models with ensemble of expert denoisers. arXiv preprint ar arXix:2211.0132 2022.
Hritik Bansal and Aditya Grover. Leaving Reality to Imagination: Robust Classification via Generated Datasets. In International Conference on Learning Representations (ICLR) Workshop, 2023.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale, 2022.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi-paramett neural image synthesis. Advances in Neural Information Processing Systems, 11, 2022.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised representation knows about. Transactions on Machine Learning Research, 2022. URL https: / //openreview wtwpiiiww.eeeeeeeeeeeeeeeeeeee
13

---

Published in Transactions on Machine Learning Research (09/2024)
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learning. arxiv preprinl arXiv:2308.0399 033777 2023.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribttion-wwar margin loss. Conference on Neural Information Processing Systems (NeurIPS), 2019.
Arantxa Casanova, Marlenc Carcil, Jakob Verbeck, Michal Drozdzal, and Adriana Romero Soriano. Instance- conditioned gan. Advances in Newral Information Processing Systems, 34:27517- 2222717--7529, 2021.
Lucy Chai, Jun Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Jae Won Cho, Dong-Jin Kim, Yunjae Jung, and In So Kweon. Medal: Maximum classifier discrepancy for active learning. IEEE transactions on neural networks and learning systems, 2022a.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. 2022b.
Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. Tn European Conference on Computer Vision (ECCV) Workshop, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long tailed
recognition. IEFE Transactions on Pattern Analysis andd Machine Intelligence (TPAMT), 2022.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balancee loss based on effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jia Deng, Wei Dong, Richard Socher, Ti-Jia Li, Kai Li, and Fei-Fei Ti. ImageNet: A large-scale hierarchical image database, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Fei Du, Peng Yang, Qi Jia, Fengtao Nan, Xiaoting Chen, and Yun Yang. Global and local mixture consistency cumulative learning for long-tailed visual recognitions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 15814-111111 15823,, 2023.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based angmentation. arXiv preprint arXiv:2305.1222 2023.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183 11922 PMLR, 2017.
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307. 12980, 2023.
Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. Dig in: Evaluating disparities in image generations with indicators for geographic diversity, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ruifei Hc, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XIAOJUAN QI. IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? International Conference on Learning Representations (ICT.R, 2023.
14

---

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Conference on Published in Transactions on Machine Learning Research (09/2024)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambaner, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to nash equilibrium. CoRR, abs/1706.08500, 2017. URL http:://aaii...o//aab////a////// http///b/bb//000500000000
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiw:2207.15500 2022.
Newral Information Processing Systems (NeurIPS), 2020.
Yong Hu, Dongla Guo, Zengwei Fan, Chen Dong, Qiuhong Huang, Shengkai Xie, Guifang Liu, Jing Tan, Boping Li, Qiwei Xie, et al. An improved algorithm for imbalanced data and small sample size classification. Journal of Data Analysis and Information Processing, 3(03):27, 2015.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint wrXiv:2303.11897 2023.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive grapl-group-cccuracy.. In Conference on Causal Learning and Reasoning, pp. 336 351. PMLR, 2022.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning. arXiv preprint arXiv:2106.05:0 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albort Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recoguition. arXie preprint 1rx: 19100099.9 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling Representation and Classifier for Loig-Tailed Recognition. In International Conference on Learning Represenlations (ICLR), 2020.
Minguk Kang, Jun-Yau Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for Text-to-Iman Image Synthesis. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition CaciiiionnCCE (CVPR), 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-offiiien prompt tuning. arXiv preprinl ar Xix:22100 08661, 2021.
Alexander C Li, Mihir Prabludesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your Diffusion Model is Secretly Sero-Shot Classifier. arXiv preprint arXiv:2303. 16203, 2023.
Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the Power of Synthetic Data on Few-shot Object Detection. arXiv preprint arXiw:230.. 230.22333., 2023.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In JEEE International Conference on Computer Vision (ICCV), 2017.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-678 6792. PMLR, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recogration (CVPR), 2019.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite, Stable bias: Analyzing societal representations in diffusion models, 2023.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Divyanshu Mishra, He Zhao, Pramit Saha, Aris T Papageorghion, and J Alison Noble. Dual conditioned diffusion models for out-of-distribution detection: Application to fetal ultrasound videos. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 216 226. Springer, 2023.
Muhammaad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. 2020.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorcalistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learning (ICML), 2022.
Seulki Park, Youngkyu Hong, Byeongho Hco, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arviv preprint ar 710.000000000. 010.06924, 2017.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Tajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256 1272, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 2021.
PMLR,
Aditya Ramesh, Mikhail Pavlov, Gabricl Goh, Scott Gray, Cholsea Voss, Alec Radford, Mark Chen, and Ilya Sulskever. Zero-shot Lexl-Lo-image generation. In International Conference on Machine Leurning (ICML), 2021.
Aditya Ramesh, Prafulla. Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint ar 2026v22204.06125, 2022.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33:4175 74166,4166--0 2020.
Stephan Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision (ECCV), 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rosolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Serim Ryou, Seong-Gyun Jeong, and Pietro Perona. Anchor loss: Modulating loss scale based on prediction difficulty. In IEEE International Conference on Computer Vision (ICCV), 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ar preprint wrXiv:1911.0879/ 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic Text-to- Image Diffusion Models with Deep Language Understanding. ar preprint arXiv:2205.11487 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. orXiv preprint arXiv22202.00522 2022.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In IEEE International Conference Computer Vision (ICCV), 2021.
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic ImageNet clones. In IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), 2023.
Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton Ferrer. Generating high fidelity data from low-density regions using diffusion models, 2022.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile compuling and communications review, 5(1):3 2001.
Mayuri S Shelke, Prashant R Deshmukh, and Vijaya K Shandilya. A review on imbalanced data handling using undersampling and oversampling technique. Int. J. Recent Trends Eng. Res, 3(4))444-449, 2017.
Li Shen, Zhouchen Lin, and Qingming Huang, Relay backpropagation for effective learning of deep convo- lutional neural networks. In European Conference on Compuler Vision (ECCV), 2016.
Joonghyuk Shin, Minguk Kang, and Jaesik Park. Fill-up: Balancing long-tailed data with generative models. arXiv preprint orXiv:2306.0728 07200, 2023.
Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Diversity is Definitely Needed: Improving Model-Agnostie Zero-shot Classification via Stable Diffusion, 2023.
Berfin Simsek, Melissa Hall, and Levent Sagun. Understanding out-of-distribution accuracies through quan- tifying difficulty of test samples. arXiv preprint arXiv:2203.15100 2022.
Jascha Sohl- Dickstic,,,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning (ICML), 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference on Learning Represenlations (ICLR), 2020.
Yang Song and Stelano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neurol information processing systems, 32, 2019.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. 2023.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523 19536, 2022.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Changyao Tian, Wenhai Wang, Xizhon Zhu, Jifeng Dai, and Yu Qiao. VI-Itr: Learning class-wisc visual- linguistic representation for laig-tailed visual recognition. In European Conference on Computer Vision (ECCV), 2022.
Yonglong Tian, Tijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners, 2023.
Soobin Um and Jong Chul Ye. Don't play favorites: Minority guidance for diffusion models. arXiv preprint arXiv:2301.1239), 2023.
Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.0r11 07865, 2023.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuils and Systems for Video Technology, 27(12):2591-2600, 2016.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Fd Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Jiaxi Wu, Jiaxin Chen, and Di Huang. Entropy-based active learning for object detection with progres- sive diversity constraint. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 9397-9406, 2022.
Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixcl-level annotations for semantic segmentation using diffusion modcls. ar preprint WNiv:2303.11681, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong, Imagereward: Learning and evaluating human preferences for -ext-to-image generation. arXiv preprint arXiv:2304.0500 2023.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look at subpopulation shift. arXiv preprint arXie:2302.1220 2023.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out- of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharomi, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Sxpektor. What you is what you read? improving text-image alignment evaluation. arXiv preprint 0rXic:2305.10400 2023.
Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Lú, and Jun Zhu. Diffusion Models and Semi- Supervised Learners Benefit Mutually with Few Labels. arXie preprint arXiv:2302. 10586, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 771..090044.. 2017.
Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico+ Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 18036-16044 2023.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint ar Xrxi::2200. 034939 2022.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for taiget recognition. In IEEE Conference on Computer Vision and Pattern Recognâtion (CVPR), 2021.
18

---

Published in Transactions on Machine Learning Research (09/2024)
A Appendix
A.1 Derivation of equation 3
Simply writing the Bayes rule, we have:
p(a, h,
p(hla, y) ()p(<)l))(())))) p(hly)p(v)
V logp(x//,, v) = v log
= Vr log
p(h, v)
= V. log or y y)p(())) = Vz logp(aly) + Vz log (()p())
(9)
Note that by using a pre-trained diffusion model, we have access to the estimated class conditional score function Va log Po (zly)- We then assume there cxists a critcrion function C(æ, fo) that evaluates the usefulness of a sample æ based on the classifier fe- Consequently, we model p(h|z, y) as:
phh|z, y) exp (C(x, y. Sa))
(10)
Z
where Z is a normalizing constant. As a result, we can generate useful samples based on the criteria function C(x, y, fo), and following Eq. 9, we have:
V Ao (x) (x = V log pa(x))) + w (x,
(11)
where w a scaling factor that controls the strength of the signal from our criterion function C. Following Eq. 2, we have,
Vx log v) V (o) Vog fo V.C,,,,
(12)
A.2 Toy 2-dimensional Example of Criteria Guidance
Figure illustrates the experimental results using a simple 2-dimensional dataset for a classification task. The dataset contains two classes represented by blue and red data points. Within each class, the data consists of two modes: the majority mode, containing 90% of the data points, and the minority mode, which holds the remaining 10%. We initially train diffusion model on this dataset. Sampling from the trained diffusion model generates synthetic data that closely follows the distribution of the original training data, showing an imbalance between the modes of each class (see Figure (b)).
To encourage gencration of data from the mode with lower density, we introduce a binary variable into the model. this context, = 0 indicates the minority mode, while 1 signifies the majority mode. Following the criteria guidance discussed in Section 3.1.1, without retraining the model, we modify the sampling process that the generator is guided towards generating samples of higher entropy. To that end, we train a linear classifier for several epochs until it effectively classifies the majority mode, but the decision boundary intersects the minority mode, resulting in misclassification of those points. Teveraging the classifier's uncertainty around this decision boundary, we guide the generative model to produce higher entropy samples. This results in more synthetic samples being generated from the minority modes each class (see Figure 6 (c)).
Integrating this synthetic data with the original data produces more uniform distribution across modes, leading more balanced classifier. This is desirable in our context it mitigates the biases inherent in the original dataset and improves the model's generalization.
B Samples Stylistic Bias
Stylistic Bias is one of the challenges in synthetic data generation where the generative model consistently produces images with particular style for some prompts, which does not correspond to the style of the real data. This results in mismatch between synthetic and real data, potentially impacting the performance of machine learning models trained on such data.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Decision boundary
14
(a) Real data
(b) DDPM regular sampling
(c) DDPM feedback guided sampling
Figure 6: Experimental results 2-dimensional classification dalaset showcasing the effect of feedback- guided sampling. Panel (a): Real data consists of two classes represented by blue and red data points. Within each class, two modes are identified: a majority mode comprising 90% of the data and a minority mode containing the remaining 10%. Panel (b): The syuthetic data generated by regular sampling of DDPM replicates the imbalances of the original dataset. Panel (c): Synthetic data generated after modifying the diffusion model guided by feedback from a linear classifier. Feedback-guided sampling leads to more samples being generated from the minority modes. Combining with real data, it results in more balanced data with increased representation from the minority mode, ultimately improving classifier performance.
Figure 7: Here we plot more samples from Imagenet- where stylistic bias appears in synthetic generations. This scenario arises when the generative model produces images with particular style for some prompts, which does not match the style of the real data. See Section 3.2 for more details. (a) real samples from class scorpian, (b) synthtic samples using LDM, (c) synthetic samples with image and text conditioning.
20

---

Published in Transactions on Machine Learning Research (09/2024)
(c)
Figure 8: Here we plot more samples from Imagenet- where stylistic bias appears in synthetic generations. This scenario arises when the generative model produces images with particular style for some prompts, which does not match the style of the real data. See Section 3.2 for more details. (a) real samples from class triceratops, (b) synthetic samples using LDM, (c) synthetic samples with image and text conditioning.
21

---

Published in Transactions on Machine Learning Research (09/2024)
C Samples of Feedback Guided Sampling
In this section we provide more synthetic samples using Feedback guided sampling. See Figure 9 and 11 for more details.
Figure 9: Synthetic samples of two different classes of Tel Column 1: rocking chair, Column 2: flower pot. First row: Real samples from Imagenet-LTT Second row: synthetic samples vanilla LDM. Third row: synthotic samples using Entropy as the guidance.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 10: Synthetic samples of three different classes of NICO++. Column 1: class ship in context autumn,
Column 2: class eat in context autumn, Column 3: class robbit in autumn. First row: Real samples from NICO++, Second row: synthetic samples LDM. Third row: synthetic samples using Entropy as the guidance and dropout.
23

---

Published in Transactions on Machine Learning Research (09/2024)
FIEE
Figure 11: Synthetic samples of three different classes of Places-LT dataset. Column class bedroom, Column 2: class corridor, Column 3: class cottage. First row: Real samples from Places-LT, Second row: synthetic samples using Entropy as the guidance and dropout image embeddings.
24

---

Published in Transactions on Machine Learning Research (09/2024)
D Impact of the number of generated images
classifier trained ImageNet-LI when using different amount of generaled images. We show that adding generated synthelic data significantly help to increase the overall performance of the model. In addition, we observe significant gain on the few classes which highlight that generated images are well-suited for imbalanced real data sccnarios.
Figure 12 shows the performances
Tost accuracy on classes modium
Test acturacy on classes few
Fill-up
60
50
GLMIC
Mix-Upp GLMM
80
Balanced Softmax
50
Balanced Soltmax
50
45
DecompleLWWS
10
LDM-TG Entropy
LDM-PG Entropy
"S0O
2'6
10K"
"00K""
Number of synihetic sumples
Number synthelic sumples
20
Test avcuracy clusses many
Fill-up
Fill-up
68
Test accuracy ovorall
66
ERM
GEMC
Up GLM
35
Balanced Softmax
Brlent Remix
Soltaa
Decaupl-LWS
Remix
Decoople-WWS
Entropy
Entropy
10K
T0OK
B00
IM
2.6M
10K
100K
500K
1M
2AM
Number of synthetic samples
Number synthetic samples
Test accuracy depending on the number of generated synthetic data used to train the classifier. The first curve shows the accuracy the class Few, while the second shows the accuracy the class Medium and the last one shows the accuracy on class Many. Our method significantly outperforms Fill- Up(Shin et al., 2023) while using less synthetic data.
Figure 12:
E Impact of using balanced softmax with synthetic data
In experiments, have used balanced softmax to train our classifier to increase the performances class Few. We also ran experiments using weighted average of a traditional cross-entropy loss using balanced softmax with the same loss without balanced softmax. In this experiment, we added scalar coefficient a which controls the weight of the balanced softmax loss in contrast to the standard loss. In Figure 13, we plot the test accuracy with respect to this balanced softmax weight. Without balanced softmax, the accuracy on the class many is extremely high while the accuracy on class few much lower. However by increasing the balanced softmax coefficient, we significantly increase the performances on class few and medium as well the overall accuracy. However, this comes at the price of lower performance for classes Many.
F Additional Experiments and Details
General setup in sample generation We use the LDM v2-1-unelip (Rombach et al., 2022) as the state- of-the-art latent diffusion model that supports dual image-text conditioning, We use a pretrained classifier on the real data to guide the sampling process of the LDM. For Imagenet-LT, the classifier trained using ERM with learning rate of 0.1 (decaying) and weight-decay of 0,0005 and batch-size of 32. For NICO++ we pre-trained classifier Imagnet and then fine-tune it NICO++.
25

---

Published in Transactions on Machine Learning Research (09/2024)
75
70
63
Test eecuraccy
60
45
40
Overall
Mony
35
0.0
0.2
0.1
0.6
0.8
1.0
Balance softmax coefficient
Figure 13: Test accuracy Few, Modium, Many and Overall with respect to the balanced softmax coefficient. All the models use 1.3M synthetic samples.
We apply 30 steps of reverse diffusion during the sampling. To apply different criteria in the sampling process, use the pretrained classifier on the real data. For lower computational complexity, use float16 datatype in PyTorch. Furthermore, 30 reverse steps, we only compute and apply the criterion 6 times. Through experiments we find that 5 is optimal as the generated samples look very similar to applying the criterion in every step. For the hardness criterion in Eq. 3.1.1 where we need the Hy and Σy for each class, we pre-compute these values. We compute the mean and covariance inverse the feature representation of the classifier over all real samples. These values are then loaded and used during the sampling process.
apply the gradient of the criterion function every 5 steps. So for
F.1 Computational Cost
In this work, we use state of the art generative models to synthesize samples for an imbalanced classification problem. Since we leverage feedback from the classifier to the generative model, our framework requires pretrained classifier, once the samples generated, the classifier is retrained from scratch. Depending the size of the dataset and the architecture used for the classifier, and the number of hyper-parameters to tune the classifier, the computational cost varies. The generation time for synthetics samples are studied more carefully below. However, the dataset is generated only once and it can be reused for luning the classifier in the second stage.
For generations, we leverage DDIM sampling and we apply the feedback guidance every steps during the sampling process. We calculate the average wall-clock time (in seconds) per sample generation. Results are reported in Table 5, average of 1000 samples, computed the same model and the same GPU machinc (V100) without batch-generation. Faster GPU machincs with larger RAM result in faster sample generation.
Note that results are reported per sample generation, one can potentially run the sampling process in parallel. We consider the improvement the sampling complexity as a future direction of research. Several approaches can be taken to improve the sampling time complexity:
Using model distillation for diffusion models such consistency models (Song et al., 2023)
progressive distillation (Salimans & Ho, 2022) that significantly reduce the sampling time. Reducing the frequency of guidance to minimize the number of queries to the classifier.
Reusing the gradients of the classifier guidance criterion the sampling process during steps that we skip the computation. This can help with reducing the frequency of applying the guidance.
26

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Average generation time per sample in seconds. Results are reported on an average 1000 samples with float16 precision. Values are subject to change on different GPU machines and different computation precision.
Method
Time (seconds)
LDM
3.819 0.011
LDM-FG (Loss)
21.817 0.009
LDM-FG (Hardness)
21.937 + 0.014
LDM-FG (Entropy)
+ 0.012
F.2 ImageNet-LT
We follow the setup in previous work (Kang et al., 2019) and use ResNext50 architecture. We apply the balanced softmax for all the LDM models reported for Imagenet-LT. We train the classifier for 150 epochs with batch size of 512. We also use standard data augmentations such random cropping, color- color-jittering, blur and grayscale during training.
Furthermore, below we provide an ablation study on combining different criteria. Specifically, we study the scenario where each sample is generated with 0.33 probability from each of the three given criteria (Entropy, Loss and Hardness). We upsample the Imagenet-LT datasot to 1.3M samples. As seen in Table 6, entropy outperforms other criteria and the combination of criteria.
Table Classification accuracy on ImageNet-LT using ResNext50 backbone. different criteria..
Ablation on combining
Method
# Syn. data
ImageNet-LT
Overall
Many
Medium
Few
LDM-FG (Loss)
1.3M
60.41
66. 14
57.68
54.1
LDM-FG (Hardness)
1.3M
56.70
58.07
55.38
57.32
LDM-FG (Entropy)
1.3M
64.7
69.8
62.3
59.1
LDM-FG (Combined)
1.3M
62.38
67.66
59.96
56.23
F.3 NICO+
We follow the setup in previous work (Yang et al., 2023), where pre-trained ResNet50 model on ImageNet is used for all the methods. We assume access to the attributes labels (contexts). For training our LDM model we only apply ERM without any extra algorithmic changes. We use the SGD with momentum of 0.9 and train for 50 epochs. We apply standard data augmentation such as resize and center crop and apply ImageNet statistics normalization. For every method, we try 10 sets of hyper-parameters (learning rate, batch-size") (Yang et al., 2023). We perform model selection and early stopping based on average validation accuracy. We then train the selected model random seeds and report the test performance.
In Table 7 provide ablation analysis on the hyper-perameter that controls the strength of the feedback guidance. The rest of ceper-parametery in the generative model or the classifier are tuned for every set-up. As it can be seen, among the w values that we tried, all improved the worst-group-accurae compared to the case where w = 0, however the best results are achieved with w.0.33.
F.4 Places-LT
We follow the selup in the literature (Kang et al., 2020; Liu et al., 2019) and use ResNet-152 pretrained on Iet. We apply two stages of training (Ren et al., 2020). Stage one where the model is trained for
TI Learningg randomly selected from 10 Uniform(-4--22 and batch-size randomly selected from 2 2niform(667).
27

---

Published in Transactions on Machine Learning Research (09/2024)
Table 7: Ablation study on the NICO+ dataset with the entropy as feedback guidance. In each setup, we select the ayper-parameter based on average accuracy on the validation and report the worst-group accuracy on the test set.
Feedback guidance coefficient
Worst-group accuracy
32.66 + 1.33
0.01
45.60 + 2.63
W 0.03
49.20 + 0.97
w 0.05
12.10 + 1.84
30 epochs. Once the base model is obtained, further fine-tune the last layer of the model for 10 epochs using Meta Sampler and Balanced Softmax (Ren et al., 2020). We use the SGD optimizer with learning rale of 0.0028, momentum of 0.9 and weight decay 0.0005 and batch-size of 512. During generalion, we apply dropout on image-embedding with 0.5 probability. Furthermore, we apply 30 steps of DDIM with clip-guidance scale of and foedback guidance scale of 0.03.
28


Reviews:
[{"review_id": "cBhQZGTj1N", "reviewer": "Reviewer_zK2a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper presents a empirically-driven pipeline that seems intuitively matched to the task at hand. The reported experiments show state-of-the-art results. \n- The paper cites a decent amount of related literature, and discusses them at a good amount of detail (although since this is not my area, I am not sure if they missed fundamental work).\n- The text and illustrations are clear and effectively describe the method and results.\n- Ablation is performed well. E.g. Table 1 and 2,3 and 4 shows that Entropy is probably the most important/useful conditioning signal (for the downstream task). This would not be obvious if the authors had chosen to omit the results using Loss and Hardness. \n\nWeaknesses:\n- What does the classical Data Processing Inequality say about trying to get more useful information out of the dataset that was used to train the generative model, than the dataset itself? If a generative model $f$ is trained on a dataset $X$, then the information contained within $f$ cannot be greater than the information in $X$. This would seem to imply that rather than using the generative model to make synthetic examples, we should just use the original data used to train the generative model instead. Why then does using a generative model for synthetic data work? I am not familiar with this area so it is possible that there are some fundamental works you should cite here to clarify the setup in light of the Data Processing Inequality. I tick \"no\" to claims and evidence primarily for this point - it could be an easy fix for the authors to respond to, perhaps a few sentences in the first paragraph of section 5.\n- A key finding that is mentioned several times is that \"the synthetic data must lie close to the support of the downstream task\" and \"be sufficiently diverse\". I am not sure why this is surprising, or interesting. Isn't this rather an assumption of training an ML model? We classically assume data comes iid (and is therefore sufficiently diverse) from a fixed distribution (with a fixed support). Somehow this insight is already baked into the classical empirical risk minimisation setup. What is the new insight?\n- No theory or conceptual analysis is provided, beyond the rough intuition understood from Figure 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Some recent computer vision models use a generative model to augment static datasets with synthetic data. Current practice is limited by the lack of a feedback mechanism between the generative model and the final classifier. The authors use one-shot feedback from the classifier to the generative model to drive sampling useful synthetic examples. Long-tailed and group-imbalanced datasets are used as an evaluation, and often shows state of the art results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my questions above, particularly the Data Processing Inequality. Also please address the broader impact concerns below.", "broader_impact_concerns": "The method concerns generating samples in an imbalanced data regime (containing potentially protected classes), and using those synthetic generated examples to classify synthetic data. I would expect that such a model would have potential for serious intentional misuse (e.g. targetting minorities) or potential for serious unintentional misuse (e.g. exacerbating bias in imbalanced data with the false belief that this bias has been fixed). Sadly, neither of these broader impacts are even mentioned, let alone discussed. The only evaluations that are considered are classification accuracy and generative metrics (e.g. FID, density, coverage). I do not work in generative models so I am not sure, but I would have thought that broader impacts are a necessary consideration for such work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ueo2APrvax", "reviewer": "Reviewer_sT9Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The method is simple but quite effective in terms of empirical gain.\n- The paper is well written and easy to understand\n\n**Weaknesses**\n- The method does not have noticeable technical contribution. It is a combination of existing components or heuristic proposals (e.g., the three feedback criteria)\n- Mixed empirical gains (not noticeable gain in Places-LT). But there is not much of discussion (although there is discussion subsections) for not compelling empirical gains by the proposed method.\n- There is no quantitative measure for the diversity.  Without the metric, it is difficult to argue the generated samples are diverse.\n- Lack of theoretical argument of why the proposed method works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to generate samples for long-tailed classes to supplement the scarcity of samples by using feedback from the classifiers. Specifically, the feedback from the classifier is used as an input to the diffusion model to generate samples to supplement the dataset that the classifier is updated on. There are three types of feedback -- (1) classifier loss (cross entropy), (2) entropy of output class distributions and (3) hardness score (by Sehwag et al., 2022). By empirical validations with ImageNet-LT, Places-LT, NICO++, the proposed method improves the state of the arts by large margins in two datasets (ImageNet-LT, NICO++) but not with Places-LT.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Suggestions**\n- It may be worthwhile to try to learn the weighted combination of proposed three scores into a learned score (learning the mixing coefficients for the three scores to get a single score)\n- It would be great to have a quantitative metric for measuring diversity of samples\n\n**Minor suggestions**\n- in Sec. 3.2, there is a mention about \"a dual-conditioning technique\" without a reference. Please add citation for it.\n- In the paragraph in the section 1, starting with \"Yet, the generative model literature...\", there is 'prompt engineering' phrase. The first quotation mark should be ` in LaTeX instead of '.", "broader_impact_concerns": "There is no ethical implication of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UEllBPXkzo", "reviewer": "Reviewer_RUw5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong Aspects:\n\n1. **Innovative Approach**: The submission introduces a novel framework that leverages feedback from classifiers to guide the synthesis of synthetic data, which is a creative approach to addressing the issue of imbalanced classification.\n\n2. **Extensive Experiments and Ablation Study**: The authors conduct comprehensive experiments and an ablation study to demonstrate the impact of each component of their framework, providing strong evidence of its efficacy.\n\n3. **Addressing Specific Challenges**: The submission tackles specific challenges in synthetic data generation, such as homonym ambiguity, text misinterpretation, and stylistic bias, with innovative solutions.\n\nWeaknesses:\n\n1. **The Necessity of Classifier Guidance**: A pivotal aspect of the methodology is the feedback from classifiers to guide the generation. However, this occasionally appears to be ineffective. Notably, the performance of LDM-FG, as outlined in Tables 2 and 3, does not present a competitive improvement. Particularly in Table 3, where the best result of LDM-FG is worse than that of vanilla LDM in terms of average accuracy. This observation raises questions about the utility of implementing such a complex procedure.\n\n2. **Evaluation of Guidance Strategies**: The concept of employing auxiliary information to steer the generation of synthetic data is not unprecedented. For instance, previous work [1] has leveraged the capabilities of the pre-trained CLIP model to direct image generation tasks for classification of small-scale datasets. It prompts an inquiry into whether the approach of utilizing a classifier for guidance could outperform such established methods.\n\n3. **Comparative Analysis with Alternative Approaches**: The research incorporates external data and models as part of its methodology, yet the resulting performance does not markedly exceed that don't utilize external resources, and is worse than methods using external data. Specifically, reference [2] demonstrates a significant improvement in results through the use of external data for model training. So when you use external information like LDM, the result so far is not promising.\n\n[1]: Zhang, Yifan, et al. \"Expanding small-scale datasets with guided imagination.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2]: Tian, Changyao, et al. \"Vl-ltr: Learning class-wise visual-linguistic representation for long-tailed visual recognition.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission presents a novel framework that addresses the challenges of imbalanced classification in machine learning by augmenting static datasets with synthetic samples. The key contributions and new knowledge introduced are as follows:\n\n1. **Feedback-guided Data Synthesis**: The authors propose a method that leverages one-shot feedback from a classifier to guide the sampling process of a generative model. This approach aims to synthesize samples that are useful for improving the classifier's performance, particularly for underrepresented classes.\n\n2. **Framework Effectiveness**: The framework is designed to create synthetic samples that are close to the support of the real data distribution and sufficiently diverse. The authors validate the effectiveness of their approach on long-tailed datasets such as ImageNet-LT and Places-LT, as well as a group-imbalanced dataset NICO++.\n\n3. **Feedback Criteria**: The authors explore three feedback criteria for sample generation - classifier's loss, prediction entropy, and hardness score. These criteria help in identifying and generating samples that are informative and challenging for the classifier.\n\n4. **Dual Conditioning for Realism and Diversity**: To ensure that the synthetic samples align with the real data distribution, the authors use dual conditioning with both text prompts and real image embeddings. This technique helps to mitigate issues such as homonym ambiguity, text misinterpretation, and stylistic bias.\n\n5. **Enhancing Diversity through Dropout**: To increase the diversity of synthetic samples, the authors propose applying random dropout on the image embedding. This method introduces variability in the generation process, leading to a more diverse set of samples that better represent the real-world data distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposed Adjustments:\n\n1. **Classifier Guidance (Critical)**:\n   - The submission should investigate and address the instances where classifier guidance does not yield competitive improvements. \n   - It is essential to ensure that the classifier guidance is robust and consistently beneficial across various scenarios and datasets.\n\n2. **Benchmarking Against Established Methods (Critical)**:\n   - The authors should compare their guidance strategy with established methods, such as those leveraging pre-trained CLIP models, to demonstrate the superiority or unique benefits of their approach.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the approach is simple and achieves good results at least in some important cases such as worst class accuracy and on some datasets made overall improvements on the performance. After author feedback, all reviewers converged to leaning acceptance of the paper, believing that although the method is simple the results are convincing enough to be published.
Decision: Accept (poster)