Generative Data Mining with Longtail-Guided Diffusion
Authors: David S Hayden
Mao Ye
Timur Garipov
Gregory P. Meyer
Carl Vondrick
Zhao Chen
Yuning Chai
Eric M Wolff
Siddhartha Srinivasa
Venue: ICML 2025 poster (2025)
Keywords: Synthetic Data
Longtail
Long Tail
Foundation Model
Diffusion
Guidance
VLM
CLIP
Embedding
Text
Robustness
Uncertainty
Epistemic
Aleatoric
Text
Autolabel
Imagine
Imagination
Dream
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Proactive longtail discovery and mitigation with model-guided synthetic training data generation

Abstract: It is difficult to anticipate the myriad challenges that a predictive model will encounter once deployed. Common practice entails a reactive, cyclical approach: model deployment, data mining, and retraining. We instead develop a proactive longtail discovery process by imagining additional data during training. In particular, we develop general model-based longtail signals, including a differentiable, single forward pass formulation of epistemic uncertainty that does not impact model parameters or predictive performance but can flag rare or hard inputs. We leverage these signals as guidance to generate additional training data from a latent diffusion model in a process we call Longtail Guidance (LTG). Crucially, we can perform LTG without retraining the diffusion model or the predictive model, and we do not need to expose the predictive model to intermediate diffusion states. Data generated by LTG exhibit semantically meaningful variation, yield significant generalization improvements on numerous image classification benchmarks, and can be analyzed by a VLM to proactively discover, textually explain, and address conceptual gaps in a deployed predictive model.

Generative Data Mining with Longtail-Guided Diffusion
David S. Hayden
Mao Ye
Timur Garipov2 Gregory P. Meyer' Carl Vondrick"
Zhao Chen
Yuning Chai
Eric Wolff Siddhartha S. Srinivasa
Abstract
Longtal Guidance
It is difficult to anticipate the myriad challenges that predictive model will encounter once de- ployed. Common practice entails a reactive, cycli- cal approach: model deployment, data mining. and retraining. We instead develop proactive longtail discovery process by imagining addi- tional data during training. In particular, we de- velop general model-based longtail signals, in- cluding differentiable, single forward pass for- mulation of epistemic uncertainty that does not impact model parameters or predictive perfor- mance but can flag rare or hard inputs. We lever- age these signals as guidance generate addi- tional training data from latent diffusion model in process we call Longtail Guidance (LTG). Crucially, we can perform LTG without retraining the diffusion model or the predictive model, and we do not need to expose the predictive model to intermediate diffusion states. Data generated by LTG exhibit semantically meaningful variation. yield significant generalization improvements on numerous image classification benchmarks, and can be analyzed by a VLM to proactively discover, textually explain, and address conceptual gaps in a deployed predictive model.
Figure I. Difffssion with Longtail Guidance (LTG) generates syn- thetic data that are difficult or rare for an existing predictive model. The predictive model he fine-tuned on this data improve gen- eralization performance and the the synthetic data can be analyzed to understand conceptual gaps in the predictive model. Guided generations exhibit more extreme views compared to unguided gen- erations. Results in Section 4 use low-to-mid guidance weights.
Accuracy
P(Class)
Accuracy, P(Class
0.5
0.5
5
1.0
0.4
1. Introduction
16
0.8
Longtail encounters are common in production ML systems but difficult to anticipate and be robust towards. Recently, the answer has been: add more training data and more com- pute (Kaplan al., 2020). However, the time and cost to acquire additional data, especially longtail data, can be pro- hibitive (e.g. train instances occurs time for every 10 car instances in Berkeley DeepDrive (Yu et al., 2020)). Addi-
0.1
0 20 40
(Epistemic) Weight
(Entropy) Weight
Figure 2. Predictive model performance (left axes, red) and model- based longtail signals fül (right axes, green) on synthetic data generated by varying Longtail Guidance weights model-based longtail signals: Epistemic (left plot) and Entropy (right plot), Guided generations predictably exhibit lower correct class prob- ability, P(Class), lower accuracy, and higher longtail signals, f compared to unguided generations (zero guidance weight), indicat- ing they are more difficult and more longtail from the predictive model's perspective. We ensure that longtail-guided synthetic data generations do not stray out-of -iitrii-iiiittoinn by ensuring P(Class) lower than unguided generations but well above zero probability.
Cruise, LLC. San Francisco, CA OOpenAl, San Francisco, CA Colombia University, New York, NY "Uwwrrk, Palo Alto, CA Meta, Menlo Park, CA. Correspondence to: David Hayden <dyved. getcruise.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Generative Data Mining with -angtail-Guided Diffusion
Predirtio Model S
Prodicime Mide fn
Eicholling
Main Toes
Tuhed
Miin Head
Bankhoon
21:
Desncaing Moid to
Deooder 17
Badiltone
Epiitozic Hleae
Onade Main
Locatall Supeii
Grad
Loss
Tost Ensodar T
photo of.
Gradbent
Data
Thitring Thats
Auguacated Tisisiaa Data
Fine-Tune Predictiva Modal with Epistomic Longtall Signal
Ganerate Additional Byrthalic Training Data with Longtal Guidaane
Figure 3.
In Longtail Guidance, we iteratively fine-tune existing predictive model fa with longtail signal f (left) then freeze model weights & and generate synthetic data with latent diffusion model eo guided by f that is, by definition, rare or disproportionately hard for fo (right). Synthetic data are added fo training set, and the process repeats.
tional compute can traded for improved performance; see distillation (Yu et al., 2023b; Gou et al., 2021) and synthetic data generation (Du et al., 2024; Zhang et al., 2023b) for offline approaches, chain of thought (Yao et al., 2024; Wei al., 2022) for online approaches. While online approaches show promise, they are not applicable to real-time, safety- critical systems such as autonomous driving, which have strict parameter count and inference latency budgets.
Du et al., 2024) can generate synthetic data that improves model generalization performance up to a point, but they do not condition on the deployed predictive model.
Following, we develop model-basel longtail signals (Sec- tion 2), including a lightweight Epistemic Head (Sec- tion 2.1), that effectively indicate an example as being hard or rare while keeping the original predictive model intact. We then develop Longtail Guidance (Section 3), a simple method for coupling Internet-scale knowledge in diffusion models with the specific struggles of existing predictive model. We show that synthetic data produced with Longtail Guidance yields outsized generalization improvements over existing synthetic data generation approaches and that these synthetic data exhibit semantically meaningful variations, including occlusion and extreme views, see Figures 1, 2. and Section for comparisons. In summary, we contribute:
Production ML systems are iteratively developed a con- tinuous, reactive cycle of: collect data, deploy model, en- counter longtail edge cases, repeat. We wish to expedite this cycle and move it towards proactive approach of longtail discovery. Existing offline approaches appeal to larger mod- els (knowledge distillation, synthetic data generation) or summarize training data (data distillation). Yet, they do not directly account for what is rare or difficult for an existing, deppend predictive model.
1. Epistemic Head: differentiable, single forward- pass formulation of epistemic uncertainty that does not im- pact existing model weights or performance, detects rare or hard examples, and guides data generation to- wards high-value training examples.
We wish to trade additional offline compute for performance by generatively mining additional training data that is hard or rare from the perspective of an existing predictive model. Distinct from adversarial training and robustness approaches (Bansal & Grover, 2023; Zhang et al., 2022a; Gowal et al., 2021; Hendrycks et al., 2021) which imbue models with invariance distribution shift or small, often imperceptible changes, aim generate semantically meaningful, in- distribution examples that cause high model uncertainty.
Longtail Guidance (LTG): a latent diffusion guidance technique that generates high-value, difficult and/or rare training examples from the perspective of an exist- ing predictive model. It requires no changes in training for the diffusion model or the predictive model.
Foundation generative models such as Stable Diffusion (Rombach et al., 2022), Pixart (Chen et al., 2025), GPT (Brown et al., 2020; OpenAI, 2023), and Claude (Anthropic, 2024) are exposed to Internet-scall data, but it is not ob- vious how to use them to generate data that is specifically challenging to an existing predictive model. We seek to explicitly couple the vast knowledge contained in founda- tion generative models with the specific challenges faced by an existing predictive model. But how can we ensure that the generative model produces data that is relevant (has meaningful learning signal) given predictive model? Existing approaches. like prompt tuning and reasoning in an embedding space (such CLIP) (Zhang al., 2023b;
3. Longtail Introspection: through VLM analysis of LTG- generated data, we find keywords describing what an existing predictive model struggles with. Keywords can be used to prompt diffusion models for high-value data or to inform future real data collection.
2. Model-Based Longtail Signals
Longtail is a suitcase word (Minsky, 2006) it does not have single definition. Two reasonable definitions include:
Rare: An input is longtail if instances similar to it are
2

---

Generative Data Mining with Longtail-Guides Diffusion
or hard. To address longtail scenarios, we would ideally account for both. To do so, we decompose predictive uncer- tainty into two components: epistemic and aleatoric (Fig- ure 4). For test input with unknown label y and predictive model parameters & drawn from distribution +, epistemic uncertainty is defined as (Depeweg et al., 2018),
Epistemic
Classes
Aleatoric
Classes
Fly: +) = U()) - E (M(y +)]
(2)
where we implicitly condition on in each term, and where Ui is uncertainty measure, such entropy for discrete y or variance for continuous y. If entropy, Eqn 2 is equivalent to measuring the mutual information between parameter distribution & and unknown target y, which informs us about how much knowing about one tells us about the other.
vs aleatoric Egg (y | 6)] uncer- lainty. Epistemic increases with distance from the data manifold. Aleatoric increases with proximity to the decision boundary.
Figure 4. Epistemic E(y. o)
rare in training data. This definition naturally captures a data-centric view in which events are longtail if their overall occurrence rare (independent of any model).
In principle, epistemic uncertainty can be computed by sam- pling from the posterior predictive distribution:
Hard: An input is longtail if it is disproportionately difficult for a given model to correctly reason about.
/ p(y Φ p(ó) Tarain Strain) do.
ply x) =
(3)
The first term is a likelihood for test label y and the sec- ond term is a posterior distribution for model parameters & conditioned on all previous training data (Imain Frain)..
We develop differentiable model-based longtail signals that can flag both rare and hard instances. We then use model- based longtail signals offline to generate additional longtail synthetic training data that provide outsized generalization improvements. Following, we introduce the Epistemic Head, a lightweight addition to any predictive model that does not impact predictive performance but provides superior longtail signals based on detecting hard or rare examples.
Performing the inference required to sample from Eqn 3 is intractable for modern neural networks. Nevertheless, it can be approximated. One approach is variational infer- ence, typified by Monte Carlo dropout (Gal & Ghahramani, 2016), where the same input is passed through network many times (often 50 or more forward passes), with test time variation in each pass enabled by random Bernoulli dropout. Unfortunately. this would be expensive to compute and is not conveniently differentiable due to discrete sam- pling (Jang et al., 2016). It is also bested in practical terms by a model ensemble, where small number of models, often 3 - 5, are trained (independently (Lakshminaray et al., 2017) or otherwise (Maddox et al., 2019)). However, computing a gradient (which we will need to generate long- tail synthetic data) across K instances of a model introduces substantial memory overhead and compute latency.
We wish develop longtail signals that are general to loss, architecture (transformer, convolutional), or output space (discrete, continuous). Idcally, they do not affect model performance or require changes to model training. Obvious candidates would be an uncertainty measure of the output, such as entropy or variance. Others can be gathered from the anomaly and out-of-distribution detection literature. One is the Helmholtz free energy, defined the negative log partition function of an energy model p(y x)).
E(x) -ogg { Fla.s) dy'
(1)
In Figure 3 (left), we introduce a lightweight ensembling technique called the Epistemic Head, which provides su- perior longtail signal with no impact model performance and negligible changes to model parameter count, training time, and inference time (see Supplement A.5). Inspired by LoRA (Hu et al., 2021) and the Hydra architecture (Tran et al., 2020), we duplicate the head of an existing predic- tion model K times and jointly train with the same loss as the base model but with diversity encouraged through an oracle loss (Guzman-Riven et al., 2012) that only propa- gates per-example loss through the best-performing head. Head outputs act as fixed-point samples from the posterior predictive (Eqn 3) under a prior determined by weight ini- tialization and permit differentiable computation of longtail signals in a single forward pass, including Fly,, t).
where æ is some input with energy y. This energy can be computed in typical classification models as the nega- tive, temperature-scaled log-sum-exp of the logits, E(x)= -Tlog Σ, efi(t)/T where fa()) is the 2thh logit of pre- dictive model [z) that forms a probability distribution by softmax. Energy was shown to be effective in discriminating in-distribution and out-of-distribution data (Liu et al., 2020). Like entropy and variance, it requires no model adjustment or retraining. We use entropy and energy as baselines.
2.1. Epistemic Head
Traditional measures of predictive uncertainty, like entropy, do not distinguish what rare from what is ambiguous
3

---

schedule Generative Data Mining with Longtail-Guided Diffusion
Rare
Hard
sampled noise at each step rather than the process mean. In DDIM, the reverse process is instead modeled Markovian. At each step, it predicts.
Epistemic Head
2 - -a,- 120 + + 0,c.
(7)
Epistemic
Ensemble
for terminal state estimate +0") directional vector v point- ing towards Ze, and sampled noise E, NO,
Entropy
(() 0.5 (I+ v1 area (ze e/ =
at
(8)
Energy
at- (1-0 e0 1).
(9)
0.0
0.2
0.4
0.6
0.8
Average Precision
SDE formulations of diffusion (Song et al., 2020b) general- ize DDPM and DDIM. Denoising models can be learned in one framework (DDPM) and sampled in another (DDIM).
Figure 5. The Epistemic Head is better indicator of rare or hard ImageNet-LT validation examples than entropy, energy, or epis- temic signals from an independently-trinn trained ensemble.
As stated, diffusion models sampled according to Eqns 6, are unconditional; they will sample instances that are dis- tributed approximately according to the data. This can be changed with guidance. In classifier-free guidance (Ho & Salimans, 2022), the diffusion model is trained on pairs (x, c) for data æ and conditioning vector e (for example, class labels or CLIP-encoded text). In contrast, classifier guidance (Dhariwal & Nichol, 2021) can be used during the sampling process even when no conditioning information was available at diffusion training time. It operates by bias- ing the denoising estimate in the direction of the gradient of differentiable signal, V (
f2(x) = -
Σ p(y oe)) K
uu(plo dx)). (4) K
1
Base model performance is protected by stop-grad at train- ing time that, similar to LoRA (Hu et al., 2021), the Epistemic Head does not impact existing model weights. In Figure 5, we demonstrate that longtail signals from the Epistemic Head more effectively indicate rare or hard ex- amples than do entropy or energy. In particular, we train ViT-B ImageNet-LT classifiers according to (Xu et al., 2023) and compare how well our proposed longtail signals (en- tropy, energy, epistemic) detect rare hard test examples. Examples are defined as rare if they come from longtail class. Examples are defined as hard if the model incorrectly predicts the label. In Supplement A.6, we further show that detected examples are disproportionately hard.
é = co t) w wv.. (Z,,
(10)
Commonly, [x,, = logp(ui xr, (), the log probability of class under trained classifier far. But this only works if classifier for is trained on intermediate noisy diffusion states 11:7 and accepts denoising step (thus, it trains on triples of (noisy data Se, step I, and label y) instead of standard formulation of (clean data and label y)). Training for on intermediate diffusion states is required because the distribution of x, differ from the original data x. Classifier- free guidance can be combined with classifier guidance.
3. Longtail Guidance
To motivate Longtail Guidance, briefly review diffu- sion. Diffusion models learn continuous data distribution pe(xo) J pe (eo:r)dz17 from finite set of data samples by defining a forward noising process q(I1T over la- tent states æ1.2, and learning a reverse denoising process po(oor) In the DDPM (Ho et al., 2020) and DDIM (Song al., 2020a) formulations, the forward process is a Markov chain that iteratively adds noise according 01.7 that decreases over steps I.....T.
Using classifier guidance to generate synthetic data in the longtail of existing production model Ja. as defined by model-based longtail signal (* is appealing because it can be applied to existing diffusion models without re- training. However, as stated, classifier guidance presents a dilemma: we must either fine-tune a noise-aware model for (I2, t) from the original model fo (z) on noisy, interme- diate diffusion states, which point før no longer reflects the production model performance of fa. or we must deploy production model for that wastes capacity intermediate diffusion states that it will not encounter in production.
g( ()))) N(Vor, (1 a))
(5)
However, they differ in the learned reverse process. DDPM models Markov chain,
There is an additional challenge with using classifier guid- ance for longtail data generation: SOTA diffusion models perform training and sampling in lower dimensional la- tent space 2 by first encoding the original data using a
p=(2)(x1-1) = N(uo( t), t)),
(6)
where, by reparameterization for numerical stability, the diffusion network co (Fe, t) learns predict the previously-

---

Generative Data Mining with Longtail-Guider Diffusion
Low
Medium
High
Figure 6. Synthetic data generated for 100 ImageNet classes with increasing Longtail Guidance strength, guided by a SOTA ViT ImageNet- LT classifier for Views frequently become more extreme, occluded, cut off. They also become more difficult for fo (see Figure 2). Best experiment results use low-to-mid Longtail Guidance weights (see Section 4 for details).
pretrained VAE, noising (at training) and denoising (at infer- ence) among latent states Zo:r, then decoding the final result back to data space X. How can we pass latent 2 through production model So when it operates in data space X²?
higher production model longtail signal (See Figure 3 and Algorithm for complete details):
é co(2), t) - =. (D(P(2.))) (11)
It is unintuitive that LTG would work since production model fo has only ever trained on clean (data, label) pairs, not intermediate diffusion states. Empirically, however, we find that tion model fo exhibits lower probability of the correct class, lower accuracy, and higher longtail signal. And so long as we do not adjust the longtail guidance weight w too high, the diffusion model reliably generates data that adheres to the expected class label.
Algorithm Longtail Guidance
Input:
Latent diffusion model ca(z,, l). predictor So (1r) la- tent decoder D, noise schedule oI.7, weight w
generate synthetic data for which produc-
Initialize: 2T (0, I)
for f = T 1, 1,.., 0 do
Estimate terminal latent state Ef = P(2)) as in Eqn. 8
Decode terminal data state: si = ()())
Compute model longtail signal (a²)) as in Eqn. 4
Bias denoising estimate as in Eqn. 11
Figures 1, 6 and Supplement A.9 show example longtail syn- thetic ImageNet data using SOTA VIT classifier (Xu et al., 2023) for guidance. Figure 2 quantitatively analyzes model performance and longtail signals on LTG-guided synthetic data generation over multiple runs of 24k generations span- ning all ImageNet classes. Notably, the probability of the expected class can be decreased to one-third of its original value (from the perspective of production model fg), while staying in-distribution (as evidenced by generalization im- provements in Section 4, see also Section 3.1) and the model longtail signal can be more than doubled over baseline (un- guided) synthetic data generation. Qualitatively, LTG data exhibit semantically meaningful changes, including more extreme, occluded, or cut-off views.
Compute 2 as in Eq.
end
return x= = D(20)
With Longtail Guidance, we find simple diffusion guid- ance approach that couples longtail signals from an existing production model to the Internet-scale knowledge of latent diffusion model. Surprisingly, it requires no retraining of either the diffusion model or the production model and, in particular, it does not require the production model to be trained on intermediate diffusion states or time-conditioned inputs, as common in Classifier Guidance.
The key idea in Longtail Guidance (LTG) is that we can dif- ferentiably estimate a terminal latent state 29 P(z)) with appropriate diffusion samplers (including DDIM), decode to an estimated terminal data state & = D( compute longtail signal (=+)) from the existing production model (that has only ever seen real production data), and then bias the denoising estimate (in latent space) in the direction of
Longtail Guidance is similar to Universal Guidance (Bansal et al., 2023) in that both perform differentiable decoding of latent states (see also (Jiang et al., 2023; Song et al., 2023; Dou & Song, 2024)). However, Universal Guidance is sig- nificantly more expensive and fails to remain in-distribution when guided by model- longtail signals. In particular, Universal Guidance performs so so-called recurrent and back-
5

---

Generative Data Mining with Longtail-Guidet Diffusion
tional data augmentations approaches. In Section 4.2, we demonstrate that LTG improves generalization performance of SOTA VIT models that compensate for class-imbalance, particularly for non-synthetic longtail data. In Section 4.3, we reduce syntheric data generated with Longtail Guidance to a set of text descriptions that describe attributes of a predictive model's longtail. We demonstrate that these de- scriptions are meaningful by showing that they produce higher-value synthetic data than manually prompt-tuned dif- fusion (in terms of predictive model generalization perfor- mance). In Section 4.4, we examine why Longtail Guidance outperforms existing synthetic data generation approaches.
Figure 7. Universal Guidance (top) vs Longtail Guidance (bottom). Universal Guidance, when driven by model-based longtail signals, successfully raises those signals but it does so by generating data that no longer in-distribution. It also much more expensive.
ward sampling passes, where recurrent sampling permits each diffusion step to refine its denoising estimate (each time computing costly gradient) and each backward pass optimizes an objective (within each denoising and recurrent iterate, itself requiring multiple optimization iterates). See Figure 7 for direct comparison.
4.1. LTG Improves Predictive Model Generalization
We compare Longtail Guidance with three measures (Epis- temic, Entropy, Energy) to the recent Guided Imagination Framework (GIF) (Zhang et al., 2023b) and Dream the Impossible (Dream-ID) (Du et al., 2024) synthetic data gen- eration approaches. For baselines, we compare to prompt- tuned Stable Diffusion 1.4 (SD), prompt-tuned DALL-E2, and masked autoencoder (MAE) data generation (He et al., 2022). We also compare to data augmentation baselines: Cutout VDevs,es, 2017), GridMask (Chen et al., 2020), Ran- dAugment (Cubuk et al., 2020), AutoAugment (Cubuk et al., 2019), CutMix (Yun et al., 2019). AugMix (Hendrycks et al., 2019) and adversarial robustness approaches Duepeugment (Hendrycks et al., 2021), and MEMO (Zhang et al., 2022a). Finally, we report CLIP zero-shot and distillation perfor- mance (Radford et al., 2021).
Although Longtail Guidance is not the first approach to use the predicted, decoded terminal state D(P(x))), we uniquely (to our knowledge) show in Supplement A.7 why the predictive model performing guidance does not need to be trained on intermediate diffusion states: (()))) is closer in distribution to real training data than is xt.
LTG requires no training for either the diffusion or the pro- duction model. Each can be used off- she-l. It's primary limitation is that the gradient d / /d calculated by decod- ing is expensive (see Supplement A.5).
We evaluate on seven natural image datasets spanning fine- grained, coarse- grained, and mixed coarse/fine classification tasks. Datasets include ImageNet, ImageNet-V2. ImageNet- A, Stanford Cars (Krause et al., 2013), Oxford Flowers (Nilsback & Zisserman, 2008), Oxford Pets (Parkhi et al., 2012), and Caltech101 (Fei-Fei et al., 2004). We reproduce baseline predictive models (Original) that are not exposed to synthetic data, with comparable performance (within 1%) of the baselines used in (Du et al., 2024; Zhang et al., 2023b). We iteratively perform additional fine-tuning with synthetic training data that is equal in quantity across all conditions, use the same augmentations (random rotation, reflection, and center cropping), and report generalization performance as top-1 I accuracy. Results are averaged over three runs. Additional experiment details are in Supplement A. and dataset details are in Supplement A.3. We fix hyperparam- eters (LTG weight, number of Epistemic Heads) across all datasets according to Supplement A.8.
3.1. Remaining In-Distribution
We ensure that LTG-generated data remain in-distribution but longtail by choosing guidance weight such that (1) the probability of the desired class is lower than baseline unguided synthetic data generation but nonzero, and (2) that the model- longtail signal fü evaluates higher than it does for baseline unguided synthetic data. Figure 2 shows that both are achievable; in fact, we find that a single weight can be chosen one time and used for all future generations (across classes, datasets and fine-tuning epochs). With this approach, we find that FID (Heusel et al., 2017) scores are lower and generative precision+recall (Kynkäänniemi et al., 2019) are higher when comparing L.TG-generated synthetic data to real data (within-classes) than when comparing real data to itself (between classes).
4. Experiments
In Table 1, we compare to baselines under the Dataset Ex- pansion task defined in GIF (Zhang et al., 2023b). For parity with the architecture and quantity of synthetic data
In Section 1. we compare predictive model generalization improvements when training data is augmented with syn- thetic data generated by Longtail Guidance, when training data is augmented with existing synthetic data generation approaches, and when training data is augmented with tradi-
Baseline model checkpoints were not available repro- duce them using the reported architecture and training recipe.
6

---

Generative Data Mining with Longtail-Guided Diffusion
used by GIF, we fine-tune ResNet-50 models for 100 epochs and generate synthetic data equivalent x0x the original dataset size for Caltech, Cars and Flowers, and 30x the original dataset size for Pets. GIF generates all synthetic data at once whereas we evenly distribute data generation throughout training epochs to better capture evolving model- based longtail challenges, see Supplement A. for details. In each expansion, synthetic examples per class are equal in quantity to the original number of real examples.
Table I. Top-1 Accuracy on coarse- coarse-grained (Caltech) and fine- grained (Pets, Cars, Flowers) natural image datasets under the
Dataset Expansion task as defined in (Zhang et al., 2023b).
Dataset
Caltech
Cars
Flowers
Pets
Avg
Original
26.3
19.8
74.1
6.8
31.8
CLIP
82.1
55,8
71.2
85.4
72.3
CLIP Distill
33.2
18.9
75.1
11.1
34.6
Expanded
Cutout
51.5
25.8
77.8
38.7
48.5
GridMask
51.6
28.4
80.7
37.6
49.6
In Table 2, we compare to baselines under the in-distribution synthetic data generation task as defined in Dream the Im- possible (Dream-ID) (Du et al., 2024) and discussed in Sup- plement 1. For parity with the architecture and quantity of synthetic data used by Dream-ID, we train ResNet-34 mod- els for 100 epochs and generate Ik synthetic data per class. Models are trained on ImageNet- 100 and evaluated on the same ImageNet- 100, ImageNet-A, and maageNet-V2 sub- sets as defined in Dream-ID and restated in Supplement A. In each benchmark, Longtail Guidance provides substan- tial improvements over existing synthetic data generation methods and data augmentation baselines. In particular, it provides an average 5.6 points of additional top-1 accuracy compared to GIF or Dream-ID, and average 15.0 points of additional top-1 accuracy compared to text-pon ext-prompted Sta- ble Diffusion. Aggregate improvements from the Epistemic Head over alternative model-based longtail signals (energy, entropy) are mild but significant and come at negligible in- ference or training time cost as described in Supplement A.5. In Supplement A.4, we show that predictive model general- ization can be further improved by engaging in more cycles of fine-tuning and synthetic data generation.
RandAugment
57.8
43.2
83.8
48.0
58.2
MAE
50.6
25.9
76.3
39.9
48.2
DALL-E2
61.3
48.3
84.1
61.7
63.8
SD
51.1
51.7
78.8
57.9
59.9
GIF-MAE
58.4
44.5
84.4
52.4
59.9
GIF-DALLE
63.0
53.1
88.2
66.4
67.7
GIF-SD
65.1
75.7
88.3
73.4
75.6
LTG (Energy)
70.4
85.3
90.3
80.0
81.5
LTG (Entropy)
70.6
84.6
89.9
81.6
81.7
LTG (Epistemic)
71.5
85.1
90.9
82.0
82.4
Table 2. Top-1 Accuracy the in-distribution synthetic data gen-
cration task for ImageNet variants as defined (Du et al., 2024)
Methods
ImageNet ImageNet-A Image
Avg
Original
87.28
8.69
77.80
57.92
RandAugment
88. 10
11.39
78.90
59.46
CutMix
87.98
9.67
79.70
59.12
AutoAugment
88.00
10.85
79.70
59.52
AugMix
87.74
10.96
79.20
59.30
Augmentment
86.86
10.79
78.30
58.65
MEMO
88.00
10.85
78.60
59.15
SD DREAM-ID
87.74
11.18
79.20
59.37
88.46
12.13
80.40
60.33
LTG (Energy)
90.00
20.20
81.05
63.75
Strikingly, we find that predictive model generalization when fine-tuned on I.TG-generated synthetic data eclipses generalization when fine-tuned on GlF-generated data at substantially lower synthetic data volumes: 20% for Cal- tech, 25% for Cars, 30% for Flowers, and 23% for Pets, for an overall average of 24.5%. Thus, L.GG-genrraed data not only provide higher overall generalization improvements, but they do so at more than 4x the synthetic data efficiency. Furthermore, LTG-generated data provide value in both high synthetic data volume regimes (20 30x original dataset size for GIF comparison) and low synthetic data volume regimes (less than 1x original dataset size for Dream-ID and ImageNet-LT comparisons).
LTG (Entropy)
90.04
22.70
80,66
64.47
LTG (Epistemic)
90.30
22.09
81.54
64.64
many data augmentations including Rand/ugment, Mixup, and CutMix. Our baseline predictive model, LiVT, nearly matches the generalization performance (top-1 accuracy) of what reported in (Xu al., 2023) (LiVT Reproduced: 60.6 vs LiVT Reported: 60.9). We then fine-tune LiVT on 24k additional synthetic data evenly distributed across all 1K classes for 100 epochs with le learning rate. We use the same Stable Diffusion 1.4 baseline and sampling details as in Section 4.1 (LiVT SD), and also compare Longtail Guidance with three types of model-based longtail signals (LiVT LTG (Energy, Entropy, Epistemic)), Overall gener- alization improvements on the balanced validation set are mild but significant (+1.3 top-1 accuracy). However, per- formance for longtail classes (Few) improves by four points for baseline diffusion (+10%) but marked 10 points for Longtail Guidance (+24%)
4.2. LTG Improves Longtail Performance
In Table 3, we train SOTA VaT-based models (LiVT) from scratch on ImageNet-LT according to the longtail- compensation approach of (Xu et al., 2023). In summary, training includes MAE pretraining followed by 100 epochs of BCE loss with a logit adjustment to account for class im- balance. Initial LiVT training is on real data only and uses
Synthetic data always improves Medium and Few perfor- mance, which are most impacted by fixed number of
7

---

Generative Data Mining with Longtail-Guided Diffusion
data generations per class. This experiment demonstrates that Longtail Guidance works with multiple architectures (ResNet, ViT), multiple losses (CE, logit-adjusted BCE), and that it disproportionatel improves longtail performance. also demonstrates that LTG can be composed with finely- crafted training recipes to further improve predictive model generalization performance.
Underside
Young
Damaged
Dew
Nighttime
Figure 8. Longtail keywords from LTG LTG-generated Flowers data.
thetic data Flg generated by LTG as guided by f For each synthetic instance, N novel keywords are found by computing the token embeddings that are furthest in cosine similarity from all real data token embeddings. An LLM (OpenAI, 2023) is then provided with a set of (caption, novel keyword) pairs for each class and asked to summarize them into P refined prompts. We call this Longtail Introspection.
Table 3. Top-1 I Accuracy on ImageNet-LT by dataset split (Many, Med, Few) and overall (Acc). The validation set balanced between all 1000 classes but training is highly imbalanced: Few classes have 5 19 examples, Medium classes have 20 99 examples, Many classes have 100 1200 examples.
Method
Many
Med.
Few
Acc
CE (Cui al., 2019)
64.0
33.8
5.8
41.6
Refined prompts are used to generate additional synthetic data Tinspect by diffusion (without LTG). We train So on synthetic data generated by Longtail Introspection and com- pare it to synthetic data generated by manual prompt tuning (40 unique prompts per class, as in (Zhang al., 2023b); details in Supplement A.2). Table 4 shows that Longtail Introspection significantly outperforms manually prompt- tuned diffusion (defined in Supplement A. quantitatively supporting that data generated by LTG exhibit meaningful variation. Figure 8 shows example longtail keywords.
I.DAM (Cao et al., 2019)
60.4
46.9
30.7
49.8
c-RT (Kang et al., 2019)
61.8
46.2
27.3
49.6
7-Norm (Kang et al., 2019)
59.1
46.9
30.7
49.4
Causal (Tang al., 2020)
62.7
48.8
31.6
51.8
Logit Adj. (Wang et al., 2020)
61.1
47.5
27.6
50.1
RIDE(4E) (Wang et al., 2020)
68.3
53.5
35.9
56.8
MiSLAS (Zhong ct al., 2021)
62.9
50.7
34.3
52.7
DisAAlgnn (Zhang et al., 2021)
61.3
52.2
31.4
52.9
ACE (Cai al., 2021)
71.7
54.6
23.5
56.6
PaCo (Cui et al., 2021)
68.0
56.4
37.2
58.2
TADE (Zhang et al., 2022b)
66.5
57.0
43.5
58.8
TSC (Li et al., 2022e)
63.5
49.7
30.4
52.4
GCL (Li et al., 2022d)
63.0
52.7
37.1
54.5
TLC (1.i al., 2022a)
68.9
55.7
40.8
55.1
Table 4. Synthetic data generated by prompts based on VLM de- scriptions of LTG data outperform synthetic data generated by manually prompt-tuned diffusion, quantitatively suggesting that data generated by LTG exhibit meaningful and challenging varia- tion from the perspective of predictive model fe-
BCL. (Zhu et al., 2022)
67.6
54.6
36.6
57.2
NCL (Li ct al., 2022c)
67.3
55.4
39.0
57.7
SAFA (Hong et al., 2022)
63.8
49,9
33.4
53.1
DOC (Wang al., 2022)
65.1
52.8
34.2
55.0
DLSA (Xu et al., 2022)
67.8
54.5
38.8
57.5
Condition
Accuracy
VIT (Dosovitskiy, 2020)
50.5
23.5
6.9
31.6
MAE (He al., 2022)
74.7
48.2
19.4
54.5
No Synthetic Data
74.1
DeiT (Touvron et al., 2022)
70.4
40,9
12.8
48.4
Manual Prompt Tuning
78.8
LiVT (Xu et al., 2023)
72.7
56.6
40.4
60.6
Longtail Introspection
84.3
LiVT SD
71.8
58.1
44.3
61.5
LiVT LTG (Energy)
70.5
57.9
50.0
61.7
LiVT LTG (Entropy)
70.9
57.9
49.8
61.8
4.4. Discussion
LiVT (Fpistemic)
71.4
57.7
50.0
61.9
Longtail Guidance significantly outperforms leading syn- thetic data generation baselines including GIF and Dream- ID even though it requires no prompt tuning. To explain this, we note that GIF and Dream-ID reason in latent CLIP space to create new embedding vectors with which to prompt a diffusion model. For each real data instance, GIF creates K additional synthetic instances by jointly optimizing K em- bedding vectors (initialized by CLIP image encoding) such that they are difficult for CLIP zero-shot classification (high entropy), remain close to the target class (high log prob of target class), and diverse (high KL divergence embedding). In Dream-ID, separate model is trained to predict class embedding vectors from (real data, CLiP-based class text embedding vector) pairs. From this, a manifold in CLIP space can be sampled along class boundaries to generate new diffusion conditioning vectors.
4.3. LTG Produces Meaningful Longtail Data
We examine synthetic data generated by LTG to determine whether they exhibit meaningful variation from baseline synthetic data generated by prompt-tuned diffusion. This is a difficult and fundamentally qualitative task to manually perform at scale. To make it quantitative, we ask: do text descriptions of LTG-generated data lead to predictive model generalization improvements when they are used to generate additional synthetic data (without L.TG)?
mean
Starting with the strongest baseline model fo trained on Flowers (Original from Table 1), we ask a VLM (Liu et al., 2024) to caption real training instances Trest and also syn-
8

---

Generative Data Mining with Longtail-Guided Diffusion
Caltechao1. p-27%
Cars, 2=21%
Flowers, 3=33%
2021; Ho & Salimans, 2022). Universal Guidance (Bansal et al., 2023) and Diffusion Posterior Sampling (Dou & Song, 2024) are most relevant and are discussed in Section 3.
Per Claso Accurssy Diffurncce
Synthetic training data from generative models has been considered since GANs (Li et al., 2022b), but has started to come of age with diffusion (Azizi et al., 2023; Zhou et al., 2023), particularly for high-resolution datasets where fidelity matters. GIF (Zhang et al., 2023b) and Dream-ID (Du et al., 2024), discussed in Section 4.4, are most relevant.
0.0
Class Index
102
146
162
Figure 9. Reasoning latent CLIP space for synthetic data gener- ation does not account for differences in what is difficult for CLIP versus what is difficult for a production model. We plot (sorted) absolute difference in per-class accuracy between CLIP zero- shot classifier and a well-trained production model. Even when aggregate performance is similar, as in Flowers (74.0% vs 71.2% top-1 accuracy), average per-class difference is high (33%).
Model signals have been used for out-of- distriiritntoinn or ad- versarial detection (Huang et al., 2021; Hsu et al., 2020; Co- hen et al., 2020), particularly model uncertainty (Van Amers- foort et al., 2020) or feature density (Lee et al., 2018). Epis- temic uncertainty has been developed in expensive Bayesian (Gal & Ghahramani, 2016) or model ensemble contexts (Liu et al., 2019a; Depeweg al., 2018; Wilson & Izmailov, 2020; Lakshminarayanan et al., 2017), though, to our knowl- edge, not in a single, differentiable forward pass as we have done. Work on longtail robustness has primarily focused on addressing a-priori known class imbalance. Datasets include ImageNet-LT, Places-LT, (Liu et al., 2019b) and iNaturalist (Van Horn al., 2018). Mitigations include pretraining (He et al., 2022; Bao et al., 2021), distillation (Xiang et al., 2020), reweighted loss (Xu et al., 2023; Ross & Dollár, 2017), selective real data mining (Jiang et al., 2022), or low-density data sampling (Um & Ye, 2024).
We posit that synthesizing data that is difficult or rare for a foundation model, like CLIP, is different from synthesizing data that is difficult or rare for a specific, deployed predictive model. Figure 9 quantifies this by showing that, even when CLIP zero-shot performance and production model have similar aggregate performance 74.1% vs 71.2%, the classes (and data instances) that are difficult for CLIP are different from the classes that are difficult for the production model- with average per-class accuracy differences of 34%!
In both GIF and Dream-ID, reasoning is only conditioned on training data and restricted CLIP embedding space. In Longtail Guidance, instead condition on longtail sig- nals coming directly from the predictive model we wish to improve. We know that these signals identify rare or dis- proportionately hard examples (Figure 5) and that synthetic data generated by LTG reliably have lower probability of correct classification, lower accuracy, and higher longtail signals (Figure 2) under the predictive model. This is most strikingly demonstrated in the more than 80% generalization improvement over Dream-ID on the naturally challenging examples of ImageNet-A.
Conclusion
We develop model-based longtail signals that do not impact model weights or performance and are leveraged by Longtail Guidance, synthetic data generation approach that explic- itly conditions diffusion on an existing predictive model to generate examples that are rare or hard from that model's perspective. We show that training on LTG-generated data provides strong, data-efficient generalization improvements across eight datasets. We further demonstrate that longtail synthetic data generations can be rendered into meaningful text descriptions and keywords that can aid future (real or synthetic) data collection priorities.
5. Related Work
Diffusion has seen wide success, particularly in image gen- eration, where it outperform GANs in image quality and diversity without suffering from unstable training or mode collapse (Dhariwal & Nichol, 2021). Recent work has seen progress in handling large data dimensions with la- tent spaces (Chen et al., 2025; Podell et al., 2023; Rombach et al., 2022) or hourglass networks (Crowson et al., 2024), improved sampling (Lu et al., 2022; Karras et al., 2022: Ho et al., 2020; Song al., 2020a), additional data do- mains (Ran et al., 2024; Pronovost et al., 2023; Zhong et al., 2023), and personalization (Ruiz et al., 2023; Kumari et al., 2023). Much work has also been done on new forms of guidance (Wallace et al., 2023b; Yu et al., 2023a; Wallace et al., 2023a; Zhang et al., 2023a) beyond just classifier guidance and classifier-free guidance (Dhariwal & Nichol,
Predictive models are being deployed more than ever before. Increasingly, they will encounter longtail scenarios that hu- man operators cannot easily predict in advance. Foundation models can be used to mitigate some risk, but we cannot shochorn the entirety of Internet-scale knowledge into every deployed model due to capacity and compute constraints. By letting an existing predictive model speak for itself. we can expend offline compute to generatively mine high-value synthetic training data. Further reducing these synthetic examples to human- and machine-readable text suggests a future where we can move away from slow, reactive longtail mitigation towards fast, proactive longtail discovery.

---

Generative Data Mining with Longtail-Guided Diffusion
Impact Statement
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L.,, Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pp. 74-91. Springer, 2025.
We develop model-based longtail signals, model- conditioned longtail synthetic data generation, and longtail interpretation techniques that have the potential discover and mitigate production model failure modes before they occur. This has tremendous potential for improving the quality and safety of deployed ML applications. However, they may also have negative impacts if human operators rely too heavily on our approaches and not enough on traditional safeguards. We recommend our methods be used to supplement, not replace, standard deployment practices: most importantly, that regular and satisfactory evaluation on non-synthetic evaluation data be used as requirement for model deployment.
Chen, P., Liu, S., Zhao, H., Wang, X., and Jia, J. Gridmask data augmentation. arXiv preprint 2001:22011.04066, 2020.
Cohen, G., Sapiro, G., and Giryes, R. Detecting adversarial samples using influence functions and nearest neighbors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14453-14462 2020.
Crowson, K., Baumann, S. A., Birch, A., Abraham, T. M..
Kaplan, D. Z., and Shippole, E. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Forty-first International Conference on Machine Learning, 2024.
References
Anthropic. Introducing the next generation of claude, 2024. URL https:/ http:/w..tthh.... anthropic com/
news/claude-33 Family. Accessed: 2025-01-03.
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 113-123, 2019.
Azizi, S., Komblith, S., Saharia, C., Norouzi, M., and Fleet, D. J. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08 2023.
Bansal, A., Chu, H.-M.,, Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein, T. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 843 852, 2023.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Ran- daugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
Bansal, H. and Grover, A. Leaving reality to imagina- tion: Robust classification via generated datasets. arXiv preprint ar Xivv:2302.025003 2023.
Cui, J., Zhong, Z., Liu, S., Yu, B., and Jia, J. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715- 724, 2021.
Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
Cui, Y., Jia, M., Lin, T.-Y.. Song, Y., and Belongie, S. Class- balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
2019.
Depeweg, S., Hernandez-Lobato, J.-M., Doshi-Veles F., and Udluft, S. Decomposition of uncertainty in bayesian deep learning for efficient and risk- learning. In International conference on machine learning, pp. 1184 1193. PMLR, 2018.
Cai, J., Wang, Y., and Hwang, J.-N. Ace: Ally comple- mentary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 112-121,
Devlin, J. Bert: Pre- traininin of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04800 2018
Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. Learning imbalanced datasets with label-distribution- aware margin loss. Advances in neural information pro- cessing systems, 32, 2019.
Vriess T. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1eXXv:1708.0455 2017.
10

---

Generative Data Mining with Longtail-Guided Diffusion
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Dosovitskiy, An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12544 2022.
Dou, Z. and Song, Y. Diffusion posterior sampling for linear inverse problem solving: filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6855 2020.
Hong, Y., Zhang, J.. Sun, Z., and Yan, K. Safa: Sample- adaptive feature augmentation for long-tailed image clas- sification. In European Conference on Computer Vision, pp. 587-603. Springer, 2022.
Du, X., Sun, Y.. Zhu, J., and Li, Y. Dream the impossible: Outlier imagination with diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Fei-Fei, L.. Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop. pp. 178-178, IEEE, 2004.
Hsu, Y.-C., Shen, Y., Jin, H., and Kira, Z. Generalized odin: Detecting dis-iiiiiiittiion image without learn- ing from out-of-distribution data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10951-10960, 2020.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L.,, and Chen, W. Lora: Low-rand adaptation of large language models. arXiv preprini arXiv:2 1eee2106.00666 09685, 2021.
Gou, J.. Yu. B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-183) 2021.
Huang, R., Geng, A., and Li, Y. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34: 677-689, 2021.
Gowal, S., Rebuffi, S.-A., Wiles, O., Stimberg, F., Calian, D. A.. and Mann, T. A. Improving robustness using gen- erated data. Advances in Neural Information Processing Systems, 34:4218- 4233,4423222222 2021.
Jang, E., Gu, S., and Poole, B. Categorical repa- rameterization with gumbel-softmax. arXiv preprint 1X10:1611.001444 2016.
Guzman-Rivera, A., Batra, D., and Kohli, P. Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems, 25, 2012.
Jiang, C.. Cormman, A.. Park, C., Sapp, B., Zhou, Y., Anguelov, D., et al. Motiondiffuser: Controllable multi- agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9644-9653 2023.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Jiang. C. Majibi, M., Qi, C. R., Zhou, Y., and Anguelov, D. Improving the intra-class long-tail tail in 3d detection via rare example mining. In European Conference on Computer Vision, pp. 158-175. Springer, 2022.
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and I Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 1912.0222..781.. 2019.
Kang, B., Xie, S., Rohrbach, M., Yan, Z.,, Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 1oXiv:99100092111 2019.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of datoo--distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001 083611, 2020.

---

Generative Data Mining with Longtail-Guided Diffusion
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 2022.
ocr, and world knowledge, January 2024. URL https://llavaa wwiiaaaa..iiiiiabb.aa... github. io/blog/ 2024 00-30 101-30-1lavaa
Liu, J., Paisley, J., Kioumourtzoglou, M.-A., and Coull, B. Accurate uncertainty estimation and decomposition in ensemble learning. Advances in neural information processing systems, 32, 2019a.
Krause, J., Deng, J., Stark, M., and Fei-Fei, L. Collecting large-scale dataset of fine-grained cars. 2013.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-coneept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931- 1941, 2023.
Liu, W., Wang, X., Owens, J.. and Li, Y. Energy-based out- of-distribution detection. Advances in neural information processing systems, 33:21 241221464-1145, 2020.
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2537-2546, 2019b.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. Advances in neural information processing systems, 32, 2019.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095. 2022.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information pro- cessing systems, 31, 2018.
Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for bayesian uncer- tainty in deep learning. Advances in neural information processing systems, 32, 2019.
Li, B., Han, Z., Li, H., Fu, H., and Zhang, C. Trust- worthy long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6970-6979, 2022a.
Minsky, M. The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. Simon and Schuster, 2006. ISBN 9780743276641.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Li, D., Ling, H., Kim, S. W., Kreis, K., Fidler, S., and Torralba, A. Bigdatasetgan: Synthesizing imagenet with pixel-wise wise annotations. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21330-21340, 2022b.
OpenAl. Gpt-4 technical report. Technical report, OpenAl, 2023. URL https://openas. wtt.c/oo//eeeeeee resm/ressacc/// g.
Li, J., Tan, Z., Wan, J.. Lei, Z., and Guo, G. Nested col- laborative learning for long-taile visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6949-6955 6958, 2022c.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Li, M., Cheung, Y.mm,, and Lu, Y. Long-tailed visual recog- nition via gaussian clouded logit adjustment. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6929-6938, 2022d.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna. J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolutie image synthesis. arXiv preprint Si0i:2200.009522 2023.
Li, T., Cao, P., Yuan, Y., Fan, L., Yang, Y., Feris, R. S., Indyk, P., and Katabi, Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp.
Pronovost, E., Ganesina, M. R., Hendy, N., Wang, Z., Morales, A., Wang, K., and Roy. N. Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, 36: 68873-68894 2023.
2022c.
Liu, H. Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning,
12

---

Generative Data Mining with Longtail-Guided Diffusion
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. -768-.763. PMLR, 2021.
Um, S. and Ye, J. C. Self-guided generation of minority samples using diffusion models. In European Conference Computer Vision, pp. 414-430. Springer, 2024.
Van Amersfoort, J., Smith, L., Teh, Y. W., and Gal, Y. Un- certainty estimation using a single deep deterministic neural network. In International conference on machine learning, pp. 9690-9700, PMLR, 2020.
Ran, H., Guizilini, V., and Wang, Y. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17738-14748, 2024.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-reste figg-rreoluttin image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp- 10684-10695, 2022.
Wallace, B., Gokul, A., Ermon, S., and Naik, N. End- to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7280-7290, 2023a.
Ross, T.-Y. and Dollár, G. Focal loss for dense object detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2980-2988, 2017.
Wallace, B., Gokul, A., and Naik, N. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22532-22541, 2023b.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Pro- ceedings of the IEEE/CVF conference on computer vision 2023.
Wang, H., Fu, S., He, X., Fang, H., Liu, Z., and Hu, H. Towards calibrated hyper-sphere representation via dis- tribution overlap coefficient for long-tailed learning. In European Conference on Computer Vision, pp. 179-196. Springer, 2022.
and pattern recognition, pp.
Song, J. Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502. 2020a.
Song, J., Zhang, Q., Yin, H., Mardani, M., Liu, M.-Y., Kautz, J., Chen, Y., and Vahdat, A. Loss Loss-guided diffu- sion models for plug-and-play controllable generation. In International Conference on Machine Learning, pp. 32483-30 -32498. PMLR, 2023.
Wang. X., Lian, L.., Miao, Z., Liu, Z., and Yu, S. X. Long- tailed recognition by routing diverse distribution-aware experts. arXiv preprint 2uXiv:2010.01809 2020.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of of-in-of-thoghh prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224824-24831 2022.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic diffcrential equations. arXiv preprint arXiv:2011.13455 2020b.
Wilson, A. G. and Izmailov, P. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708. 2020.
Tang, K., Huang, J., and Zhang, H. Long-tailed classifi- cation by keeping the good and removing the bad mo- mentum causal effect. Advances in neural information processing systems, 33:1513-1524 2020.
Xiang, L., Ding, G., and Han, J. Learning from multiple
Touvron, H., Cord, M., and Jégou, H. Deit ii:: Revenge of the vit. In European conference computer vision, pp. 516 533. Springer, 2022.
experts: Self-paced knowledge distillation for long-tailed classification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 247-263. Springer, 2020.
Tran, L., Veeling, B. S., Roth, K., Swiatkowski, J., Dillon, J. V., Snoek, J., Mandt, S., Salimans, T., Nowozin, S., and Jenatton, R. Hydra: Preserving ensemble diversity for model distillation, arXiv preprint arXiv:2001.0469, 2020.
Xu, Y., Li, Y.L.,, Li, J., and Lu, C. Constructing balance from imbalance for long-tailed image recognition. In European Conference on Computer Vision, pp. 38-56. Springer, 2022.
13

---

Generative Data Mining with Longtail-Guided Diffusion
Xu, Z.,, Liu, R., Yang, S., Chai, Z., and Yuan, C. Learning imbalanced data with vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19793-15803, 2023.
Zhong, Z., Cui, J., Liu, S., and Jia, Improving calibra- tion for long-tailed recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16489-16498, 2021.
Yao, S., Yu, D., Zhao, J., Shafran, L. Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhong, Z., Rempe, D., Chen, Y., Ivanovic, B., Cao, Y., Xu,
D., Pavone, M., and Ray, B. Language-guided traffic simulation via scene-level diffiusion. In Conference Robot Learning, pp. 144 177. PMLR, 2023.
Yu, F., Chen, H., Wang. X., Xian, W., Chen, Y., Liu, F., Mad- havan, V., and Darrell, T. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636-2644 2020.
Zhou, Y., Sahak, H., and Ba, J. Training on thin air: Improve image classification with generated data, arXiv preprint 1rXi::2305.15116 2023.
Zhu, J., Wang. Z.., Chen, J., Chen, Y.-P. P., and Jiang, Y.- G. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6908- 6917, 2022.
Yu, J., Wang. Y., Zhao, C., Ghanem, B., and Zhang, J. Freedom: Training-free free grrrgy-guideddd conditional diffu- sion model. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 23174-23184, 2023a.
Yu, R., Liu, S., and Wang, X. Dataset distillation: A compre- hensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong clas- sifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6023 6032,, 2019.
Zhang, L., Rao. A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3836-3847, 2023a.
Zhang, M. Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642, 2022a.
Zhang, S., Li, Z.. Yan, S., He. X., and Sun, J. Distribution alignment: A unified framework for long-tail visual recog- nition. In Proceedings of the IEEE/CVF conference computer vision and pultern recognition, pp. 2361-2370, 2021.
Zhang, Y., Hooi, B., Hong, L., and Feng. J. Self-supervised aggregation of diverse experts for test-agnostic long- tailed recognition. Advances in Neural Information Pro- cessing Systems, 35:34077-34090 2022b.
Zhang, Y., Zhou, D., Hooi, B., Wang, K., and Feng, J. Expanding small-scale datasets with guided imagination. Advances in neural information processing systems, 36: 16558-76618 2023b.
14

---

Generative Data Mining with Longtail-Guided Diffusion
A. Appendix
A.I. Comparison to GIF and Dream-ID
In the main paper, Table I, we compare to GIF, the Guided Imagination Framework (Zhang et al., 2023b). We use the same predictive model architecture (ResNet50 trained from scratch), generate the same quantity of synthetic data (30x dataset size for Pets, 20x dataset size for Caltech101, Cars, and Flowers), the same diffusion model (Stable Diffusion v1.4), the same diffusion sampler (DDIM), and the same number of sampling iterates (50). We reproduce baseline models (not exposed to synthetic data or data augmentation conditions) to match the performance of GIF baselines (Original row in Table 1) to within 1% of the generalization performance reported in GIF. We then fine-tune for 100 epochs, generating synthetic data with Longtail Guidance according to the schedule in Table 5 (e.g. generate synthetic data in epoch 0, fine-tune until epoch 5, generate more synthetic data, fine-tune until epoch 10, ..... We experimentally found that synthetic data iteratively generated throughout fine-tuning outperformed synthetic data generated all at once and speculate that this is due to the model-basee based longtail evolving throughout training. We also experimenaally found that using total uncertainty (epistemic aleatoric, the first term in the RHS of Equation 4) as the LTG guidance signal from the Epistemic Head was overall slightly more performant for downstream predictive model generalization improvements (as compared to epistemic or aleatoric alone), likely because several of our benchmark datasets involve fine-grained distinctions that benefit from additional synthetic data, not just towards the edge of the class manifold (epistemic) but also along decision boundaries (aleatoric). See Figure 4 for visualization.
Synthetic data are distributed across classes according to their frequency in the original training data. Fine-tuning is with the Adam optimizer, cosine annealing learning rate schedule, and le 3 learning rate. As in GIF, we train with random rotations (+159), 224 x 224 crops, and horizontal flips.
In the main paper, Table 2, we compare to the in-distribution synthetic data generation task (Dream-II) from Dream the Impossible (Du et al., 2024). As in Dream-ID, we start with ImageNet-pretraing ResNet-34 models and fine-tune for 100 epochs with the same training details as above, except that we match the synthetic data quantity used in Dream-ID by generating a total of 1000 images per class, evenly distributed every epochs (50 images per class per epochs).
Different from GIF or Dream-ID is that we build synthetic data over the course of model fine-tuning, to better capture evolving model-based longtail challenges (whereas GIF, Dream-ID, and other synthetic data baselines generate all synthetic data at once since they not coneilcondiiioned). Also different is that LTG does not need prompt tuning to force synthetic data diversity. LTG text prompts are generie: "a photo of Class" whereas GIF prompts (and diffusion baselines) are, "a Noun Adjectiv ive of Class" where Noun is randomly sampled to be one of ("image", "oil painting". "cartoon image", "sketch", "pencil sketch") and Adject i ve is randomly sampled to be one of ("", "colorful", "stylized", "bright", "sheared", "solarized", "posterized", "high-contrast",, for a total of 40 unique prompts per class. In LTG, diversity is automatic based on the production model's evolving longtail signals.
All experiments are performed on 8xH100.
Dataset
Total Expansion Ratio
Synthetic Data Generation Epochs
Pets
x0x
0, 5, 45, 50; 52, 54,... ..., 96, 98
Caltech101, Cars, Flowers
20x
0,5.... .0.....,0... 95
Table Synthetic Data Generation Schedule by Dataset
A.2. Longtail Introspection
We construct refined prompts for each class by prompting VLL 7B (Liu et al., 2024) to generate a description for two sets of synthetic images: 100 generated by Longtail Guidance and 100 generated by baseline diffusion. VLM prompts are of the form: This is an image of <Class>. Describe it in detail. For each output VLM deseription, compute per-token BERT embeddings (Devlin, 2018). This provides us with two token embedding distributions: longtail and baseline.
For each token embedding in the longtail distribution, we compute cosine similarity to the nearest token embedding in the baseline distribution. For each longtail image, we define the K tokens that are furthest from the the baseline distribution as longtail keywords. We retain the (description, keyword) pairs of the P = 40 longtail examples per class whose keyword tokens are furthest from the base distribution.
15

---

Generative Data Mining with Longtail-Guided Diffusion
Following, we create P 40 refined prompts per class by prompting GPT-4o (OpenAl, 2023) with: <VLM description of the image> The following keywords describe the key features of the description above: <Keyword 1?2, <Keyword 2> ... Use complete sentence summarize the key features. The sentence should start with: A photo of <Class> that....
Example (image, keyword) pairs are displayed in Figure 8 of the main paper. Example refined prompts are displayed in Table 6. We emphasize that not all keywords found by this method are immediately meaningful (e.g. satin as in satin-like pedals, ether as in ethereal appearance, but they can be quickly scanned for longtail themes and, quantitatively, the process improves generalization performance beyond manual prompt tuning.
A photo of a bearded iris that showcases its deep purple petals with yellow and brown pattern running through the underside, set against blurred background emphasize the flower's texture and color.
A photo of sweet pea that showcases a young plant with single, unfurled green leaf and a small, green flower bud heginning to open, set against softly blurred green background, highlighting the delicate texture and promise of the flower to come.
A photo of red ginger that showcases vibrant red flower with a yellow center, surrounded by slightly damaged green leaves, set against a blurred natural background.
A photo of morning glory that showcases vibrant petals with gradient from deep red to surelp arranged in a spiral pattern, with smooth texture and dew droplets, set against a blurred background to emphasize the flower's 's intricate details and colors.
Table 6. Example refined prompts generated by Longtail Introspection for Flowers data. Keywords are bolded.
A.3. Dataset Details
We summarize each dataset used in Table 7. Additional details can be found in the (Zhang et al., 2023b) for Caltech, Cars, Pets, and Flowers or (Du et al., 2024) for ImageNet- ImageNet-A, and ImageNet-V2 variants. Notably, (Du et al., 2024) creates evaluation subsets of ImageNet-A and ImageNer-Ve V2 that overlap with the training classes of ImageNet-100, listed in Figure 10. For fair comparison, we also train on ImageNet- 100 and evaluate on the ImageNet-100 (Eval), ImageNet-A, and ImageNet-Ve V2 subsets as defined in (Du et al., 2024). We emphasize that these datasets exhibit example counts that are common for longtail data in production settings.
Dataset
Classes
#Train
#Val
#Synthetic
Caltech101
102
3060
6084
61200
Cars
196
8144
8041
162880
Flowers
102
6557
1632
131140
Pets
37
3680
3669
110400
ImageNet-100 100
100
129860
5000
100000
ImageNet-A
41
1852
ImageNet- V2*
100
10000
ImageNet-LT
1000
115846
20000
24000
Table Overview of datasets with number of classes, training samples, validation samples, and synthetic samples. *ImageNet-A and ImageNet-V2 V2 are not trained on; they are only used for evaluation.
A.4. Can We Continuously Mine for Additional High-Valus Value Synthetic Data?
Dataset
Caltech
Cars
Flowers
Pets
Avg
LTG (100 epoch)
71.5
85.1
90.9
82.0
82,4
LTG (200 epoch)
72.9
85.4
92.6
82.1
83.3
LTG (300 epoch)
72.9
85.4
92.6
82.2
83.3
Table 8. Top-1 Accuracy when training with Longtail Guidance for many epochs.
In Table 8 we ask: Longtail Guidance generates high-value training data for predictive model far can we continuously
16

---

Generative Data Mining with Longtail-Guided Diffusion
401493041
201514859
a01582220
a01608432
201616318 201687976 s01776313 n01806567 n01833805
n01382714
a01910747
a01944390
m01985126
m02007556
202071294 =02095620 =02:14855
n02123045
n02126365
no2129165
m02129604
m02165456
n02190166
m02219486
n02226429
=02279912
=02317335
n02326432
n02342885
nQ2363005
=02391049
202395406
a02403003
#02422699
#02442845 n02444819
#02480855
n02510455
n02640242
no2672831
=02687172
=02701002 =02130930
m02769748
n02782093 m027a7622 n02793495
n02759071
n02802426
nd2314860
nD2840245
102906734
m02948072 m02980441
r02999410 ro301470h
r03028079
na3032252
n03125129
n63160309
103179701
a03223513
n03291819
603388043 r03450230
n03481172
n03594734
a03627232
a03642806
203649909 a03661043
203676423 2оз724870 =03733282
n03759954
n03761084
n03773504
a03804744
=03916031 m03938244
m04804787 n04026417 =04090263 204123789
n04153751
n04296562 nC4330267
104371774
a04404412 m04465501 n04485082 n04507155 r04536866 r04579432 n04606251 na7714990 n69745940
Figure 10. List ImageNet-106 Classes trained and evaluated on as defined by (Du al., 2024)
iterate between fine-tuning and generating synthetic data for additional generalization improvements? In this experiment, we generate synthetic data according the original schedules defined in Table 5 for the first 100 epochs. We then continue fine-tuning and generating an additional 1x expansion every 5 epochs for total of 300 epochs and report generalization performance. We observe that in all cases, we can improve performance, but that gains eventually saturate. It remains a question for future work whether higher-capacity generative models could be mined longer periods of time before predictive performance saturates. If so, it suggests an exciting future possibility of continuously exchanging unused offline compute for improved predictive model performance.
A.5. Computational Costs
EPISTEMIC HEAD
The Epistemic Head with K heads has a parameter count equal to K x dmodd C, for model embedding dimension dmodel and number of output logits C. For many predictive models, this leads negligible parameter count increase, as displayed in Table less than 5% increase for all experiments in this paper. Training and inference times are impacted by less than 2% for most expensive (ImageNet-LT) experiments.
Classes
Epistemic Heads
3
10
0.03%
0.04%
100
0.27%
0.45%
1000
2.67%
4.44%
Table 9, Epistemic Head Parameter Count as a Percentage of VIT-B Parameter Count
LONGTAIL GUIDANCE
We generate 512 512 resolution synthetic images at unoptimized rate of 6.32 image second for baseline text-prompted diffusion and 1.01 image second for diffusion with Longtail Guidance (using Stable Diffusion v1.4 in FP16 with 50 DDIM sampling steps on 8xH100 GPUs). The majority of the LTG guidance cost is in diffcrentiably decoding through the VAE. In particular, in the gradient
dEp dz.
(12)
dip
calculated by the main paper's Algorithm 1, the second term is VRAM-intensinn (45GB for batch size 8 without gradient checkpointing). This cost occurs primarily because the 83.6m parameter VAE decodes latent dimensions of 64 64 x 4 to data dimension 512 x 512 x 3.
A.6. Longtail Signals
In Figure 11 we show that high quantiles == 90% of model-based longtail signals are all strong indicators of examples that are dispropoorionnatel hard for the model (as determined by average accuracy above and below the quantile). However, as shown in Figure 5 of the main paper, the Epistemic Head more effectively detects rare or hard examples.
In Figure 12, we visualize a toy example of high epistemic (left) and high aleatoric (right) uncertainty for three Epistemic Heads each classifying over three classes. High epistemic uncertainty occurs when each sample from the predictive posterior
17

---

Generative Data Mining with Longtail-Guidet Diffusion
(i.e. each Epistemic Head) gives mutually incompatible answers (such as all being confident about different class). High aleatoric uncertainty (which includes high entropy) occurs when each sample has similarly and highly ambiguous (e.g. uniform) belief over classes.
High
Low
Epistemic
High Epistemic
High Aleatoric
Head
Epistemic Endemble
0.8
Sample
Entropy
0.4
0.2
Energy
0.1
0.2
0.3
Accuracy
0.4
0.5
Class
Class
Figure 11. High quantiles of longtail signal indicate examples that disproportionately hard.
Figure 12. Probability of classification under high epistemic and high alcatoric uncertainty. There three samples from the pos- terior predictive (y-axis). Each maintains a distribution over three classes (x-axis). Epistemic uncertainty (left) goes high when each sample has mutually incompatible beliefs. Aleatoric uncertainty (right) goes high when samples are highly uncertain.
A.7. Why Does Longtail Guidance Work without Training the Predictive Model on Intermediate Diffusion States?
A key finding of this work is that existing predictive model fø does not need to be retrained intermediate, noisy diffusion states to effectively guide diffusion model towards high-value synthetic training examples that are rare or hard from f's perspective. This realization frees us from the dilemma of having to decide whether to waste predictive model capacity training on intermediate diffusion data it will never see in production or risking divergence from the production model by fine-tuning on intermediate noisy states. But why does this work?
In Figure 13, we visualize the intermediate, noisy diffusion states of two quantities: the decoded, predicted terminal state ((P())) (top row, what LTG uses as guidance input predictive model fa) and the decoded data state, D(2,) (bottom row, what classifier guidance would traditionally performs guidance if not in latent space). We observe that the terminal state predictions much more readily resemble natural image data at much earlier time in the diffusion process (within the first 10% of denoising steps) than do the data states. In fact, decoded data states have off-distribution noise artifacts up through the first 90% of the diffusion denoising steps. We speculate that one reason the original classifier guidance work (Dhariwal Nichol, 2021) performed guidance on intermediate noisy states is that they can be more efficiently generated than the predicted clean terminal states; intermediate states merely require sampling a random clean data point, sampling random timestep, and applying the noise schedule. Predicted clean terminal states from given denoising timestep additionally require forward pass from the denoising network (see Equation 8).
We make this quantitative with the Frechet Inception Distance (FID) (Heusel et al., 2017), defined as
FID
||p, - Mg 1 + Tr(2, + Σi 2(2,,21))
(13)
where (Hr; Er); (Mg. Eng) the and covariance of Inception- V3's pool3 features for real and synthetic data, respectively. Lower FID indicates that the generated data more closely match and cover the real data. We measure FID for two conditions:
FID between the decoded, predicted terminal state :! = (*.())) and real ImageNet-I.T training data, and
2. FID between the decoded data state 2, and real ImageNet-LT training data.
Results are plotted in Figure 15. Observe that the decoded, predicted terminal state ((P(2)) are dramatically closer in distribution to real training data than are the naively decoded data states, o, for nearly all denoising iterations.
Figure 14 demonstrates that, because the terminal state estimates D(P(a?)) are closer real training data, the predictive model able to effectively guide data generations towards higher longtail signals (model entropy in this case), with clear
18

---

Generative Data Mining with Longtail-Guided Diffusion
longtail signal separation between different longtail guidance weights occurring within the first 25% of denoising iterations.
In contrast, naively guiding based data states 22 causes the predictive model to be unable effectively guide data generation until the last 5% of denoising iterations, when there are no longer many degrees of freedom for the diffusion model to meaningfully change image content.
Timestep
Figure Performing Longtail Guidance on predicted, decoded terminal states 2, (top row) provides production model fo data that are more in-distribution and less corrupted by intermediate diffusion noise than does Naive guidance performed each intermediate decoded state 2, (bottom row). Figures 14, for quantitative analysis.
G Longtail
140
Decoded State Xt (Naive)
Frechet Inception Distance
120
x (LTG)
100
Guidance Weight
15.0
5.0
0.0
80
G with Naive Guidance
60
40
3-
20
Naive Guidance Whight
1-
15.0
1000
800
súo
Diftusion Timestop
400
200
1000
800
600
400
200
Diffusion Timestep
Figure 14. Performing Longtail Guidance the predicted, de- coded terminal data state # (top row) provides production model fo with cleaner data much earlier in the diffusion denoising pro- cess, enabling it to meaningfully exercise guidance as compared naively performing guidance the predicted decoded data state r,, Y-axes are the guiding model longtail signal (in this case, entropy). See Figure 13 for visual depiction.
Figure 15. Decnded terminal data states 2e have hetter with respect to real training data than do intermediate data states Tr. suggesting that LTG works with an existing predictive model for even though is trained intermediate diffusion states, precisely because the states upon which LTG performs guidance better match what the predictive model has already seen.
A.8. Ablations
In Table 16, ablate Longtail Guidance signal (energy, entropy, epistemic) by Longtail Guidance Weight (1.0, 10.0, 50.0, 200.0) for the Pets dataset. In Table 17, we ablate over the number of Epistemic Heads the same task. In both cases, we report predictive model generalization performance when trained on the 30 synthetic data expansion task as defined in Section 4.1. We find highest performance for energy and entropy at guidance weight 10.0 and highest performance for epistemic at guidance weight 50.0. Similarly, we find highest performance for K = Epistemic Heads. Performance declines with additional heads, likely because the oracle loss used to train the Epistemic Head causes each head to be exposed to & examples in expectation; too many heads and they lose the ability represent the predictive model.
A.9. Additional Examples
In Figure 18, we show high-resolution examples of synthetic data generated by baseline diffusion and by Longtail Guidance. Guidance performed by SOTA ViT-based ImageNet-LT classifier described Section 4.2.
19

---

Generative Data Mining with Longtail-Guided Diffusion
Longtail Guidance Weight
Longtail Signal
1.0
10.0
50.0
200.0
Epistemic Heads
3
7
Energy
78.8
80.0
78.9
76.7
LTG (Epistemic)
81.2
82.0
80.6
79.9
Entropy
76.1
81.6
80.9
74.9
Epistemic 76.8
78.5
82.0
79.1
Figure 17. Ablation over Epistemic Heads on Predictive Model Generalization Performance on the Pets Dataset Expan- sion Task, holding guidance weight fixed 50.0.
Figure 16. Ablation over Longtail Guidance Weight on Predictive Model Generalization Performance the Pets Dataset Expansion Task, holding the number of Epistemic Heads fixed K 5. Entries bolded for best hyperparameter in each row.
Baseline Diffusion
Longtail Guidance
Figure 18. Additional example of baseline diffusion (left) and Longtail Guidance (right). Baseline diffusion tends generate canonical, well-posed views that help predictive model generalize, but only up to a point. In contrast, Longtail Guidance produces more extreme, occluded, challenging views from the perspective of predictive model, enabling significantly improved generalization performance. In order, classes are: jellyfish, flamingo, lion, monarch butterfly, ant, candle, joep, forklift, and ambulance.
20


Reviews:
[{"review_id": "Cu2HHr5bOC", "reviewer": "Reviewer_srKG", "summary": "This paper proposes a proactive long-tail discovery process that helps the model learn rare or hard concepts.\nSpecifically, the authors develop model-based long-tail signals and use these signals to generate additional training data from latent diffusion models.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].\n\nWe are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.\n\n8. [srKG] states, “if the authors want to validate the scenario in practice, they need to employ a diffusion model trained on a small dataset and use it to train a predictive model for a much larger dataset,” and, “applying this [production of rare or hard concepts] to a classification task, which is easier than a generative task, is like using a sledgehammer to crack a nut.\n\nIt is frequently the case that deployed predictive models have access to much less capacity and training data than do larger foundation models (including Internet-scale generative models). This can be due to limited memory and compute budgets and limited opportunities for real data collection (as we note in lines 38-48, 69-73). This is the premise behind the substantial literature of model and data distillation, including the works we cite in the paper (Yu et al., 2023b; Gou et al., 2021), and the more nascent literature on synthetic training data, including the works we cite in the paper (Du et al., 2024; Azizi et al., 2023; Zhou et al., 2023, Zhang et al., 2023b; Li et al., 2022b).\n\nWhen working with a deployed predictive model, it is critical to understand scenarios that the model struggles with. As we detail in Response 1 to [igG6], our primary contribution is towards defining, mitigating, and proactively understanding a given predictive model’s longtail. Diffusion guidance only plays a supporting role. Thus, while we appreciate the reference on minority diffusion sampling, and will include it in our camera-ready, we also note in our analysis (Section 4.3, Figure 9) that what is difficult for one model (even a foundation model like CLIP) is not necessarily the same thing as what is difficult for a given, deployed predictive model. We in fact show that Longtail Guidance demonstrates significant generalization improvements over strong synthetic data generation baselines (that use a foundation model for diffusion guidance) across eight datasets, with as many as 1000 distinct classes (Tables 1, 2, Supplemental A.1).", "claims_and_evidence": "Producing rare or hard concepts is also an important issue in generative tasks. Applying this to a classification task, which is easier than a generative task, is like using a sledgehammer to crack a nut.\nIndeed, the dataset used in the experiments is much smaller than the one on which the Stable Diffusion model is trained.", "methods_and_evaluation_criteria": ".", "theoretical_claims": ".", "experimental_designs_or_analyses": "If the authors want to validate the scenario in practice, they need to employ a diffusion model trained on a small dataset and use it to train a predictive model for a much larger dataset.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "Potential related work:\n\nUm, Soobin, and Jong Chul Ye, Self-guided generation of minority samples using diffusion models, ECCV 2024.", "other_strengths_and_weaknesses": ".", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XuGaOlKXSB", "reviewer": "Reviewer_Qmgq", "summary": "This paper proposes a novel approach to generate long-tail data using diffusion models. The authors introduce an epistemic head and a long-tail guidance mechanism, enabling the model to detect and generate long-tail data effectively. Experimental results demonstrate that the proposed Longtail-Guided Diffusion model significantly enhances dataset quality, as evidenced by improved downstream task performance and meaningful data generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].\n\nWe are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.\n\n7. [Qmgq] asks about computational efficiency\n\nComputational efficiency is discussed for Longtail Guidance and the Epistemic Head in Supplement A.6. In brief, we generate baseline synthetic images (no Longtail Guidance) at a rate of 6.32 images / second (50 DDIM steps, fp16, no gradient checkpointing, 8xH100). We generate Longtail-Guided synthetic images at 1.01 images / second. The largest cost is the gradient calculation through the VAE, which consumes 45GB of VRAM in fp16 for batch size 8. The Epistemic Head impacts training and inference times by less than 2% and contains less than 5% of the original predictive model’s parameters. See also our response about 4x increased data efficiency over GIF in Response 6 to Yasy.", "claims_and_evidence": "The claims presented in the paper are clear and well-articulated.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-founded and appropriate.", "theoretical_claims": "The theoretical claims and proofs presented in the paper are sound and well-justified.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and support the claims made in the paper.", "relation_to_broader_scientific_literature": "The contributions of this paper are novel and advance the field of longtail data generation in diffusion models.", "essential_references_not_discussed": "No essential references appear to be missing from the discussion.", "other_strengths_and_weaknesses": "The strengths and weaknesses of the paper have been thoroughly addressed in the sections above.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Could the authors provide computational efficiency about applying the longtail-guiided diffusion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I1LDu1kukF", "reviewer": "Reviewer_Yasy", "summary": "The paper proposes a diffusion model guidance technique (LTG) that generates synthetic training data tailored to the specific long-tail of a deployed predictive model. It introduces a differentiable module for estimating epistemic uncertainty that helps identify rare or hard examples without altering model weights or predictive performance. It also proposes method that extracts textual descriptions of model weaknesses from LTG-generated data using vision-language models (VLMs).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].\n\nWe are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.\n\n\n5. [Yasy] says, “the paper does not quantitatively verify whether the generated data truly remain in-distribution.”\n\nThank you for raising this concern. Longtail guidance demonstrates significant generalization improvements over strong synthetic data generation baselines across eight datasets, with as many as 1000 distinct classes (Tables 1, 2, Supplemental A.1). It is common practice to trust (real) evaluation data and, in fact, our method enables users to reserve more real data for evaluation by more heavily relying on synthetic training data.\n\nFurthermore, in Section 4.2 and Table 3, we show that forming new diffusion prompts from VLM descriptions of longtail-guided synthetic data and then training the original predictive model on that new data outperforms the baseline where diffusion prompts are formed from VLM descriptions of baseline synthetic data (without longtail guidance). This quantitatively supports that longtail-guided data are meaningful and remain in-distribution.\n\nWe agree that there is a tension between synthetic data distributional alignment and predictive model bias. Because we are working with natural image datasets, alignment can be equated with realism. Alignment and realism are explicitly controlled by the diffusion model’s text guidance weight. Predictive model bias is explicitly controlled by the longtail guidance weight.\n\nWe address the tradeoff between alignment and bias by holding the text guidance weight constant and selecting a longtail guidance weight such that the probability of the desired class (under the current predictive model) is lower than baseline synthetic data but not so low that it drops to zero. It is a hyperparameter that was selected one time – it does not need to be done for each generation or dataset. We describe this in lines 238-258 and show in Figure 2 that there is a strong inverse relationship between longtail guidance weight and the probability of correct classification under the predictive model for generated data (before retraining).\n\nMore broadly, while predictive model bias is a concern, it is also a concern that synthetic data generated without signals from the predictive model more rapidly saturate in new concepts (see rebuttal 6 to [Yasy]), leading to weaker generalization improvements. Our experiments strongly support that giving the predictive model “a voice” in the generating process is to its benefit. \n\n\n\n6. [Yasy] is concerned that, “the central problem in the use of diffusion models for synthesising images useful for training classifiers is the sheer amount of synthetic data needed for meaningful classifier improvements. The paper doesn't quite address this problem. The method still generates synthetic data at a scale of 20× to 30× the original dataset size.”\n\nWe note that the results in Table 2 (Dream-ID comparison) all generate longtail-guided data at less than 1x the size of the training data and that, in Supplemental A.1, we show ImageNet-LT results that generates much fewer synthetic data – just 20% of the original dataset and, yet, still yield significant generalization improvements!\n\nThe 20-30x expansion in Table 1 is for comparing apples-to-apples with the GIF baseline (this is what we mean by “for parity.”). Further inspection of our existing results shows that predictive model generalization from our longtail guidance method surpasses GIF in all cases using just 20-30% (average 25%) of the data expansion that GIF uses (caltech: 4x expansion vs 20x), cars (5x expansion vs 20x), pets (7x expansion vs 30x), flowers (6x expansion vs 20x). Training with even more longtail-guided data yields further generalization improvements as we report in the paper.\n\nWe reported the equivalent expansion numbers in the paper for direct comparison, but we will take advantage of the ninth page in the camera ready to also add a graph that demonstrates that longtail guidance drives generalization improvements in a much more data-efficient way (4x more efficient) compared to GIF!", "claims_and_evidence": "The paper claims LTG does not require retraining of the diffusion or predictive model; would retraining the predictive model on intermediate diffusion states further improve performance?\nWhile the generalization improvements suggest that LTG-generated data are useful, the paper does not quantitatively verify whether the generated data truly remain in-distribution as claimed.", "methods_and_evaluation_criteria": "LTG is designed to expose model weaknesses, so conditioning generation on model longtail signals is a natural and effective approach.\nThe benchmark datasets include real-world longtail scenarios, and tested on imbalanced, fine-grained, and out-of-distribution datasets.\nThe central problem in the use of diffusion model for synthesising images useful for training classifiers is the sheer amount of synthetic data needed for meaningful classifier improvements. The paper doesn't quite address this problem. The method still generates synthetic data at a scale of 20× to 30× the original dataset size.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design and analyses in the paper appear generally sound. The iterative fine-tuning approach for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once.", "relation_to_broader_scientific_literature": "The paper nicely incorporates ideas from deep learning-based uncertainty estimation literature to address an important issue in using diffusion models for generating synthetic data. It would be interesting to explore how well this method generalizes to more recent diffusion models and to evaluate its data efficiency, given that it still generates data at >= 30 times the size of the existing dataset.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99r5zPVUGw", "reviewer": "Reviewer_iGg6", "summary": "The paper proposes Longtail Guidance (LTG), a method for generating rare or difficult training examples that a predictive model struggles with. The authors propose using an epistemic head to estimate uncertainty and use its signals guide a diffusion model to synthesize hard or rare data without retraining either the predictive or generative model. The authors present experiments showing that LTG-generated data significantly improve model generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].\n\nWe are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.\n\n1. [igG6] is concerned that the paper’s main idea is diffusion guidance based on energy signals and requests contextualization of additional related works.\n\nOur main idea is grounding the notion of \"predictive model longtail\" into measurable signals. We demonstrate that we can not only generate additional longtail examples by this definition without changing the predictive model or the diffusion model, but that these additional examples significantly improve predictive model generalization on real eval data. Diffusion guidance plays  a supporting role. We also show that predictive model longtails are model-specific (Figure 9) and proactively explain a predictive model's longtail with human-interpretable text (Table 3).\n\nWe agree that existing works have already demonstrated flexible guidance schemes in data space and, in fact, compare against one such work (Universal Guidance) in Section 3 (lines 260-290). We also provide new evidence based on FID scores for why this type of guidance works in Supplemental A.8 (Figures 13-15). We will expand the related works discussion (including the additional references) with the ninth page allowed in the camera ready\n\nUniversal Guidance, which we directly compare to in Section 3 and Figure 7, already cites the Diffusion Posterior Sampling paper [1] (and its relationship to it). In short, Universal Guidance and Diffusion Posterior Sampling are related in that both use a point estimate for p(x_0 | x_t) when performing guidance. Diffusion Posterior Sampling is focused on Gaussian and Poisson measurement noise (though their framework extends to nonlinear inverse problems where gradients can be calculated). Universal Guidance represents this extension (through the use of a nonlinear observation model represented by the diffusion model’s associated VAE) and takes it two steps further with expensive backward and recurrent sampling steps discussed in our paper, lines 260-270. Longtail Guidance exists between Diffusion Posterior Sampling and Universal Guidance because it differentiably decodes from a diffusion latent to the data space through a nonlinear, differentiable observation model (the VAE), but does not perform additional sampling steps (which are expensive and cause synthetic data to fall out of distribution, see Figure 7).\n\nLoss-Guided Diffusion [2] builds on [1] by using Monte Carlo samples from an approximating distribution to further improve guidance. We do not use it in our approach since we would have to calculate the gradient through the VAE for each sample, which we already note to be an expensive operation (lines 292-295 and Supplement A.6). We will include and contextualize [1], [2, 3, 4] to our list of related works on diffusion control methods (ControlNet, DOODL, Freedom, Edict) and distinguish by whether or not they require additional diffusion or predictor training.\n\nWe emphasize that we are primarily concerned with downstream predictive model performance – a task that neither Diffusion Posterior Sampling nor Loss-Guided Diffusion evaluate on (but which our strongest baselines, GIF, Dream-ID, and LiVT do).\n\n\n2. [iGg6] is concerned about whether we address model bias, distribution shift, and trustworthiness of synthetic data.\n\nPlease see response 5. to [Yasy].\n\n\n3. [iGg6] asks for clarification and inlined descriptions of our experiment setup.\n\nWe fully describe our experiments in Supplement A.2. We will inline in the main paper with the ninth page allotted for the camera ready. Please also note that the 20-30x expansion is only for one apples-to-apples evaluation (with GIF). Other evaluations use less than 1x expansion of synthetic data. See response 6. to [Yasy].\n\n\n4. [igG6] is concerned that some examples with mid-to-high LTG weight look implausible\n\nThe implausibility of some synthetic data with high Longtail Guidance weights are displayed for demonstration purposes only. The guidance weights in the experiments are in the low-to-mid range of the qualitative examples. We will clarify in the camera-ready", "claims_and_evidence": "Most claims are empirically supported, but some rely on unverified assumptions or could benefit from stronger theoretical grounding. \n\n1. The improved generalization accuracy results ImageNet-LT and other benchmarks are quite broad (including the supplementary results). \n2. The epistemic head’s comparison with entropy and energy baselines are fairly convincing.\n3. The claims on in-distribution and hard longtail samples are not explicitly verified. The paper assumes that guiding diffusion models using uncertainty signals produces semantically valid longtail examples, but it lacks a principled evaluation of whether these synthetic samples reflect real-world distribution shifts or are just artifacts of the model’s biases. \n  a) Is there any principled way to understand whether you can even trust the long-tail data generated? For example, many examples on the mid-to-high LTG look unplausible (e.g. Figure 1, bottom right figures, Figure 6 -- several unplausible images for each class--). \n  b) the authors suggest using these examples for retraining. How do you deal with self-reinforcing biases (e.g. given point 3.a)?", "methods_and_evaluation_criteria": "The methods seem appropriate. Using guiding a diffusion for generating hard training samples is also a sensible choice. \n\nThe authors evaluate on several sensible classification tasks and the improvements in rare or difficult examples seem compelling. However, the authors should include additional basic explanatory information on the experimental set-up. In Section 4.1 it is not immediately obvious why there is a need to generate 20x or 30x the amount of training data. The \"Dataset Expansion task as defined in GIF (Zhang et al., 2023b).\" should be at least minimally explained, and the reader shouldn't need to consult another paper to understand the results in this work. It is not clear what the authors mean by \"for parity\", which is used often. Similar comments apply for the rest of this section, which we suggest the authors revise.", "theoretical_claims": "The authors do not present major theoretical results.", "experimental_designs_or_analyses": "See prev. points on \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "The prior work section is lacking focus, the work on diffusion isn't overly relevant, the remaining work isn't compared critically to the current work.\n\nThe paper’s main idea to use energy signals to guide diffusion sampling are common in the classifier guidance and plug-and-play diffusion literature (e.g., Loss-Guided Diffusion, Diffusion Posterior Sampling). However, the paper does not explicitly contextualize these works.\n\nThe finding on the possibility to guide diffusion without explicitly training an inference model on different noise levels is well known in the field. Many papers show that you can use signals from the data space for guidance without the need to learn an inference models at different noise stages, e.g. [1,2], and a substantial corpus of literature on finetuning diffusion models which achieves the same, e.g. [3, 4].\n\n[1] Hyungjin Chung, Jeongsoo Kim, MichaelThompson Mccann, MarcLouisKlasky, andJongChul Ye. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learning Representations (ICLR), 2023.\n\n[2] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. International Conference on Maching Learning (ICML), 2023.\"\n\n[3] Venkatraman, S., Jain, M., Scimeca, L., Kim, M., Sendera, M., Hasan, M., Rowe, L., Mittal, S., Lemos, P., Bengio, E., et al. Amortizing intractable inference in diffusion models for vision, language, and control. Neural Information Processing Systems (NeurIPS), 2024\n\n[4] Domingo-Enrich, C., Drozdzal, M., Karrer, B., and Chen, R. T. Q. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "--", "other_comments_or_suggestions": "--", "questions_for_authors": "1. Is there any principled way to understand whether you can even trust the long-tail data generated? For example, many examples on the mid-to-high LTG look unplausible (e.g. Figure 1, bottom right figures, Figure 6 -- several unplausible images for each class--). \n\n2. the authors suggest using these examples for retraining. How do you deal with self-reinforcing biases (e.g. given point 3.a)?\n\n3. see previous comments on \"Methods And Evaluation Criteria\"\n\n4. see previous comments on \"Relation To Broader Scientific Literature\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a guidance technique for diffusion models, called LTG, which generates synthetic training data tailored to the long-tail distribution specific to a deployed predictive model. It introduces a differentiable module for estimating epistemic uncertainty, enabling the identification of rare or challenging examples without modifying model weights or affecting predictive performance. Additionally, it proposes a method to extract textual descriptions of model weaknesses from LTG-generated data using vision-language models (VLMs).

While most reviewers find the result interesting and the studied problem important, the work can be improved by comparing the results more comprehensively and properly discussing the related work. We recommend that the authors incorporate the reviewers' feedback in future versions.
Decision: Accept (poster)