Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models
Authors: Dang Nguyen
Jiping Li
Jinghao Zheng
Baharan Mirzasoleiman
Venue: ICLR 2026 Poster (2026)
Keywords: Synthetic data generation
Image augmentation
Diffusion models
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a novel targeted image augmentation method using synthetic images from diffusion models

Abstract: Synthetically augmenting training datasets with diffusion models has become an effective strategy for improving the generalization of image classifiers. However, existing approaches typically increase dataset size by 10–30× and struggle to ensure generation diversity, leading to substantial computational overhead. In this work, we introduce TADA (**TA**rgeted **D**iffusion **A**ugmentation), a principled framework that selectively augments examples that are not learned early in training using faithful synthetic images that preserve semantic features while varying noise. We show that augmenting only this targeted subset consistently outperforms augmenting the entire dataset. Through theoretical analysis on a two-layer CNN, we prove that TADA improves generalization by promoting homogeneity in feature learning speed without amplifying noise. Extensive experiments demonstrate that by augmenting only 30–40% of the training data, TADA improves generalization by up to 2.8% across diverse architectures including ResNet, ViT, ConvNeXt, and Swin Transformer on CIFAR-10/100, TinyImageNet, and ImageNet, using optimizers such as SGD and SAM. Notably, TADA combined with SGD outperforms the state-of-the-art optimizer SAM on CIFAR-100 and TinyImageNet. Furthermore, TADA shows promising improvements on object detection benchmarks, demonstrating its applicability beyond image classification. Our code is available at https://github.com/BigML-CS-UCLA/TADA.

Published as a conference paper at ICLR 2026
Do WE NEED ALL THE SYNTHETIC DATA? TARGETED IMAGE AUGMENTATION VIA DIFFUSION MODELS
Dang Nguyen"t Jiping Li
Jinghao Zheng*!
Baharan Mirzasoleiman'
University of California Los Angeles (UCLA) "Ecole Polytechnique Federale de Lausanne (EPFL)
ABSTRACT
Synthetically augmenting training datasets with diffusion models has become effective strategy for improving the generalization of image classifiers. However, existing approaches typically increase dataset size by 10-30x and struggle ensure generation diversity, leading to substantial computational overhead. In this work, we introduce TADA (TArgeted Diffusion Augmentation), a principled framework that selectively augments examples that are not learned early in training using faithful synthetic images that preserve semantic features while varying noise. We show that augmenting only this targeted subset consistently outperforms augmenting the entire dataset. Through theoretical analysis on a two-layer CNN, prove that TADA improves generalization by promoting homogeneity in feature learning speed without amplifying noise. Extensive experiments demonstrate that by augmenting only 30-40% of the training data, TADA improves generalization by up to 2.8% across diverse architectures including ResNet, ViT, ConvNeXt, and Swin Transformer on CIFAR-IO/ 10/10, TinylmageNet, and ImageNet, using optimizers such as SGD and SAM. Notably. TADA combined with SGD outperforms the state-of-the-art optimizer SAM on CIFAR-100 and ImagemtgeNet. Furthermore, TADA shows promising improvements on object detection benchmarks, demonstrating its applicability beyond image classification. Our code is available at
com Bi gML CS-UCIA/TADA
INTRODUCTION
Data augmentation has been essential obtaining state-of-the-art in image classification tasks. In particular, adding synthetic images generated by diffusion models (Rombach et al. 2022 Nichol
202T Saharia et al., 2022) improves the accuracy (Azizi al. 2023) and effective robustness (Bansal & Grover 2023) of image classification, beyond what is achieved by weak (random crop. flip,
al.
color jitter, ctc) and strong data augmentation strategies (PixMix, DeepAugment, etc) (Hendrycks
et al., 2021 2022) or data augmentation using traditional generative models (Goodfellow et al. 2020 Brock et al. 2018). Existing works, however, generate synthetic images by conditioning the diffusion model on class labels (Bansal & Grover [2033] Azizi et al. (003),, noisy versions of entire training data (Zhou et al., (2023). Unlike weak and strong augmentation techniques, data augmentation techniques based diffusion models often struggle to ensure diversity and increase
the size of the training data by up to 10x (Azizi et al. 2023) to 30x (Fu et al.) 2024) to yield
satisfactory performance improvement. This raises key question:
Does synthetically augmenting the full data yield optimal performance? Can we identify a part of the data that outperforms full data, when synthetically augmented?
At first, this seems implausible as adding synthetic images corresponding to only part of the training data introduces shift between training and test data distributions and harms the in-distribution performance. However, recent results in the optimization literature have revealed that learning features more uniform speed during training improves the generalization performance (Nguyen et al. 2024). This is shown by comparing learning dynamics of Sharpness- Aware-Miniminn (SAM) with Gradient Descent (GD) optimizers. SAM a state-of-the-art optimizer that finds flatter local minima by simultaneously minimizing the value and sharpness of the loss (Foret et al. 2020).
"Equal contribution

---

Published as a conference paper at ICLR 2026
Real Fast
Real Slow
Syn Slow
bird
cat
bird
cat
bird
cat
Figure 1: Examples of slow- and fast-learnable images and our faithful synthetie images corresponding to slow-learnable examples generated for CIFAR-10. 10. Our synthetic data preserves features in slow- learnable images but replace noise. This amplifies slow-learnable features without magnifying noise. This is difficult to achieve with standard augmentations like random cropping or flipping, highlighting the value of generative augmentation. Additional images are given in Figure 7
In doing so, SAM learns slow-learnable features faster than GD and achieves superior performance. This suggests that augmenting the slow-learnable part of the data to accelerate their learning can improve the generalization performance. despite slight distribution shift. Yet, how to generate such synthetic data remains an open question.
In our work, we provide a rigorous answer to the above question. We propose TADA (TArgeted Diffusion Augmentation). principled framework for selectively augmenting slaw-lea examples using diffusion models. First, by analyzing a two-layer convolutional neural network (CNN), we show that SAM suppresses learning noise from the data, while speeding up learning slow-learnable features. Then, we prove that generating faithful synthetic images containing slow-leamnblee features with different noise effectively speeds up learning such features without causing noise overfitting. To find examples with slow-learnable features, partition the data two parts by clustering model outputs early in training and identify the cluster with higher average loss. Then, we generate faithful images corresponding the slow-learnable examples, by using real data guide the diffusion process. That is, we add noise to examples that are not learned early in training and denoise them to generate faithful synthetic data (see examples in Figure []. This enables synthetically augmenting only the slow-learnable part of the data by up to 5x to get further performance improvement. In contrast, upsampling slom-lear examples examples-whiet whicch appears be a simpler and more intuitive approach- more than once could amplify the noise and significantly harm the performance, suggesting the necessity of using synthetic data. Finally, we prove the convergence properties of training on our synthetically augmented data with stochastic gradient methods.
We conduct extensive experiments training ResNet, VIT, DenseNet, ConvNeXt, and Swin Trans- former on CIFARIO. 10, CIFAR 100 (Krizhevsky al. (2009), TinylmageNet (Le & Yang 2015), and ImageNet (Deng et al. 2009). We show that TADA consistently outperforms both upsampling and full synthetic augmentation, improving SGD and SAM by up 2.8% while augmenting only 30-40% of the data. Notably, TADA combined with SGD outperforms SAM on CIFAR-100 and TinylmageNet and achieves state-of-the -fttoo--ttttttt art performance. Moreover, we validate its scalability ImageNet with ResNet18 8 and ResNet50, and demonstrate its effectiveness beyond classification by improving object detection performance. TADA remains effective across different diffusion models and can be seamlessly combined with existing weak and strong augmentation strategies to further boost performance.
RELATED WORKS
Generative Models for Augmentation. There has been a recent surge of studies on synthetic data augmentation using diffusion models. For example, Azizi al. (2023) applied diffusion models to ImageNet classification, while further studies (Trabucco et al. 2023 He et al.) (203) explored their application in zero- or few-shot settings. Despite promising results, this line of research faces fun- damental challenges in achieving diversity, faithfulness, and efficiency. Recent work attempts to over- come this through intricate prompt-conditioning mechanisms, customized embedding optimizations, or multi-stage diffusion processes. For example, DiffuseMix (Islam al.) 2024a) and GenMix (Islam) eet al., 2024b) use prompt- -uiiee editing with complex mixing strategies to avoid unrealistic artifacts, while Dixf-Mix (Wang et al.) 2024) balances foreground fidelity and background diversity through inter-class mixup. Diff-ll (Wang & Chen (205)) introduces novel inversion-circh interpolation and two-stage denoising process to jointly promote diversity and faithfulness. For fine-grained
2

---

Published as a conference paper at ICLR 2026
image classification, SaSPA (Michaeli & Fried, 2024) preserves structural integrity by conditioning on edges and subject representations, and DifiCoRe-Mo (Islam & AKHTAR 2025) uses constrained diffusion with negative prompting and hard-cosine filtering maintain semantic consistency.
Although these approaches improve synthetic image quality and diversity, they typically require generating extremely large synthetie datasets- often 10x to 30x the size of the original data- to achieve meaningful performance gains, making them computationally expensive. A few recent works, such as Boomerang (Luzi al. 2022) and -efiC, address this cost by performing local manifold sampling, enabling strong performance with a x augmentation ratio. However, these methods still involve substantial system-level complexity and high generation costs.
Our approach departs from this trend by focusing on which examples to augment rather than designing increasingly complex generation pipelines. We theoretically and empirically show that augment- ing only the 30%-40% of examples that are not learned early in training is sufficient-aa superior- to full-data augmentation. TADA is simple, computationally lightweight, and generator- agnostic. It can be seamlessly combined with state-of-the-art diffusion-based augmentation methods, as demonstrated with both DiffuseMix and Boomerang in our experiments.
often
(SAM). SAM ís an optimization technique that obtains state-of-the art performance on a variety of tasks, by simultaneously minimizing the loss and its sharpness (Foret Zheng et al. 22021) In doing so, it improves the generalization in expense of doubling the training time. SAM has also been shown to be beneficial in settings such as label noise (Foret al.]
Sharrnees-vvar--miiiiiite
ct al. 2020;
2020 Zheng et al. 2021) out-of-distribution (Springer al. 2024), and domain generalization (Cha et al.
2021 Wang et al. 2023).
The superior generalization performance of SAM has been contributed smaller Hessian spectra (Foret et al.] 20200) Kaur et al. [023] Wen et al. 2022] Bartlett et al.] (023). sparser solution (Andriushchenko & Flammarion] (0222, and benign overfitting in presence of weaker signal (Chen)
et al. [2022). Most recently. SAM is shown to learn features at more uniform speed (Nguyen et al. 2024). In our work, we show that targeted synthetic data augmentation can improve generalization by making training dynamics more similar to SAM.
3 PRELIMINARY
In this section, we introduce our theoretical framework for analyzing synthetic data augmentation with diffusion models. We also discuss SAM's ability in learning features at more uniform speed.
Data Distribution. We adopt a similar data distribution used in recent works on feature learn- ing (Allen-Zhu& & Li 20200 Chen et al. 2022 Jelassi & Li [022] Cao et al.) [022] Kou et al.) 2023
Deng et al. 2023 Chen et al. 2023) to model data containing two features Va. U. and noise patches. Definition 3.1 (Data distribution). A data point has the form (z. y) D(Be, Sd a) (R')*x (=1).
where y (adamacher(O.5), 0 Ba B. R. and x: (1x(1) (2) .... (P)) contains P patches.
Exactly one patch is given by the fast-learnable feature 3e . y Ue for some unit vector v, with probability a > O. Otherwise, the patch is given by the slow-learnable feature Sa y Va for some unit vector Ve Va =
The other P - 1 patches are i.i.d. Gaussian noise & from No0, (dZ//)Ja) for some constant dp.
Probability a controls the frequency of feature v, in the data distribution. The distribution parameters Be, 3a characterize the feature strength in the data. B. > 3a ensures that the fast-learnable feature is represented better in the population and thus learned faster. The faster speed of learning captures various notions of simplicity, such as simpler shape, larger magnitude, and less variation. Note that image data in practice are high-dimensional and the noises become dispersed. For simplicity, assume P 2, that the noise patch is orthogonal from the two features, and that summations involving noise cross-terms (E:, > become negligible.
Two-layer CNN Model. We use a dataset D { v.) two-layer nonlinear CNN with activation functions ((2) =
from distribution 3.1 to train a
Definition 3.2 (Two-layer CNN). For one data point (x, y), the two-layer Convolutional Neural Network (CNN) with weights W [wi, U22 ... w. E RdxJ where w, is weight of the j-th
3

---

Published as a conference paper at ICLR 2026
neuron (filter), has the form:
f(x; W) Σ
Σ j=1p=1
P o
200.3
J 3
Σ
(u)
( [ wa) E)² +y
if vd
for our /
( De)
if ve
Empirical Risk Minimization, We consider minimizing the following empirical logistic loss:
N
N
((W)
Σ (an W)) N
N Σ
log(1 [1 + exp( (xppp-Wfffr f W)).
via (1) sharpness-aware minimization (SAM) (Foret et al. 2020) and (2) gradient descent (GD), whose filter-wise update rules, with some learning rate 1j > O, are respectively given by:
SAM:
w (++1) = 3
V L(W + (
VV(((())))
where p = = p/
w
N
GD wj (1+1)
= =w
((W())
: (t) V ( ((W()))
Here V ((W()) denotes the full gradient w.r.t. filter wy at iteration Vi, 1, w the per-example gradient for E [N]. and (L()()) denotes the full gradient matrix.
((W())) denotes
High-level idea of SAM. By perturbing the weights with gradient ascent (c(*) p v (OW()))
SAM looks ahead in the worst weight direction and forces the training algorithm to escape an unstable (sharp) local minimum. In practice, this leads to more generalizable solutions.
SAM Learns Features More Homogeneously. With the above setting, the alignment of vı 2e with weights, i.e., (w)(*), va) and (t) va), indicate how much they are learned by the CNN at iteration t. SAM's (normalized) gradient for the slow-learnable feature is larger than GD by factor of (Nguyen 2024):
(!0) (3 w, Va) 2/3
k-
p(0),32 v.)
That is, SAM amplifies the slow dlarnaaree feature and learns it faster than GD. doing so, it learns fea- tures more homogeneous speed. While Eq. suuugesss that the empirical choice of & should depend on the relative strength and difficulty of the features, simply upsampling examples with slow-learnable features more than once results in performance degradation,
confirm in experiments.
LEARNING FEATURES HOMOGENEOUSLY WITHOUT OVERFITTING NOISE
In this section, we first prove that SAM suppresses learning noise from the data, while promoting homogeneous feature learning. Then, we introduce TADA (TArgeted Diffusion Augmentation) and discuss generating synthetic data amplify features images without magnifying their noise. This allows amplifying dlw-laaae features by le > 2 to further boost performance. Finally, we show convergence of training on our synthetically augmented data.
4.1
SAM SUPPRESSES LEARNING NOISE FROM THE DATA
First, we theoretically analyze how SAM suppresses learning noise in the above setting. Intuitively, as SAM pushes the learning dynamics away from sharp landscapes, it simultancously helps the model avoid areas where certain noises concentrate. This becomes a natural defense against noise overfitting in high-curvature areas. On the other hand, gradient descent is unaware of local smoothness, so solutions that may sit in a flat, noise-resilient basin.
Fomally, we prove that starting with the same weights W((), SAM step suppresses the model's alignment with noise directions more effectively than equivalent gradient descent step. Let & denote the sets of noises for dataset D. Let Wie denote the perturbed weights of filter j for SAM.
We then define T,,,t
(Φ: + [N], sgn((w) (.)) sgn(y:)) and ,,,, { $
4

---

Published as a conference paper at ICLR 2026
i E [M], (gn(( xb)) * sgn(y:)} be the sets of noises where the sign of alignment matches or mismatches the sign of the label. We define - - accordingly for each GD weight wn!")
We measure filter-wise noise learning for set of noises using the metric NoiseAlign (Z, )
T Σ in in (V wit (O)) ob|, which intuitively measures how much noise is learned by the model.
The following theorem quantifies how SAM learns noise to a smaller extent compared to GD. For sim- plicity, we analyze the early training phase. However, our results should hold throughout the training.
Theorem 4.1. With controlled logit terms 1 = sigmoid(- ys (x,; W()), large data size N, small learning rate n, and small SAM perturbation parameter p (see Appendix Aj. SAM and GD updates from the same parameters have the following property, early in training:
1. Inert Noises: Alignment with noises that belong I and Itt) will get closer after each update, so they will not be learned eventually by GD or SAM.
2. Noise Learning:
The other noises will continue being learned in the sense that Kww, (t)) is monotonically increasing. For these noises, SAM slows down noise learning by looking ahead to noise-sensitive (sharp) directions, while GD updates "blindly". The following SAM and GD learning dynamics hold & E I/,22. and &, E I in lerms of noise gradient:
SAM: (V wy.e ((W(++ ( E.)1
3 (0)
(un)
19 .E,) (1-
3p'*) N
2 1 12 &MIL& |15.13
N
GD: (V w, ((W()), N 100 (w)) (w0)) E,) 1
(1)
Furthermore, on average, the perturbed SAM gradient aligns strictly less with these noises,
NoiseAlign (T ()) < &NoiseAlggn (I 2,++ (0)))
A special case of this theorem is that with the same initializations W ~ NO0 a.), nearly half of the noises will not be learned, and SAM in early training prevents overfitting, while GD does not.
All the proof can be found in Appendix A Our results are aligned with (Chen et al., 2023) which showed, in a different setting, that SAM can achieve benign overfitting when SGD cannot.
Remark. Theorem 4.1 implies that to resemble feature learning with SAM and ensure superior convergence, it is also crucial to avoid magnifying noise when amplifying the slow-learnable feature.
4.2 SYNTHETIC DATA AUGMENTATION TO AMPLIFY FEATURES BUT NOT NOISE
Next, we discuss how TADA identifies examples containing slow-learnable features and generates faithful synthetic data to amplify these features without magnifying the noise in the data.
Identifying slow-learnable examples. To find slow-learnable features in the data, we find examples that are not learned robustly at the early phase of training. If an example contains at least one fast-learnable feature that is learned by the model early in training, the model relies on such features to lower the loss and potentially correctly classify the example early in training. Thus, examples that only include slaw-leaenable features can be identified based on loss or misclassification, or by partitioning model outputs to two clusters after few training epochs and selecting the cluster with the higher average loss. In our experiments, we use clustering to identify examples with slow-learnable features as it yields better performance, as we confirm in our ablation studies. We note that finding examples with alow-learnable features is not the main focus of our work. Our main contribution is characterizing how to amplify slow-lcarnable features without amplifying noise in the data.
Amplifying slow-learnable features. Next, we show that generating synthetic data containing Low-learnable features with different noise considerably boosts the generalization performance, while upsampling how-learabble examples amplifies the noise in the data and harms generalization.
Recall that D is the original dataset with D N.. We assume exactly (1 a.N Z samples have only Ud and let Dv,, De be the modified datasets via upsampling and generation with factor io and new size Naca aN k(1 N... For Dy, the replicated noises (E. aN + 1,.... N)
5

---

Published as a conference paper at ICLR 2026
introduce dependence. Additionally, for De. the generative model will have its noises % (potentially higher) for synthetic data, and we assume the noises are i..d., orthogonal to features, and independent from feature noise from some distribution Dy.
With similar notations, let &a, Py denote the multi-sets of all the noises for Do. DU respectively.
We define 7, {Φ;; E &g i [Nnew], sgn Φi) = sgn(v:)). -/ I 2,-- similar fashion for DG and Dy as before.
70.00 in
Next, we show that early in training, upsampling and generation contribute similarly to feature learning, but upsampling accelerates learning noises in I./(1))
3.+ while synthetic generation does not. " denote the full gradients w.r. the j-th neuron for upsampling and generation at iteration t respectively. When dropping the index i, the term treated random variable. Then for one gradient update,
Theorem 4.2 (Comparison of feature & noise learning). Let Vy (C((()))) and VG (L()
1. Feature Learning & Inert Noises: Both gradients attempt contribute equally to feature learning and will not eventually learn certain noises (that are those in I I..(0)
2. Noise Learning: Upsampling, compared with generation, amplifies learning of noise on the repeated subset by a factor of k. In particular, for generation, Φ; E IG. where Φ; could be Ei or the generation noise 7s.
3 ((0) (10 (.) 11
KV w. (W))),
Nnew
However, for upsampling, & IU.(0)
3 400 (0 "4)%1E1) i 1= Neww
KW ((W(())
(0) IP i=aN +1, N....,N ,4)
A special case of this theorem is that with the same initializations W NO, af) for all the early iterations 0 S t < T. if the synthetic noises are sufficiently small in the sense that
E (1) ?)111 < (k + 1)E (mo)) " , 1
where (t), (t) denote the logits as random variables under the corresponding noise at iteration t,
then noises are overfitted less:
E (ioiseAlign(I G.(() (0)
< E NoiseAlign (7 [0)
Theorem 4.2
suggests that TADA can prevent noise overfitting expectation by synthetically generating faithful images that preserve features in real data with independent noise. In particular, this metric of generation noise does not need to be strictly less than that of data noise during early training; instead, there cxists a tolerance factor up to A + 1.
For a diffusion model, as long as the noises in the synthetic data |1 1l are not too large, noise learning under generation is more diffused due to its independence and operates more similarly to SAM, whereas upsampling amplifies learning on the duplicated noise, potentially elevating it to a new feature and increasing the NoiseAlign metric.
Remark.
4.3 SUPERIOR CONVERGENCE OF SYNTHETIC DATA AUGMENTATION OVER UPSAMPLING
Next, we show the superior convergence of mini-batch Stochastic Gradient Descent (SGD) which used in practice-wher when training with MADA-generated synthetic augmentation compared to direct upsampling. Consider training the above CNN using SGD with batch size B: w ((+1)
w 7 Σ'= V V. ((W())). The convergence rate of mini-batch SGD is inversely proportional
Due to identical noises in Dv. The following I/4-()) and Ty: are also defined via multi-sets to keep their sizes consistent with the generation counterparts, meaning that count all the replicated noises.

---

Published as a conference paper at ICLR 2026
to the batch size Ghadimi & Lan 2013). The following theorem shows that upsampling inflates the variance of mini-batch gradients and thus slows down convergence.
Note that it does not directly compare the relative magnitudes of the variances, since ag (k), (k) depend on datasets. Instead, it quantifies the sources of extra variance:
mani-batch&ae gradients). Suppose we train the model using mini-batch batch SGD with proper batch size B. Let EiEDe lllg: 9G Ball"] (k), Di llg, gu [|2] S ( be the variances of the per-e example gradients for generation and upsampling (where gi is the gradient of the i-th data and g is the full gradient). Let gu and 9G be the mini-batee gradients. We have:
Theorem 4.3 (Variance
1låe - -9all (a(k)
B
07r(k)
(1
k(k ()(1-a)) |+
ED 1llu gv 11] S I, where 12
B
(a k(1 - x))3
From Theorem we see that all the variance for generation solely comes from the per-example variance, potentially getting larger when synthetic images diversify the dataset. However, upsampling induces one extra term that results from repeated noises and unnecessarily inflates the variance due to dependence within the dataset. This empirically justified in our ablation studies Section 5.2
Corollary 4.4. As long as the generation noise small enough, i.e., og S of (k)), convergence of mini-batch SGD on synthetically augmented data is faster than upsampled data.
4.4 GENERATING FAITHFUL SYNTHETIC IMAGES VIA DIFFUSION MODELS
Finally, we describe how TADA generates faithfil images for the alow-learnable part of the data. From Theorem we know that while the noise in the synthetic data can be larger than that of the original data, it should be small enough yield similar feature learning behavior to SAM.
Synthetic image generation with diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020 Nichol & Dhariwal] 2021 Yang al. 2023) involves forward process to iteratively add noise the images, followed by reverse process to learn to denoise the images. Specifically, the forward process progressively adds noise the data zo over I steps, with each step modeled a Gaussian transition: ((x,|x,- = (X;;; /1 -, 3,1, where 3; controls the noise added at each step. The reverse process inverts the forward process, learns denoise the data, with the goal of recovering the original data in from a noisy sample Zr: Pa (x, )x,) = 1x Ho (x, t). Ex(x,, t)), where mean p4 is conditioned on the sample at the previous time step and variance Eg follows a fixed schedule.
To ensure generating images that are faithful to real data, we use the real images guidance to generate synthetic images. Specifically, while using the class name (e.g., "a photo a dog") as the text prompt, we also incorporated the original real samples as guidance. More formally, instead of sampling pure noisy image IT ~ (0, I) as the initialization of the reverse path, we add noise to reference (real) image z/ such that the noise level corresponds to a certain time-step t.. Then we be- gin demoising from time-step t.., using an open-source text-to-image model, e.g. GLIDE (Nichol et al.) 2021), to iteratively predict a less noisy image x, 1t (t- = T' T - .... 1) using the given text prompt and the noisy latent image x; as inputs. This technique enables produce synthetic images that are simi- lar, yet distinct, from the original examples, and has been successfully used for synthetic image genera- tion for few-shot learning (He et al.) 2023). Pseudocode of TADA is illustrated in Appendix D Alg. [
Generality. While we describe TADA using diffusion models, our targeted augmentation strategy compatible with other diffusion models and synthetic generation pipelines; as shown our experiments, it can be instantiated with different generative backbones.
EXPERIMENT
In this section, we evaluate the effectiveness of our synthetic augmentation strategy on various datasets and model architectures. We also conduct ablation study on different parts of our method.
Base training datasets. We use common benchmark datasets for image classification including CIFAR1O, CIFAR100 (Krizhevsky et al. 2009), Tmin--ImageNet (Le & Yang 2015). ImageNet (Deng
7

---

Published as a conference paper at ICLR 2026
CIFARLO
CIFAR100
Tin-ImageNeet
5.0
4.8
4.6
4.4
4.2
4.6
3.8
3.6
SGD
SAM
SGD
SAM
SGD
SAM
Figure 2: Test classification error of ResNet18 on CIFARIO. CIFARI00 and TinylmageNet, For USEFUL, we use factor of 2, as higher harms the performance. In contrast for TADA, k = 5, 5, 4 for CIFARIG, 10, CIFARI 100, and Tiny-ImageNNt, respectively. TADA improves both SGD and SAM. Notably, it enables SGD to outperform SAM on CIFARI00 and TinyImageNet.
Top-1 Accuracy
Top-5 Accuracy
76
93
Orig
74
USEFUL
92
Boomerang
Accuracy (56
Boomerang
91
72
90
70
66
ResNet18
ResNet50
ResNet18
ResNet50
Figure Top-1 and top-5 accuracies of training ResNet18 and ResNet50 on ImageNet.
ct al.
2009), Flowers- 102 NNilsback & Zisserman 2008). Aircraft (Maji et al. 2013), and Stanford (Krause al. 2013).
Cars
The original training datasets without any modifications. (2) USEFUL (Nguyen al. 2024): Augmented dataset with upsampled real images that are not learned in early training. (3) TADA: Replace the upsampled samples USEFUL with their corresponding synthetic images. For k 2, we use generated images at different diffusion denoising steps, instead of generating from scratch.
Augmented training datasets. We train different models on: (1) Original:
Hyperparameters. For USEFUL, we set the upsampling factor & mance. For TADA, we use & = 5 for CIFARIO and CIFARI00 and k = 4 for LinyImageNet. For generating synthetic images using GLIDE, we use guidance scale of and run denoising for 100 steps, saving generated images every 10 steps. More training details are in Appendix F.
yields the best perfor-
5.1 TADA EFFECTIVE ACROSS DATASETS AND ARCHITECTURES
Different datasets. Figure clearll shows that TADA significantly reduces the test classification етrоr compared both the Original and USEFUL methods across all datasets, namely CIFAR 10, CIFAR100, and Tiny-ImageNet. For Tiny ImageNet, TADA yields an improvement of 2.8% when training with SAM. The superior performance of TADA compared USBFUL well aligned with our theoretical results in Section 4.3 Notably, SGD with TADA outperforms SAM on CIFAR100 and Tiny lmtgeNe,,,,, further confirming the effectiveness of our approach. Importantly, this advantage extends to large-scale settings in Figure TADA achieves the highest Top-1 and Top-5 accuracies on ImageNet when training both ResNet1 8 and ResNet50. Moreover, outperforms Boomerang while augmenting only 65% of the dataset, compared to Boomerang's 100% synthetic augmentation.
Different model architectures. To further evaluate the generalization of our approach, we conduct experiments on multiple model architectures using CIFAR10 as the base dataset. Specifically, apply TADA to CNNs (VGG19, DenseNet121) and Transformers (ViT-S). Figure 4 presents the test classification error for different architectures. The results demonstrate that TADA achieves consistently lower classification error than both the Original and USEFUL methods across all architectures, under both SGD and SAM optimization settings. Moreover, when applied to
8

---

Published as a conference paper at ICLR 2026
VGG19
DensetNet121
VIT-S
USSEUU TADA
20.0
3.5
19.0
9.0
18.0
5.0
8.5
4.5
8.0
17.0
Figure 4: Test classification error of VGG19, DenseNet121, and ViT-S CIFARIO. For USEFUL, we use a factor of & = 2-as higher & hurst the performance-whil while for TADA, we use k = 5.
Table 1:
Test error of ConvNeXt-T and Swin-T on CIFAR-I0 using SGD and k =2
Table 2: Test error of pre-trained ResNet18 on Flowers- 102, Aircraft, and Stanford Cars datasets.
Method
ConvNeXIT
SwinT
Method
Flowers- 102
Aircraft
Stanford Cars
Original
37.33 3.12
16.10 0.19
Original
8.55 + 0.19
26.02 + 0.27
15.45 + 0.02
USEFUL
34.16 = 2.47
14.93 + 0.07
DiffuseMix
8.92 = 0.15
25.65 = 0.20
15.19 + 0.06
TADA
27.40 = 1.99
14.57 + 0.10
TADA+DiMM
8.08 1 0.16
25.12 0.35
14.96 0.07
CIFAR1O
CIFAR100
CIFAR10
Syn-rard Syn-all TADA
4.6
Tent Enror 184
4.4
20.5
3.5
20.0
19.5
3.0
3.8
3.6
SAM 19.0
2.5
SGD
Figure 5: (left & middle) Comparison between different synthetic image augmentation strategies when training ResNer18 on CIFAR 10 and CIFARION 100. For Syn-rand and TADA, we use k resulting in only 30% and 40% additional examples compared to 100% of Syn-all. (right) TADA with = 5 can be stacked with TrivialAugment (TA) to further boosts the performance when training ResNet18 CIFARIO, achieving (to our knowledge) SOTA test classification error.
state-of-the art architectures such as ConvNeXt (Liu al. 2022) and Swin Transformer (Liu et al.) 2021). TADA still outperforms the baselines by a substantial margin, as reported in Table I These findings confirm the effectiveness of our approach across different model structures.
Transfer learning and stacking with other synthetic image generations. We evaluate TADA in transfer learning setting, where we fine-tune ResNet18 pre-trained on ImageNet- 1K on 3 popular fine-graines image classification datasets including Flowers-100 Aircraft, and Stanford Cars datasets. Table compares TADA with DiffuseMix (Islam et al. (024a) which is SOTA data augmentation method. It enhances diversity of synthetic images by blending partial natural images with images generated via InstructPix2Pix (Brooks al. (023) diffusion model. Applying TADA to augment slow-learnable images with DiffuseMix significantly outperform no-augmentation and augmenting entire dataset with DiffuseMix.
Do We Need All the Synthetic Data? To answer this question, compare synthetically augmenting all examples (Syn-all)- which doubles the training set size-with TADA with = 2-which results in increase of approximately 30% and 40% of the total training data-in CIFAR 10 and CIFAR 100. Notably, the generation time for TA DA is reduced to 0.3x and 0.4x that of Syn-all, making it more efficient. In addition, we consider baseline (Syn-rand) where random images are augmented with their corresponding synthetic ones. Figure shows that TADA has much lower test classification error compared to Syn-rand (same cost) and Syn-all (higher cost). This highlights the effectiveness of our targeted data augmentation.
5.2 ABLATION STUDIES
Application to non-diffusion augmentation. Beyond diffusion- based augmentation, our targeted augmentation generalizes to traditional data augmentation. We evaluate this by applying TrivialAug- ment (TA) (Müller& & Hutter 202I) to augment upsampled data. As summarized in Table targeted augmentation consistently outperforms full-data augmentation.

---

Published as a conference paper at ICLR 2026
5.
Orgnal
5.0
5.0
Render
4.8
0 4.3
0
4.3
4.7
USEFUL
TABA
Orggal
4.75
4.5
Real
4.6
5GD
SAM
Or gig (SAM)
Or gin (550)
4.5
4.5
43-
4.00
4.
3.75
10
100
Uosamoling fector
SGD
Dano stp
Figure 6: Training R on CIFARIO. (left) The effect of amplification factor k on test error for upsampling vs generation. Red points indicate the optimal choice of k. k boosts generation. (middle) Generating synthetic CIFARIO 10 images from real images outperform start- ing from random noise. (right) Effect of the number of denoising steps on the performance with k 2.
hurts upsampling but
TADA stacked with strong augmentation. Figure right shows that TADA stacked with Triv- ialAugment achieves state-of- ofee-----th-tttttt results when training ResNet18 on CIFARIO. Appendix F2 shows similar results for CIFAR 100 and Imag-em
Identifying slow-learnable features. Table 8 in Appendix F.2 shows that identifying slow-learnable features by clustering model outputs outperforms selection based on high-loss or misclassification.
For larger k upsampling hurts but generation helps. Figure 6a illustrates the performance of TADA and USEFUL on CIFARIO 10 when varying the upsampling factors. Upsampling achieves the best performance at k 2 due to overfitting noise at larger k. But using synthetic images, TADA benefits from larger values of k. yielding the best performance at k = 5 for CIFARIO and CIFARI00. and k = 4 for Tiny-ImageNe Detailed results for CIFAR 100 and Tiny-Imagen ImageNet can be found in Table 6 in Appendix F.2 This corroborates our theoretical findings in Section 4.2
Choices of initialization for denoising, We compare generating synthetic images by adding noise and denoising real images with denoising from random noises. Figure 6bb illustrates that real data guidance is necessary for targeted synthetic image augmentation. When using random noises to generate synthetic images, performance of & 8 on the augmented training datasets is even worse than that on the original set, generated images do not effectively amplify slow-learnable features.
Number of denoising steps. Figure figure/ocddemosstiee demonstrates the effect of the number of denoising steps on the performance of TADA. Using fewer steps generates images that are too close to real images, am- plifying the noise in the real images. In contrast, using too many steps results in too much noise, which also harms the performance. Overall, using 50 steps yields the best results for both SGD and SAM.
Convergence. Figure g in Appendix F22 shows the mini-batch gradient variance when training ResNet18 the augmented CIFARIO dataset using upsampling and generation. Generation results in lower gradient variance compared to upsampling for different k. confirming our results in Sec. 4.3
Object detection experiments. On ASCCOCCO object detection (Table T0 TADA consistently improves both AP50 and mAP50 - 95 when training YOLOv5m from scratch. It outperforms In- stance-Augmentation and all other baselines while using 25% fewer augmented images. These results demonstrate that TADA extends beyond classification and remains effective for dense prediction tasks.
CONCLUSION
Diffusion-based synthetic data augmentation improves generalization but often enlarges datasets by 10-30x, incurring high computational cost and limited diversity control. We proposed TADA (TArgeted Diffusion Augmentation), principled framework that selectively augments examples not learned early in training using faithful synthetic images with diverse noise. We showed that augmenting only this targeted subset consistently outperforms full augmentation, and our theoretical analysis explains the gains through more uniform feature learning without noise amplification. Across architectures (ResNet, ViT. ConvNeXt, Swin), datasets (CIFAR- 1EFAR-10/100, 10/100, TinyImageNet, ImageNet), and optimizers (SGD, SAM), augmenting just 30-40% of the data yields up to 2.8% improvement. Notably, TADA with SGD surpasses SAM on CIFAR- 100 and Imagee. Results on object detection further demonstrate its effectiveness beyond classification.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research was supported in part by in part by the NSF CAREER Award 2146492, NSF- AI Institute for Cosmic Origins (CosmicAI), and NSF AI Institute for Foundations of Machine Learning (IFML).
REFERENCES
Zeyuan Allen Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXhv:2012.09816, 2020.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-awan mini- mization. In International Conference on Machine Learning, pp. 639-668. PMLR, 2022.
Shekoofch Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic Data from Diffusion Models Improves ImageNet Classification, April 2023. URL http://arxiv. org/ absa 2304 08466 arXiv:2304.08466 [cs].
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.02503, 2023.
Peter L Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness-aware minimiza- tion: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research, 24(316):--36, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18392-18402, 2023.
Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convo- lutional neural networks. Advances in neural information processing systems, 35:25237-25250. 2022.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22405-22418, Curran Asso- ciates, Inc., 2021. URL uttpattpsss/ppppeeiiss.ooooopppp paaper files/paper pdf
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. Advances in neural information processing systems, 35: 23049-23062, 2022.
Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsich, and Quanquan Gu. Why does sharpness-awas aware minimization generalize better than sgd? arXiv preprint arXiv:2310.07269 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Yihe Deng. Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. arXiv preprint arX1v:2306.049949, 2023.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020..
11

---

Published as a conference paper at ICLR 2026
Yunxiang Fu, Chaoqi Chen, Yu Qiao, and Yizhou Yu. DreamDA: Generative Data Augmenta- tion with Diffusion Models, March 2024. URL arXiv:2403. 12803 [es].
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341-23) 1-2368., 2013.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(41),139-14,, 2020.
Chengeheng Guo, Bo Zhao. and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, February 2023. URL arXiv:2210.0755 [cs].
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 8849, 2021.
Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song. and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16783-16792, 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6840-6851 6851. Curran Associates, Inc.,
2020. URL ltttps: http:////..rroccc.o..........ses Eile e hlle://cbbccfec3333880000611/112
nwwr 9ca4bbbcbbbbb Paper .ddf
Khawar Islam and NAVEED AKHTAR. Context-guided responsible data augmentation with diffusion models. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation
Models, 2025.
Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label-preserving data augmentation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 27621-27630, 2024a.
Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, Karthik Nandakumar, and Naveed Akhtar. Genmix: effective data augmentation with generative diffusion model image editing. arXiv preprint arXiv:2412.02366 2024b.
Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, pp. 9965-10040. PMLR, 2022.
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Proceedings on, pp. 51-65. PMLR, 2023.
Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference Machine Learning, pp. 17615- 17659. PMLR, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- mne-grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pp. 554-5611 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
12

---

Published as a conference paper at ICLR 2026
Orest Kupyn and Christian Rupprecht. Dataset enhancement with instance-level augmentations. In European Conference on Computer Vision, pp.
Springer, 2024.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. httpp:://githii
com wttp:/oooocaaaaaaaeea http://ifffffweeeee ffev/ 2022. commit 4dd291a.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer viston, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1 1986, 2022.
Lorenzo Luzi, Paul M Mayer, Josue Cusco-Rodrigee Ali Siahkoohi, and Richard G Baraniuk.
Boomerang: Local sampling on image manifolds using diffusion models. arXiv preprint 1xXiv:2210.1//000 2022.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine- Fine-grained visual classification of aircraft. Technical report, 2013.
Eyal Michaeli and Ohad Fried. Advancing fine-grained classification by structure and subject preserving augmentation. Advances in Neural Information Processing Systems, 37:22316-2222 2024.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmenta- tion. In Proceedings of the IEEE//VF International Conference Computer Vision (ICCV), pp. 774-782, October 2021.
Dang Nguyen, Paymon Haddad, Eric Gan, and Baharan Mirzasoleiman. Changing the training data distribution reduce simplicity bias improves in-distribution generalization. Advances in Neural Information Processing Systems, 37:68854-68896, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10744 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162 8171. PMLR, 18-24 Jul 2021. URL httt::ts::/// http://oooeeeeeigeoee wtt.aig................/...../.. wttpp//s///./////ooooolll... html
Maria- Ereia- Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision. graphics & image processing, pp. 722-729. IEEE, 2008
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bus E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL ht http:///ooceeeeeeneeee n wwtpppppp.ppppppp..eer files/paper 201 9 e hbwbcaaaaaaaccao 2 88 wtw.ee7ff222ffff722
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10685-10695, 2022.
13

---

Published as a conference paper at ICLR 2026
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265-2265, Lille, France, 07-09 Jul 2015. PMIR. URL https:// proceedings m prwss/v3/ v31/soh1 www.kccsiiinnn..mm html
Jacob Mitchell Springer, Vaishnavh Nagarajan, and Aditi Raghunathan. Sharpness- minimiza- tion enhances feature quality via balanced learning. In The Tweifth International Conference on Learning Representations, 2024.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective Data Augmen- tation With Diffusion Models, May 2023. URL arXiv:2302.07944 [cs].
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3769-3778, June 2023.
Yanghao Wang and Long Chen. Inversion circle interpolation: Diffusion-based image augmentation for data-scarce classification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25560-25569, 2025.
Zhicai Wang, Longhui Wei, Tan Wang, Heyu Chen, Yanbin Hao, Xiang Wang, Xiangnan He, and Qi Tian. Enhance image classification via inter-class image mixup with diffusion model. In Proceedings of the IECE/CVF conference on computer vision and pattern recognition, pp. 17223- 17233, 2024.
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness? arXiv preprint arXiv:2211.05729, 2022.
Ling Yang. Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Comput. Surv, 56(4), November 2023. ISSN 0360-0300. doi: 10.1145/3626235. URL https:
100 11455 /3662255
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5-8-5588 October 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710,0949 2017.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-3847, 2023.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. $156-8165. June 2021.
Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training on thin air: Improve image classification with generated data, arXiv preprint arXiv:2305. 203.1, 2023.
14

---

Published as a conference paper at ICLR 2026
A FORMAL PROOFS
A.1 FULL GRADIENT FORMULAS & USEFUL. LEMMAS
From the setting, at iteration i, we can take the derivative with respect to the j-th filter un in the following three training schemes:
1. GD training on the augmented dataset via upsampling: Lemmas
A.2
2. GD training on the augmented dataset via synthetic generation: Lemmas A.3 A.4
3. SAM and GD training on the original dataset: Lemmas A.5
A.6
Recall that & is the augmenting factor, and the augmented datasets DG Dy now have size Nnew
aN + k[1 - o.N.N
Lemma A.1. (Upsampling: full gradient) In the augmented dataset Dy after upsampling the "slow- learnable" subset with factor k. for and j [J], the gradient of the loss L (W()) with respect to w is
aN
DU 20 L(W())
3
Σ
(1) (e0) (82(00)) 100
+ ve(wo) ,E:)
Nnww
N
3k
kil)))
(88(10)) va)" + tiluo .:: E:))
Nuew
t=
where ( sigmoid(- W for the two-layer CNN model f.
Proof. We compute the gradient directly from the loss function:
VU ((W(())
1
exp(- W())
Vif( W()) us V V/,, (z) (2); W())
10f0
Neew
exp(
N
3
Nnew
U1
p=1
3
(12 (1) va) (llof//..)))))
Nnew
Nae
(()) (zg(1o5) v2) + (u -.,)
N Neew
i=aN+1
3
(1)
(2 (t)
+ vi(ws") E.)
N Naew
3k
Σ
(00)) (2)(tug) v2 + uu (, ..,))
New
1=o++1
where the last step follows from the fact that there are k copies of the same subset in the upsampled portion of the data.
Lemma A.2. (Upsampling: feature noise gradients) In the same setting, VU L(W(O) learns the features and noises as follows:
w
1. Fast-learnable feature gradient:
3,33 aN
VVU we ((W((), ve) = Nnew
Σ (1) (Ww v2)2
t=1
15

---

Published as a conference paper at ICLR 2026
2. Slow-learnable feature gradient:
(DU :
((W(())) va)
3k:5
1 ( va)
New
ioN+11
3. Noise gradient:
(a). For En i = 1,... ,NN
aN
V w 10) ((W())))))
3
Nnew
(b). For Es. i aN +
N
(V w (n
((W()) Ei) =
3k Nuew
(/ ye (w))
The proof follows directly from taking the inner product with the gradient formula in Lemma A.1 Then we proceed using that Ve, Ud, &, form a orthogonal set and that summations involving noise cross-terms become negligible.
Proof.
Lemma A.3. (Generation: full gradient) In the augmented dataset De after synthetically generating the "slow-learnable" subset with a factor k, for and j E [J], the gradient of the loss L (W with respect to w is
oN
FU ((W((t))
3
(2)( (1) v + v E.)
w(th
N
3
Σ
(10) (88(0))) va) (1 EE,)
New iaN+1
N
Σ
(1) (8 va)2 + v: (0) xx)27))
N
N=++1
where ( sigmoid( /y: (ar;; W())) for the two-layer CNN model f.
Proof. The proof similar to that of Lemma A.I except that for data that contain Va, the first (1-ar (1-a)N data points come from the original dataset, and the rest comes from synthetic generation with noise 7,, = N 1,...,,., Nnew.
Lemma A.4. (Generation: feature & noise gradients) In the same setting, VG ((W()) learns the w0
features and noises as follows:
1. Fast-learnable feature gradient:
3 aN
(VG in (n L(W(()), ve)
Σ 100 ww (1) v2)2
Nnew
2. Slow-learnable feature gradient:
3 Nnew imo++1
New
Q 10 ((W(1)), va)
(1) (w) va)2
3. Noise gradient: Let (or (E U (xe) (ri denote the set of noises in De (which can be the original or generation noise). Then for any &i E [Nnew]
G ((W(")), wit
3
())
(1) 21) (2.)2 110.11
Nuww
16

---

Published as a conference paper at ICLR 2026
Proof. Similar Lemma A.2 We directly take the inner product and assume that all the noises dispersed such that summations involving their tross-terms become insignificant in high dimension. Recall we also assume that the generation noise is orthogonal features and independent of & n
Following the same process, we have the following gradients for SAM and GD on the original dataset. Lemma A.5. (Original dataset: SAM & GD gradients) In the original dataset D, for and (t)
j E [J] of GD, the gradient of the loss ((W(!)) with respect to w is
3 oN
(8)
(t) vee + vi E.)
N
E=1
3
Σ
ua) Da + (uo E.) , E.).
N
f=o++1
where ( sigmoid( /: (a;;; )) for the two-layer CNN model f.
Suppose we train with SAM. Then the perturbed gradient has the same expression except that replace w. (1) with the perturbed weights w and replace r( with 1 sigmoid(-y: (z;; ( cl))):
V L(W() + (t) V to ((1) (P(C(( V(C(W((1))
3
3
(t)
(82 (w vo) 2o ye (.
N
3
Σ
(e (2)(10 va) va + ui (w E.)P
1=o+11
Lemma A.6. (Original dataset: SAM & GD feature & noise gradients) In the same setting,
with ((W()) learns the features and noises as follows:
1. Fast-learnable feature gradient:
3,33 aN N
<V (o L(W
(1)
(1) (w
v2)
2. Slow-learnable feature gradient:
3
C ((W())) va))
Σ
(2) va)
N
i=a++1
3. Noise gradient:
3
w0 ((
The SAM feature & noise gradients are similar except that we replace w 100 with (t) (())
j.e f,e
Remark. Suppose the data point æ, has feature va and noise d.. We have that
1 sigmoid
(2 (1) va) vi sb.)3
17

---

Published as a conference paper at ICLR 2026
and 6 sigmoid Σ
(20 J.e va)² vilee $1)3
The same formula holds if the feature is Ve.
Similar to Nguyen al. (2024), assume that the logit terms controlled in some to reduce the nonlinearity. More formally, for Theorem 4.1 we will use the approximation 2.t) n! = ( for any early iteration t. At high level, this follows from the small perturbation and the fact that the weights have not "significantly" learned any noises (despite some overfitting) in the early phase.
sigmoid Σ
K va) - viwu .1)3 = sigmoid
Ba (un jee: va)
yi (1 jer 1113
Hence, this is implicitly assuming a small p such that w/" and w are not too different, and large (f) number of neurons J such that the noise term in the sigmoid Σj=1 Vi (e j.er Φ,)² is 'more 'diluted" (close to 0).
Insights from gradient formulas. By directly computing these relevant gradients,
see that
some observed phenomena already become self-explanatory in the formulas. For instance, Lemma A.2 implies that for upsampling, the gradient alignment is k times larger for the noises we replicated. At a high level, this would cause overfitting to these particular noises when performing gradient update. In the next section, we formalize the intuition by examining the underlying mechanism of noise learning in greater detail.
B GD VS. SAM NOISE LEARNING: PROOF OF THEOREM 4.1
Remark, Heuristically, we say that the noise being learned well if the magnitude of the alignment ( ..)I ( is large, meaning that the weights align misalign with the noise to great extent. We say that the model is learning the noise if (w0, (2) E.)l increases over time. In practice, overly fitting or overly avoiding certain noises are both harmful to model's generalization.
Lemma B.1. (Gradient norm bound) In setting, can bound the norm of the gradient matrix VL(W(()) as:
3 (1) (w N
VVL((())
Vi.
Proof. This norm can be lower bounded by the the norm of one column:
(VL(W(0))lr
L(W())]
an
((()))]
for some neuron j
9.36
(w
24)4
a.N
(8)
2
9836
N2 (w v2)a
Σ
N
2
+
N
10
(u
EE)
N2
N
i=oN+1
by taking the norm of the gradient in Lemma A.5 and using orthogonality
3
Vi (4)
&
3
10 wu E.
3 (w))
(0)
10
Vi
N
since terms become negligible
Lemma B.2. (Noise set equivalences) With sufficiently small SAM perturbation parameter at some early iteration f, the following set relations hold:
1 -
T
(()) -,,_-
18

---

Published as a conference paper at ICLR 2026
Proof. We first note that for some early training weights W we have the following connection
between the original weights and the SAM perturbed weights:
(tv j.ct Ei)
(un)) (t)
+ p()) "
((W(()) E.)
(w)) (1) { + p 10 (V
((W)))
3p10)
(wr (t) Et
,E,)
by Lemma A.6
(4)
N
We consider sufficiently small p such that:
(wo) 19 .EiI |&lll
& D )
< p <
min
(5)
We recall that p()) VL(W (1)) Then suppose & E -,,2 that is ys and (wo Ei) have opposite signs. Eq. implies
3p(0)
(w) J.e E:) = (w0)) ( Ei)
(un
N
3p(t)
= =20 Ei) 1+
(0)
1 &i)
N
>0
which implies that sgn sgn((w)
j.6t {i) sgn( (1) Ei)) and & E (1)
Now suppose & E (t) 3.4 Eq. becomes:
f10 Ex) (wo Ei) (1 1 (p(t)
(1)
110 Ui (u0 (0) lE)11
N
kp(t)
= (20000 Ei)
|e (uu E.)
N
(0, shown below
Note that Lemma B.1
gives
1 >>1
3p
|1:
1
(w))
(2)
.t)i 1
N [|V[(W(0)] 11 (2. .iii
3
N
19 (w0)))))) ..)
2 1
3 (0) (w0) (
pll&
1
(m0 E.)I
> 1 ((w0)) ..iI
||6
by the choice of p
||&i||
(w
&. .
(6)
Hence, we have that sun all the noises, the lemma statement follows.
(1) S. {.))
..)) and & E I Since 7 and 7 cover
B. PROOF THEOREM
We then start with item (1) of the theorem. First, for GD, each noise update is computed via:
(w (f+1)
ax) (w Ex) VV m
(4)
((w(1)) Ex)
(w0)) {.) +
3n (1)
(un
(1)
by Lemma A.6
N
3
00
ye |E: (10
1 + " N
19

---

Published as a conference paper at ICLR 2026
Then by definition, for all the noises & -//-- yi and (uo)² (8) E.) have opposite signs, the above equation becomes:
( E.) = (w)) E.) (1-7
3 10th 15/ .511
Without loss of generality, we consider the case when (wo) E.) > 0. We then define the correspond- ing sequence at = {un}) E:) generated by the update:
a++1 = az((1 n,C,,,500 a.), where C. = N |
Next, we want to show that given a proper 7) this sequence is monotonic towards 0, meaning that these noises are gradually "unlearned" by the weights. We provide an inductive step below, which can be easily generalized. If the learning rate satisfies
0 0<n < C,a+ Ci Ee:
then by the update, the following inequality holds:
(1-2__011
1 1
-.,l, at
ar (1-1(()) 11,1) < ae+1 < az
az
Ciaz
Consequently, a++1 d₁ implies that
a,42 < 0++1 by the same argument.
Cia Cratti
similar argument holds for (w) (1) Ex) < Hence, we take sufficiently small
( ,.)) E T/02
min
the weights' alignment with these noises will be monotonic throughout the updates and get closer and closer to The proof for SAM similar (we replace (uo})) ( .E)) by (e E.) for & E I etc.).
Now we proceed with item (2).
Then consider & I/1 J,e,+ in the setting of Lemma B.2 Eq. now becomes:
(t)
Ex) (wf+)))
(++1)
= (w イ Ex)
(t)
1 + r7
3 (00 ye L& (2))) &xi
= (w)) (t) ..)
3 1 1 + 17 N
& (20 (1) &ii = (20 (++1) . > (wy) &.)l.
(8)
And similar expression holds for SAM. Consequently, the alignments will be monotonically away from O, continually being learned through iterations. Hence, we want to compare how the gradients align with noises: (V L(W(() + e e Exi vs. (V ((W()) G. that
By Lemma A.6
we have
For GD:
KV ((W (( ), N
3
((0) (w
For SAM: Z( E.))
N j.
3
(1)
(1)
(.) 11:12
(
3 (t)
(t) ..:) (10)))
3p(()
(t)
(t) (1:11)))
2 11E:
N iic
N
3 (t) (t) 4:) (1 ()(t)
(/ yı (w
(0 A.)1?
|E:11
N
N
3
(w
(1) E (1-
()
( 10 (w
10 .c.i
||,
N
N
20

---

Published as a conference paper at ICLR 2026
where (i) follows from Eq. 4 and (ii) uses the approximation for the logits in early training (2!) = (1.2))
Now with a sufficiently small p that satisfies Equation 3 the bound in 6 yields
(V w. ((W()(((+))))e.cccccc
3 10 (w)) EE) ? N
1 N
(p(*) 4
PllE (1) (.))))
1|&.11
e (0.1)
3
10 (w)
E.).
N
Since this inequality holds for all &; I I1 we compute the average over these noises and have that:
NoiseAlign (1( J.e,
w. j.e) NoiseAlign [I w/ Noise (I ())).
The special case of the theorem can be proved by directly setting the early training weights be the initialization W Since W(O) ~ (0, og) and & ~ (O, dd/ with a large N, and T J.-
will each contain roughly half of the noises, as sgn((w)) .Ei)) = sgn(y:) has probability 0.5 in this
Remark. The theory matches our insights that the noises tend to be fitted in general for the upsampled dataset. We do note that the bound in Lemma B.I can be loose, as it covers very extreme cases, and the actual choice of could be much larger in general without breaking the logic of our argument.
C GENERATION VS. UPSAMPLING: BIAS PERSPECTIVE AND PROOF OF THEOREM 4.2
Item (1) of this theorem can be proved in a similar fashion as for Theorem in Appendix B
For item (2), the update rules follow directly from Lemmas A.2 A.4 Again these formulas themselves
can partially explain what happens: the "generation" gradient learns every noise in the same manner,
but the "upsampling" gradient learns the replicated noises times more in one iteration.
Lemma C.1. (Static noise sets) In our setting, with a sufficiently small learning rate nT, for all j [J]
and all the early iterations t O,...,T, have that:
G,(0) - ,, = =.. G,(t)
,,+
=..0=
G.(T)
I - 4+ 3.
I1/110) 3.+
-,,,t
The same holds for the corresponding sets I..10) and T.,(0)
Proof. This directly follows from the previous proofs, where starting from the intializations, small learning rate ensures that certain noise alignments move towards and others move away from 0 early in training. Hence, in this process, the signs do not change, and these sets always contain the same noises as t progresses
We now focus on the special case of the theorem and show that in expectation, our requirement on the generation noise prevents NoiseAllgnnZ,..t (Z
(() ( from getting too large.
Lemma C implies the sets of noises to which we overfit or do not overfit remain unchanged during
early training. Hence, we eliminate the possibility that noises move from one to another, which simplifies the big picture. The technique is similar to what we did in Theorem 4. and we show that this also holds true for data augmented via generation.
For Theorem the expectation is taken with respect to the underlying data (and noise) distributions. (t())
We first start with computing NoiseAliggn 7. w v Expanding using the definition of NoiseAlign. we have that this quantity equals:
21

---

Published as a conference paper at ICLR 2026
N
New
E
w ((W
+
< Wenno ((w
I
1=1,
i=
7, 5.
= E
V
((W
Φ1)
Φ E I [0(00
L(W
Ei)
+ (1 p) E
[| (V Vn ((W())))))))))))))))))))))))))
m/) v/ [0
= P
m0 ((W ).
+ +11 (1- p) p) E
<V 30 ((W(())
(10)
where p N Neww a+)( measures the probability that the noise belongs to the original data
instead of the synthetic data. The last equality is due to the fact that Φ; E 5,t and follow the same noise distribution D or Dyy
GG((1) are generated i.i.d
Recall the following assumption:
E ( (6)
l|>ll
(k 1)
(e(t) (w) (1) ||EI|]
E
N 4 ( (t)
3
7) 11>11?
(k 1)
(0) 11
V L(W()) >>
< (k: 1)
| (W ).
{ |]
7 ~ Dy: & E~D D
where the last inequality follows the expression of noise gradient. Again, note that I contains synthetic noises that follow the distribution D,, since the synthetic data points are generated indepen- dently. However, the noises in _U((1) are not independent due upsampling and therefore do not follow the noise distribution in D.
To tackle this issue, use the gradient alignment bound above and continue with Equation T0
< <p
L(W
),
5)
+ (1 - pk - 1) | (V 3 ((wr()), (0)) E
+ k(1 a)
1
E
10
((W (0)) 3
-12 - a)
a a k(1
a:)
E
L( (()))) t)
a
a))
)
(A2- - 1)(1 a)
E
V
E
+ k(1 a)
a k(1 a)
k²(1 a))
E
E))
E
+ k<1 a)
133
a k(1 a)
aN
E
((W())
+ Σ
( 10
h
((W()))
{i:
U.00
i=a++1
NoiseAlign (I w (t)
E
The last equality follows from the distribution of noises in the upsampled dataset: the slow-learnable (1-a)) N data points are copied k times, and from the gradient computation (first part of this theorem), each is learned times more, giving rise to the two k's. The noises in each summation term follow D. as they are independent within the subset of summands.
Hence, we have shown the desired result:
E Noise (Z000) 5, ,,,",,,)))) [0)
< E NoiseAlign( (.(1) 2,+ , 20 (0)
22

---

Published as a conference paper at ICLR 2026
At a high level, this suggests that starting at the weight w long the notion of small synthetic noise is satisfied, the expected gradient alignment with noises in the set where overfitting occurs is strictly lower when the dataset augmented through generation. This reduced expected NoiseAlign leads to gradient updates that place less emphasis on fitting noise, thereby enabling the model to focus more effectively on learning the true features.
Consequently, although both upsampling and generation promote more uniform learning, their distinct interactions with noise critically influence generalization performance. This highlights the necessity of incorporating synthetic data and motivates our method.
D GENERATION vs. UPSAMPLING: VARIANCE PERSPECTIVE AND PROOF OF THEOREM
For generation, we have a fully independent dataset, so given the per-g per-gradient variance bound o the variance of the mini-batch gradient 9G can be computed as:
B
B
E [1 - و -
E
Σ 9:
R
91
Bgl*
B
B
1=1
1=1
1
(
9) 11?
B
1
Σ
[l|g: - gll
B
i=1
1
Bo²
B²
a²
(11)
B'
where we can directly take the summation of expectation due to independence among the data. For the upsampled dataset, we instead have two parts:
Fully independent part: The first a.N data points that contain the fast-learnable feature are independent from each other and the rest.
2. Replicated part: The rest of the k(1 a data points that contain duplicates. This part introduces the risk that the same data points might be selected into the same batch.
With this motivation, we let B B1 + B2, where B₁ denotes the number of data points selected from the fully independent part, and Bg denotes the number selected from the replicated part.
For theoretical simplicity, consider stratified mini-batch gradient, where B1 and Bg are fixed numbers in proportion to the sizes of the two parts. We assume that B1, B2 are both integers such that they follow the proportion:
B = B NN
O
aN + k( a)N = B
(12)
a k(1 a) A(1 (1-a)
4(1 N)
B2 = B oN + k(1 a)N = a x(1
(13)
Note that this removes one source of variance since Bi and B2 should themselves random variables in the actual gradient. Hence, the actual variance should be larger than that of the stratified version.
The variance of the stratefied mini-bater gradient ğu after upsampling can be split as:
E [llgr= - -
Σ gi iEB
1
Bgl
1 B
Σ sEB,
(g: 9) 1P
E
Σ B,
(g:
9)
112
(14)
23

---

Published as a conference paper at ICLR 2026
due to independence and unbiasedness. Here with slight abuse of notation, E B, B1, B2 represents we have for the independent part,
the indices data in each portion. Similar to Equation I1]
o²
Σ ie B,
g
9)
11
BI
a
(15)
B
B
B
a k(1
For the replicated part, since we may select copied data points, we first rewrite the summation as:
(1-a)N
Σ 9: iEB2
Ygg.
where we index over all the unique data points E B2 and introduce the random variable Y; 2 representing the number of copies of each unique point, subject to D Y; 12.
Clearly, as we are selecting B3 data points from a total of k(1 - a points ((1 - a unique ones each with & copies). Y; follows hypergeometric distriution. From classical probability theory, we have:
1. Mean:
k
k(1 a)
k
k oN + k(1 - N
B[y] B2 1:1 a)N
a + k(1 - a) k(1 a)N
2. Variance:
k(1 - a - k k(1 - a B2
Var(Y, E(]
k(1 a)N
- a
1 k(1 - a)N a+t(II-a
k(1-a)
E|Y]
(1- a)N
k(1 - a -1
a)N 1 k(1 - ()N k(1 a)N B)
E[]
(1 a)N (k(1 a)N 1)(a + k(1 a1)
a)N
k(aN + k(1 N)N B)
E
N (k(1- - a)N - 1)(a + k(1 - a))
k(1 a)N - aN + k(1 - a)N B
E|
k(1 a)N - aN k(1-a)N
a/ + A(1 N)N B
EY]
a. + k(1 o)N
k
B
B
oN k( a)N (1-
a. k(1 a)N
3. Second Moment:
E|y,3 Var(Y;) + E(y
aN k(1 a)N + +(B B
B
k
< B
1
aN k(1 a)N
aN + k(1 o:
k
B
Bk
B
aN k(1 a
aN k-1 a)N
aN + +11 a)N
4
(k-1)B
B
1+ aN k
(16)
oN
ks[ N)N
24

---

Published as a conference paper at ICLR 2026
With these quantities,
compute:
N1
(1-a)N
1
E
Σ
(g.
9)
1
E
|1
Yi 9)
since
Σ i=1
Y; B2
B
B
R2
i=1
(1
(1-a).
1
E
9. - 9 Y?
B
0²
(1-a)N
E
Y
B
d²
1:(1
a)N
(1+
(k: 1)B
by Equation 16
B aN + k(1 a)N
aN + -
(17)
We plug in Equations [5
into Equation
to obtain:
o²
B k(1 aa) + o² k(1 - ())
(1+ (k-
E
01:0
-
B a k(1-a)
aN + + a)N
²² k(k - 1)(1 a) B
+
=
B
(a + k(1 )3)² N
In the theorem statement, oZ (k) and (k) to differentiate the two and emphasize the dependence on the augmenting factor k.
E PSEUDOCODE
Algorithm [ illustrates our method. We include Table 3 to compare it with other augmentation methods, including classical ones, naive diffusion based generation, and mix- based methods such as mixup (Zhang et al. (2017)) and CutMix Yun et al. (2019)).
F ADDITIONAL EXPERIMENTS
F.1 ADDITIONAL EXPERIMENTAL SETTINGS
Datasets. The CIFARIO dataset consists of 60,000 32 x 32 color images in 10 classes, with 6000 images per class. The CIFAR 100 dataset is just like the CIFAR 10, except has 100 classes containing 600 images each. For both of these datasets, the training set has 50,000 images (5,000 per class for CIFARIO and 500 per class for CIFAR100) with the test set having 10,000 images. Tiny- Imageee comprises 100,000 images distributed across 200 classes of ImageNet (Deng al., 2009), with each class containing 500 images. These images have been resized to 64x64 dimensions and are in color. The dataset consists of 500 training images, 50 validation images, and 50 test images per class, For transfer learning experiments, we also used three fine-grained classification datasets. Flowers-102 (Nilsback Zisserman 2008) contains 189 images flowers from 102 categories, with between 40 and 258 images per class. Aircraft (Maji et al., 2013) consists of 10.000 images across 100 aircraft model variants, with roughly 100 images per class. Stanford Cars (Krause et al.) 2013) contains 16,15 185 images of 196 classes of cars, with classes defined at the level of make, model, and year.
Training on different datasets. From the setting in (Andriushchenko & Flammarion 2022 we trained Pre-Activation ResNet18 on all datasets for 200 epochs with batch size of 128. We used SGD with the momentum parameter 0.9 and set weight decay to 0.0005. We also fixed 0.1 for SAM all experiments unless explicitly stated. We used linear learning rate schedule starting at 0.1. The learning rate is decayed by a factor of 10 after 50% and 75% epochs, i.c., set the learning rate to 0.01 after 100 epochs and to 0.001 after 150 epochs. For transfer learning experiments, we fine-tuned pre-trained ResNet18 ImageNet-1K KK. On Flowers-102. we trained for 200 epochs with
25

---

Published as a conference paper at ICLR 2026
Table Comparison with other augmentation methods.
Feature
Classical Aug (Flip, Crop, etc.)
Mix-based (mixup/CutMix)
Core Objective
Learn basic invariances
Model regularization
Selection Strategy
Applied randomly to all images
Random pairing of images; aug- ments all samples
Mechanism Computation
Geometric/color transformations Negligible: native GPU opera- tions
Mix patches between real images
Low: simple arithmetic opera- tions
Guarantee (Theory)
No Image classification; transfer learning
Yes
Primary Application
Image classification;
transfer
learning
Key Advantage
Fast, effective baseline
Cheap and effective regularizer
Feature
Diflusion-based
TADA
Core Objective
Increase data quantity and diver- sity
Reduce simplicity bias; promote
balanced feature learning
Selection Strategy
Selects samples learned early in training
Mechanism
Generates synthetic images and mixes them with real images High: generates multiple times for different conditional prompts No
Generates synthetic images close to real images
Computation
Moderate: only generates a sub- set (30-40% of dataset) Yes Image classification; transfer learning
Guarantee (Theory)
Primary Application
Image classification; transfer learning
Key Advantage
Can boost performance at large compute cost
Efficient: high gain from few samples; can be combined with prior methods
an initial learning rate 0.001. On Aircraft and Stanford Cars, we trained for 100 epochs with an initial learning rate of 0.01.
Training with different architectures. We used the same training procedures for Pre- Activaaiit ResNet18, VGG19, and 121 We directly used the official Pytorch (Paszke et al. 2019)
implementation for VGG19 and DenseNet121. For ViT (Yuan et al. 2021). we adopted a Pytorch implementation at htttts://gttttiiie hidden size, the depth, the number of attention heads, and the MLP size are set to 768, 8, 8, and 2304, respectively. We adjusted the patch size 4 to fit the resolution of CIFAR10 10 and set both the initial learning rate and to 0.01. For both ConvNeXt-T and Swin-T, T, we used their official implementations but trained with SGD instead of Adam. The (learning rate, p) to (0.1, 0.05) for ConvNeXt-T and (0.01, 0.001) for Swin-T.
com wwuidraiiniiss wiw.ns//vi-pytorrhh In particular, the
plyperparameters. For USEFUL (Nguyen et al. 2024 we adopt their default hyper-parameters for separating epochs and set the upsampling factor k to 2 as it yields the best performance. For our method, the upsampling factor i set to 5 for CIFAR and CIFARI00 while that of TinlImageNet is set to 4. Table 4 summarizes the upsampling factor along with the denoising steps which the synthetic images are used to augment the base training set. transfer learning experiments, we set k to 2.
Generating synthetic data with diffusion model. We use an open-source text-to-image diffusion model, GLIDE (Nichol et al.) 2021) our baseline for image generation. We directly used the of- While
ficial Pytorch implementation at ht ppt:::/ / hubb com/ boeenai/giide-eettt.. generating, set the time steps 100 and the guidance scale 3.0. For k-way classification, input the class names C = {ci...., C with prompt =' a photo offc; V. For DiffuseMixx, we used their official implementation with single conditional prompt ("Autumn").
26

---

Published as a conference paper at ICLR 2026
Algorithm TADA (TArgeted Diffusion Augmentation)
1: Input: Original dataset D, Model f(, W Separation epoch t, Total epochs T, Time steps t.,
GLIDE model G(ug, Eg)
2: Step I: Initial Training
3: Train the model f(; W()) on dataset D for & epochs
4: Step 2: Clustering and Data Selection Based on Loss
5: for each class E D do
6: {C1. C2} & k- W()) [Cluster data into C1 and C2
7: Step 3: Image Generation from Selected Data
8: Obtain text prompt / [e.g. For class dog, I is photo of dog)
9:
for each data point प. E C2 do Initialize random noise ~ T)
10:
11:
Generate initial noisy image Lt, + vās, xt vI 6 for from t, to 1 do
12:
13:
μ4, Σ t n0(F, 8, 1), (r S. 2- sample from (M,
14:
15:
end for
Tgennentedd 350
17: D D DU (Add the generated image to the dataset)
end for
19: end for
20: Step 4: Retraining the Model
21: Train the model f(, W) on updated dataset D for T epochs
22:
Output: Final model f(,, (TT)
Table 4: Details of the upsampling factor for different methods and datasets.
Method
Dataset
k
Denoising steps
USEFUL
CIFAR10
2
CIFAR
Tiny-ImageNet
TADA
CIFAR10
40, 50, 70, 80
CIFARIO0
40, 50, 70, 80
Tiny-ImageNet
50, 70, 80
Computational resources. We used NVIDIA RTX 3090 GPU for model training and NVIDIA RTX A5000 GPUs for generating.
F.2 ADDITIONAL EXPERIMENTAL RESULTS
State-of-the-art architectures. To further demonstrate the effectiveness of our method, we evaluated it on two widely used modern vision backbones: ConvNeXI-T (Liu et al. (022). a convolutional network that revisits ConvNet design with architectural refinements inspired by transformers, and Swin-T (Liu et al. 202T), a hierarchical vision transformer that introduces shifted windows for efficient and scalable self-attention. Both models were trained on CIFAR- 10 using SGD with k As shown in Table [] our method consistently outperforms both the Original (no augmentation) and USEFUL.. particular, for ConvNeXt-T, our approach achieves a nearly 7% reduction in test error compared to the second-best method, highlighting its strong compatibility with modern architectures.
Qualitative results. Figure 7 presents examples of slow-learnable (top) and fast-learnable (bottom) samples in CIFAR-10 where the slow-learnable samples are specifically targeted in our synthetic data augmentation. Slow-learnable samples are often visually ambiguous: the object may be partially visible, relatively small compared to the background, or blended with clutter, making them harder to recognize. contrast, lestnllae samples are clear and representative of their class, with the object occupying most of the image and little background interference. These characteristies explain
27

---

Published as a conference paper at ICLR 2026
airplane
automabile
bird
cat
deer
dog
Trog
horse
ship
truck
Siee Boarkdddd Sia nab.....
Redl
Synomalic
Synkmaet Neat Potioiraat Foss eoroeeooo
Figure 7: Examples slom-learnalle (row 1) and fast-learnabe (row 3) samples in CIFAR-10. Rows 2 and 4 show the corresponding synthetic images generated with GLIDE (Section 4.4) using 50 denoising steps (FID= = 10.67).
Table 5: Targeted augmentation with non-diftusion and diffusion methods. Test classification error (%) of training ResNet18 CIFARIO. Here, Trivial Augment (TA) is applied only to the upsampled subset (either slow-learnable or full data) before being added back to the original dataset, which differs from Figure S where TA is applied online all training data. Targeted TA augmentation outperforms full-data TA augmentation, and combining targeted TA augmentaiton with diffusion- generated images yields the best performance.
Method
Test Error (%)
Original
5.07
0.04
TA on slow-learnable examples only
4.26
0.02
TA on all examples
4.48
0.08
TA + Diffusion on slow-learnable examples only
3.81
0.10
TA + Diffusion on all examples
4.20
0.13
why slow-learnable samples are acquired later in training, whereas fast-learnable ones are learned quickly. Figure shows that synthetic examples preserve key visual content (features)- as object shape, structure, and spatial arrangement while introducing subtle variations in texture or color (noise). For instance, image of bird in the second row is reproduced with the same layout and pose, but with different color for the neck part. Such nuanced transformations are difficult to achieve with standard augmentations like random cropping flipping, highlighting the value of generative augmentation in our approach.
Application non-diffusion augmentation methods. Our findings on which examples should be augmented generalize beyond diffusion- based augmentation. To verify this, we conducted additional experiments using TrivialAugment (TA) (Müller & Hutter, 2021). widely used non-diffusion augmentation method. Specifically, we either augment with only the slow-learnable examples or with full dataset, apply TA to the augmented copies, and then add them back to the original training set. As shown in Table5] augmenting only dow-learrable examples with TA outperforms applying TA to all examples. Furthermore, combining TA with targeted diffusion- augmentation yields additional improvements and surpasses full-data augmentation with both TA and synthetic data.
We note that this setting differs from the experiments Figure In Figure 5 TA is applied online to both original and augmented data during training. In contrast, in Table 5 TA is applied only the augmented subset (either slow-learnable or full data) before adding it to the original dataset, allowing us isolate the benefit of targeted example selection.
Effect of the upsampling factors. Table 6 illustrates the performance of our method and USE- FUL when varying the upsampling factors. As discussed in Section 4 USEFUL. achieves the best
28

---

Published as a conference paper at ICLR 2026
Table 6: Training ResNet18 on CIFARIO with different upsampling factor (k).
CIFAR10 (USEFULL)
CIFARIO (TADA)
CIFARTO0 (TADA)
TINY-MMAGRNET (TADA)
4.79
4.57
21.13
32.90
3
5.01
4.52
21.27
30.88
4
5.04
4.45
20.26
30.64
5
4.98
4.42
20.00
31.28
Table 7: FID and test classification error of & on CIFAR1O when augmenting with synthetic
images from different denoising steps. The upsampling factor is set to here.
STEPS
FID
ERM
SAM
O(ORGGINLL IMAGES)
5.07
4.01
10
5.36
4.94
4.03
20
6.86
4.85
3.92
30
7.48
4.83
3.79
40
8.52
4.73
3.92
50
10.67
4.56
3.69
60
13.25
4.68
3.72
70
17.46
4.91
3.84
80
24.27
4.83
3.98
34.69
4.87
3.86
100
47.35
4.99
4.28
Table 8: Training ResNNet8 8 on CIFARIO with different data selection strategy. We used SGD and set the upsampling factor i to 5 here.
METHOD
MISCLASSIFICATION
HIGH LOSS
SLOW-LEARNABLE (USEFUL AND TADA)
TEST ERROR
4.87 0.10
4.77 + 0.06
4.42 + 0.04
performance at k 2 due to overfitting noise at larger k. On the other hand, using synthetic images, our method benefits from larger values of yielding the best performance k: 5 for the CIFAR datasets and & = 4 for the Tmag-em dataset. This empirical result reinforces our theoretical findings in Section 4.2
Effeet of the number of denoising steps. Table 7 illustrates the impact of the number of denoising steps on both the FID score and the downstream performance of our method. When initializing the denoising process with real images, increasing the number of steps leads to higher FID score. Unlike prior works in synthetic data augmentation, we observe that minimizing the FID score between real and synthetic images does necessarily correlate with improved performance. Specifically, using fewer steps generates images that are overly similar to real images, potentially amplifying the inherent noise present in the original data. Conversely, too many denoising steps introduce excessive noise, which also degrades performance. Empirically, we find that using 50 denoising steps strikes the best balance, yielding optimal results for both SGD and SAM optimizers.
Alternative selection strategies. Our approach differs from core-set selection methods Guo et al. (002)). which aim to reduce dataset size while matching the performance of training on the full data. Such methods either require training the model (or a smaller proxy) and use its statistics to find the coreset or update the coreset during the training. of our approach is to improve the generalization performance over that of full data, by reducing the simplicity bias of training. We showed that this can be done by augmenting the data with faithful synthetic examples corresponding to the slow-learnable part of the data. Motivated by theory, found such examples by clustering the model output early in training (without training a proxy model updating the subset during training). However, other more heuristic approaches can be also used to find slow- leanneleeelle examples. In this section, we conducted new experiments to find low-learnable examples based on (1) high loss and (2) misclassification, the same checkpoint used in
contrast to coreset selection, the goal
29

---

Published as a conference paper at ICLR 2026
5.5
UPSAMPLE
5.0
UPSAMPLE 4)
Ours (k
4.5
Ours (k = 4)
Grad variance
4.0
3.5
2.5
2.0
1.5
100
110
120
130
140
150
160
170
180
190
Epoch
Figure 8: Variance of mini-batch gradients of ResNet18 with SGD the augmented CIFAR10 100 dataset found by USEFUI. and TADA when varying the upsampling factor k. Our method yields lower gradient variance throughout the entire training process.
experiments in the paper. As shown Table 8 our method outperforms both heuristic-based selection strategies. Corset selection methods can applied on top of synthetically augmented data to further improve data-efficiency. This is an exciting direction we leave for future work.
Generation time. Using 4 GPUs, the generation time for the entire CIFAR-10, CIFAR- 100, and Tiny ImageNet datasets is approximately 12, 12, and 26 hours, respectively. In contrast, our method requires generating only about 30%, 40%, and 60% of the total dataset size, reducing the generation time to just 3.6, 4.8, and 15.6 hours, respectively. This represents substantial improvement efficiency compared to prior approaches to synthetic data generation, which typically require producing 10-30 times more samples than the original dataset size.
gradients. Figure Figure&ppeeentss presents the mini-batch gradient variance during training of ResNer18 8 using SGD on the augmented CIFAR-10 datasets selected by USEFUL and our proposed method. Across all upsampling factors k, our method consistently yields lower gradient variance throughout the entire training process. This indicates that synthetic data provides more stable and consistent learning signal, which can contribute to more effective optimization. Notably, the variance gap between method and USEFUL widens significantly after epochs 100 and 150 150-key points at which the learning rate is decayed. This suggests that our method enables the model to adapt smoothly changes in the learning rate, likely due to the benign noise introduced in the faithful synthetic images. Lower gradient variance is often associated with more stable convergence and improved generalization. highlighting the advantage of our approach.
Variance
Training loss. Figure illustrates that training with synthetic augmentation (TADA) has lower training loss than training with real augmentation (UPSAMPLE).
Different number of clusters. Figure 10 illustrates the performance of our method with varying numbers of clusters (2, 3, and 4). For each number of clusters, we varied the number of augmented clusters, which are selected as those with the highest average losses. The baseline (0%) corresponds to no augmentation, while 100% indicates augmenting all clusters. As shown the figure, augmenting too few or too many examples is suboptimal. For ResNet18 on CIFAR-10, augmenting approximately 30% the dataset yields the best performance.
F.3 EXPERIMENTS ON IMAGENET
To demonstrate the scalability of our method, we apply it to training on ImageNet (Deng et al. 2009), large-scale dataset containing over 1.2M images across 1,000 classes and commonly
30

---

Published as a conference paper at ICLR 2026
USEFUL TADA
12
train loss
0.04
10
0.02
train loss
8
160
170
180
190
200
epochs
6
25
50
75
100
125
150
175
200
epochs
Figure 9: Training loss ResNet18 on the CIFARIO dataset when upsampling the slow-learnable examples augmenting them with synthetic data
Baseline augmentation)
clusters
3 clusters
clusters
5.2
baseline
Bactssters/aauggneeen Paugmented clusters
/
5.1
Test Error
4.9
3c/la
4c/4a
4c/2a
3c/2a
4.7
2c/1a
4.4
12%
24%
30%
100%
Augmented Examples (%)
Figure 10: Test error of ResNet18 trained on CIFAR-10 with different cluster-based augmentation configurations, Each bar represents one experimental setting, color-coded by the number of clusters: blue for clusters, orange for clusters, and green for 4 clusters. Augmented clusters are selected those with the highest average losses. The x-axis shows the percentage of augmented examples relative to the full dataset size. Error bars represent standard deviations across runs.
used a benchmark for evaluating visual recognition models. Due to computational con- straints, we adopt the FFCV library (Leclere et al., 2022) for efficient data loading and train- ing, adapting the implementation from the publicly available repository https:://///ss /qitttie /github com/libf htt.:fffoffoo/eo ima---mmaaaeee We train ResNet18 8 and ResNet50 using the smallest con- figuration provided that codebase (i.e., 16 epochs). For synthetic image generation, we em- ploy Boomerang (Luzi et al. 2022) with Stable Diffusion v1.5 (https: /ttp///gggggg...eeee co/ fioole-diffusi.oom v1 5/ stable di ffffionnn vl 5), following the settings described Boomerang's 's paper. For selecting augmented images, number augmented images about 65% the full dataset.
the checkpoint at epoch 2 and the
Table reports the ImageNet test accuracy when training ResNet18 and ResNet50 with different augmentation strategies. Our method achieves the highest Top-1 and Top-5 accuracies across both architectures, outperforming Boomerang despite using only 65% augmentation, whereas Boomerang
31

---

Published as a conference paper at ICLR 2026
Table Test accuracy training R and ResNet50 on ImageNet.
Method
Augmentation (%)
ResNet18
ResNe150
Top-1 Ace
Top-5 Acc
Top-1 Acc
Top-5 Acc
Original
67.23 + 0.02
87.76 + 0.10
73.24 0.02
91.63 +:2 0.08
USEFUL
68.86 0.20
88.68 1 0.12
74.04 + 0.15
92.06 + 0.08
Boomerang
100
68.95 = 0.04
89.02
+ 0.09
73.72 = 0.06
92.09 + 0.11
TADA+ Boomerang
65
69.28 = 0.06
89.21 + 0.09
74.77 = 0.05
92.62 + 0.07
Table 10: Performance comparison of training YOLOv5m on MS-COCO.
Method
Augmentation (%)
AP
mAP
Original
63.26
44.26
USEFUL
63.87
44.92
Instance-Aggmenttttion
100
64.17
45.75
TADA+ InstanceeAugmeenttton
75
64.94
46.28
requires 100% synthetic augmentation. This demonstrates that our targeted augmentation strategy
not only yields stronger accuracy gains but also does so with substantially lower computational cost.
F.4 OBJECT DETECTION EXPERIMENTS
To further demonstrate the applicability of our method beyond image classification, conducted experiments on object detection. We follow the setup of Augmenttttoooo (Kupyn & Rupprecht 2024), which uses pretrained ControlNet (Zhang al. 2023 based inpainter with prompt engi- neering to repaint individual objects. Following their data augmentation settings (Table (Kupyn) & Rupprecht 2024)). trained YOLOvvm model ht t/ubbb//.uu. com/uomm//oooooo u tr yyt cs yolov5 on the MS-COCO dataset (Lin et al., 20144) from scratch using default hyperparameters (batch size 40, 300 epochs). We used the synthetic images released by the authors of Instance mentation. To identify slow-learnalee images, we used the model checkpoint at epoch 5 and labeled any image containing at least one misclassified object as slow-learaable resulting 75% of the dataset.
We evaluate object detection performance using two standard metrics: AP30 and mAP50- 95- Both metrics rely on the Intersection over Union (IoU), which measures how much a predicted bounding box overlaps with the ground truth. AP5o computes Average Precision at an IoU threshold of 0.50, providing lenient measure of detection accuracy-moded with high AP effectively locate objects even if their boxes are not perfectly aligned. In contrast, mAP50-95 averages AP across IoU thresholds from 0.50 to 0.95 in steps of 0.05, rewarding models that produce both correct detections and tightly aligned bounding boxes. This makes mAP30-95 a more stringent and comprehensive metric for evaluating detection quality.
Table T0 shows that our method also applies effectively to object detection, outperforming all baselines, including InstanceAAummentation while achieving 25% reduction in data usage.
32


Reviews:
[{"review_id": "YcaqTYGATa", "reviewer": "Reviewer_pN7D", "summary": "This paper proposed a training strategy that can smartly combine real data and synthetic data for training an improved classifier. To verify the effectiveness, this paper evaluated the method on image data augmentation for classification across backbones and datasets. This paper also provided some analysis on simple MLP layers. Besides, the method is a plug-and-play module and also evaluated plugged into the DiffuseMix.", "questions": "If you can solve my concerns, the method can be very general, and then I can raise my score to 8.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The dominant strength is that the current data augmentation paper only focuses on how to generate data with high fidelity and diversity for a more robust decision boundary. However, a very small paper focuses on how to balance the real set and the synthetic set during the training process. This paper fills the blank for current generative-based data augmentation research.", "weaknesses": "This method is general, but the evaluations are limited.\n\n1/ The evaluated backbones are too weak, and whether better-pretrained backbones can overlay the benefit of your method.\n\n2/ Since this method is a plug-and-play module, why not evaluate it based on more state-of-the-art methods like [1,2,3,4]? Meanwhile, you should at least discuss them in the related work.\n\n3/ Lack of evaluations on fine-grained datasets.\n\n4/ This method seems like can be applied not only for image classification datasets, how for the augmentations in detection, segmentation even in other modalities like text and videos.\n\n\nReferences\n\n[1] Effective Data Augmentation With Diffusion Models\n\n[2] Enhance image classification via inter-class image mixup with diffusion model\n\n[3] Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification\n\n[4] Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wFyhUdpQzw", "reviewer": "Reviewer_s4Wo", "summary": "This paper proposes a novel data augmentation strategy designed to improve the generalization of image classification models. The core contribution is a method that selectively applies augmentation only to a subset of the training data identified as slow-learning samples. The authors demonstrate that this targeted approach improves classification performance.", "questions": "- What are the fundamental, identifiable differences between the samples classified as \"slow-learning\" versus \"fast-learning\"? If distinct features or patterns characterize these slow-learning samples, could a model be developed to identify them a priori? Such an approach could simplify the overall pipeline by removing the need for an initial training phase solely to identify these samples.\n- Could the proposed augmentation strategy, which focuses on difficult examples, be adapted to benefit tasks outside of classification, such as improving sample quality or diversity in image generation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The central idea of targeting slow-learning samples for augmentation is novel and intuitive. The rationale that focusing augmentation efforts on more challenging examples seems a logical approach to improving model robustness and generalization.\n- The paper provides extensive empirical validation across three different datasets, showing credibility to the proposed method's effectiveness. The observation regarding the characteristics of slow-learned samples is particularly interesting and further discussion on this would make paper more interesting.", "weaknesses": "- The theoretical analysis relies on a simplified two-layer CNN assumption. This raises questions about the direct applicability and relevance of the derived theorems to the deeper, more complex architectures commonly used in practice. The paper would be strengthened by a discussion bridging this theoretical gap.\n- I have concerns regarding the significant computational overhead of the proposed method. Utilizing a diffusion model for data generation, even for a subset of the data, is inherently more expensive than traditional augmentation techniques. Also, the multi-step pipeline may limit its practical adoption.\n- The experiments are confined to relatively small-scale datasets. It is unclear how the method would perform on larger, more complex datasets such as ImageNet. An explanation for the choice of datasets and a discussion on the method's potential scalability would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "naeXvCUfYh", "reviewer": "Reviewer_9nHM", "summary": "This paper proposes a targeted augmentation approach using diffusion models, focusing on \"slow-learnable\" images (identified via early training clustering) by adding noise and denoising to create faithful variations with different noise. A theoretical analysis with a two-layer CNN suggests this promotes uniform feature learning and reduces minibatch variance compared to upsampling. Experiments on CIFAR-10/100 and TinyImageNet show accuracy improvements (up to 2.8%) with ResNet, ViT, and other models, and the method complements optimizers like SGD and SAM.", "questions": "- Could you explain why the atypical activation function $\\sigma(z) = z^3$ was chosen over ReLU, and does the theory hold if ReLU is used instead?\n\n- Please cite, and compare/discuss results in the 'boomerang' paper, as it seems to contradict results in this paper.\n\n-  Can authors please add training-loss curves to support the convergence claim and explore why stacking with SAM works?\n\n- It would be great (but no necessary) to include a simple metric (e.g., feature similarity) to verify \"faithfulness\" of synthetic images.\n\n- Could authors test on a larger dataset and with another diffusion model to broaden applicability?\n\n## Overall \n\nThis is a helpful and promising approach for efficient augmentation, with solid small-scale results. Addressing the prior comparison, clarifying theory-practice links, and expanding experiments could make it even more impactful!", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Efficiency**: Augmenting only 30–40% of data outperforms full-dataset augmentation, offering a practical, resource-aware solution.\n\n**Empirical Support**: Ablation studies on augmentation factors and initialization provide useful insights.\n\n**Compatibility**: Works well with existing methods (e.g., SAM), boosting performance further.", "weaknesses": "**Missing Prior**: The method overlaps with \"Boomerang\" [1], which uses similar noise-add-and-denoise techniques for data augmentation for classification, but it’s not cited or compared. Notably, they use all of the dataset for synthetic data generation, and they see gains in accuracy, which contradict experiments in this paper. \n\n**Theory-Practice Gap**: The claim of mimicking SAM’s feature learning (e.g., sections 4.1–4.2 suggest SAM-like noise suppression and uniform learning) doesn’t fully align with empirical results, where gains add to SAM’s effects (abstract notes up to 2.8% improvement with SAM). This suggests the method might address different aspects of training dynamics than intended, and further analysis could clarify this discrepancy.\n\n**Convergence claim**: The assertion of faster SGD convergence (Theorem 4.3, Corollary 4.4) relies on synthetic noise variance being lower than upsampling variance, but the link to the \"small noise\" assumption (section 4.4) isn’t fully derived or supported with training curves, leaving uncertainty about its practical impact.\n\n**Idealized and unrealistic theory setting**: The model assumes simplified conditions (e.g., P=2 patches, orthogonal features in section 3), which may not capture the complexity of real image data, potentially limiting the theory’s applicability to broader settings. \n\n**Scope limitation**: Experiments are confined to small datasets, and the claim of effectiveness across diffusion models (abstract) lacks support from multiple generators, which could restrict the method’s generalizability and leave its robustness untested.\n\n[1] Luzi L, Mayer PM, Casco-Rodriguez J, Siahkoohi A, Baraniuk R. Boomerang: Local sampling on image manifolds using diffusion models. Transactions on Machine Learning Research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hu2sFwTyAS", "reviewer": "Reviewer_prBk", "summary": "This author studies diffusion-based synthetic data augmentation for image classification. Unlike prior works that augment the entire dataset (often by generating 10×–30× more images), the authors propose targeted augmentation: identify the subset of training samples whose features are “slow-learnable” and generate synthetic variants only for these.", "questions": "1) How sensitive are the results to the specifics of the clustering step? For instance, does choosing a different number of clusters or a different layer for features change which examples are selected as “slow”?\n2) Did you compare clustering versus simply picking top-θ% highest-loss examples (or uncertain examples)? Table 7 suggests clustering works better than high-loss, but can you elaborate on why?\n3) How was the denoising step (e.g. 50 steps) chosen? Would an adaptive schedule (tailored per image) improve results?\n4) Have you considered whether targeted augmentation helps other tasks (e.g. detection) or robustness measures beyond accuracy?\n5) How does performance compare to simply oversampling the slow examples (as in weighted sampling) without generating new images? This would isolate the benefit of synthetic variety.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Focusing diffusion-based augmentation on slow-learnable examples is a fresh take on synthetic augmentation.\n2)  The authors analyze a simplified two-layer CNN to compare SAM vs. SGD (showing SAM learns “noise” more slowly) and prove that generating faithful images accelerates learning of slow features without amplifying noise (Theorems 4.1–4.3).\n3) The experiments are extensive and well-controlled. On CIFAR-10/100 and TinyImageNet, across multiple architectures, the proposed method consistently outperforms baselines (random subset, full augmentation, simple upsampling) and yields up to 2.8% test accuracy\n4) By augmenting only ~30–40% of data, the method substantially reduces synthetic data generation time (e.g. 3.6h vs 12h on CIFAR-10) compared to full-data diffusion.", "weaknesses": "1) All experiments are on relatively small benchmarks (CIFAR-10/100, TinyImageNet). It is unclear if the approach scales to large datasets (e.g. full ImageNet) or real-world settings.\n2) Identifying slow-learnable examples via early clustering of model outputs (or high loss) is somewhat heuristic.\n3) The method relies on a text-conditional diffusion model (GLIDE) with class prompts and uses the real image as guidance. It is not fully clear how much the performance depends on prompt engineering or the specific diffusion backbone.\n4) The 2-layer CNN analysis, while insightful, assumes a very stylized data distribution (two patches, Gaussian noise, cubic activations) and early-training approximations. It is not guaranteed these results carry over to deep architectures and natural images.\n\nMissing relevant references:\n\n1) GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing\n\n2) Context-guided Responsible Data Augmentation with Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)