DiffAug: A Diffuse-and-Denoise Augmentation for Training Robust Classifiers
Authors: Chandramouli Shama Sastry
Sri Harsha Dumpala
Sageev Oore
Venue: NeurIPS 2024 poster (2024)
Keywords: Synthetic Augmentations; Robust Classifiers; Classifier-guided Diffusion; Perceptually Aligned Gradients
Primary area: diffusion_based_models
TL;DR: A simple and efficient diffusion-based augmentation technique to improve classifier robustness.

Abstract: We introduce DiffAug, a simple and efficient diffusion-based augmentation technique to train image classifiers for the crucial yet challenging goal of improved classifier robustness. Applying DiffAug to a given example consists of one forward-diffusion step followed by one reverse-diffusion step. Using both ResNet-50 and Vision Transformer architectures, we comprehensively evaluate classifiers trained with DiffAug and demonstrate the surprising effectiveness of single-step reverse diffusion in improving robustness to covariate shifts, certified adversarial accuracy and out of distribution detection. When we combine DiffAug with other augmentations such as AugMix and DeepAugment we demonstrate further improved robustness. Finally, building on this approach, we also improve classifier-guided diffusion wherein we observe improvements in: (i) classifier-generalization, (ii) gradient quality (i.e., improved perceptual alignment) and (iii) image generation performance. We thus introduce a computationally efficient technique for training with improved robustness that does not require any additional data, and effectively complements existing augmentation approaches.

DiffAug: A Diffuse-and-Denoise Augmentttionn for Training Robust Classifiers
Chandramouli Sastry, Sri Harsha Dumpala, Sageev Oore Dalhousie University, Canada.
Abstract
We introduce DiffAug, a simple and efficient diffusion-based augmentation tech- nique train image classifiers for the crucial yet challenging goal of improved classifier robustness. Applying DiffAug to given example consists of one forward- diffusion step followed by one reverse-diffusion step. Using both ResNet-50 and Vision Transformer architectures, we comprehensively evaluate classifiers trained with DiffAug and demonstrate the surprising effectiveness of single-step reverse diffusion in improving robustness to covariate shifts, certified adversarial accuracy and out of distribution detection. When we combine DiffAug with other aug- mentations such as AugMix and DeepAugment we demonstrate further improved robustness. Finally, building on this approach, we also improve classifier-guided diffusion wherein we observe improvements in: (i) gradient quality (i.e., improved perceptual alignment) and (iii) image generation performance. We thus introduce computationally efficient technique for training with improved robustness that does not require any additional data, and effectively complements existing augmentation approaches.
(ii)
Introduction
Motivated by the success of diffusion models in high-fidelity and photorealistic image generation, generative data augmentation is an emerging application of diffusion models. While attempts to train improved classifiers with synthetic data have proved challenging, Azizi al. [2] impressively demonstrated that extending the training dataset with synthetic images generated using Imagen [42]- - with appropriate sampling parameters (e.g. prompt and guidance strength) could indeed improve Imagenet classification. In similar experiment with Stable Diffusion (SD) [41], Sariyildiz et al. [45] studied classifiers trained exclusively on synthetic images (i.e. no real images) and discovered improvements when training on a subset of 100 Imagenet classes. The success of generative data augmentation depends erucially on sample quality [40], so these findings irrefutably highlight the superior generative abilities of diffusion models.
Despite these impressive findings, widespread adoption of diffusion models for synthetic data augmentation is constrained by high computational cost of diffusion sampling, which requires multiple steps of reverse diffusion ensure sufficient sample quality. Furthermore, both SD and Imagen are trained on upstream datasets much larger than Imagenet and some of these improvements could also be attributed to the quality and scale of the upstream dataset [19]. For example, Bansal and Grover [4] find limited ad advantages in synthetic examples generated from a diffusion model trained solely on Imagenet.
Together, these limitations motivate us to explore a diffusion-based augmentation technique that is not only computationally efficient but also enhance classifier training without relying extra data. To that end, we consider the following questions:
Can we leverage a diffusion model trained with no extra data?
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2. Can we train improved classifiers with single step of reverse diffusion?
In the context of reverse diffusion sampling, the intermediate output obtained after one reverse diffusion (i.e., denoising) step is commonly interpreted as an approximation of the final image by previous works and has been utilised to define the guidance function at each step of guided reverse- diffusion (c.g., [3, 10, 11]). Similarly, Diffusion Denoised Smoothing (DDS) [5] applies denoised smoothing [43], a certified adversarial defense for pretrained classifiers, using one reverse diffusion step. In contrast to previous work, we use the output from single reverse diffusson step as an augmentation to train classifiers (i.e., not just at inference time) as we describe next.
Diffuse-and-Denote Augmentation Considering a diffusion model defined such that time t = 0 refers to the data distribution and time generate augmentations of train examples by first applying Gaussian perturbation (i.e., forward- diffusion to a random time E [o, T) and then crucially, applying a single diffusion denoising step (i.e., one-step reverse diffusion). That is, we treat these augmentations of the original train image and refer to this technique as DiffAug. A one-step diffusion denoised example derived from Gaussian perturbed train example can also be interpreted as an intermediate sample in some reverse diffusion sequence that starts with pure noise and ends at the train example. Interpreted this way, our classifier can be viewed as having been trained on partially- synthesized images whose ostensible quality varies from unrecognizable (DiffAug using t = T) to excellent (DiffAug using t 0). This is surprising because, while Ravuri and Vinyals [40] find that expanding the train dataset even with a small fraction of (lower quality) synthetic examples can lead to noticeable drops in classification accuracy, we find that classifier accuracy over test examples does not degrade despite being explicitly trained with partially synthesized train images. Instead, show that diffusion-denoisee examples offer regularization effect when training classifiers that leads to improved classifier robustness without sacrificing clean test accuracy and without requiring additional data.
refers to isotropic Gaussian noise, we propose to
examples as
Our contributions in this work are as follows
(a) DiffAug We propose DiffAug, a simple, efficient and effective diffusion-based augmentation technique. We provide a qualitaaive and analytical discussion on the unique regularization effect complementary to other leading and classic augmentation methods introduced by DiffAu (b) Robust Classification. Using both ResNet-50 and ViT architectures, we evaluate the models in terms of their robustness to covariate shifts, adversarial examples (i.e., certified accuracy under Diffusion Denoised Smoothing (DDS)[5) and -iitr-idiitiitttion detection.
(c) HillAuu-Enneelle (DE) We extend DiffAug to test-time and introduce DE, simple test-time image aggmeentation/adapaatioo technique to improve robustness to covariate shift that is not only competitive with DDA [18], the state-of-the-an art image adaptation method but also 10x faster. (d) Perceptual Gradient Alignment. Motivated by the success of DDS and evidence of perceptually aligned gradients (PAGs) in robust classifiers, we qualitatively analyse the classifier gradients and discover the perceptual alignment described in previous works. We then theoretically analyse the gradients through the score function explain this perceptual alignment.
(e) Improved Classifier-Guided Diffusion. Finally, we build on (d) to improve gradient quality in guidance classifiers and demonstrate improvements in terms of: (i) generalization, (ii) perceptual gradient alignment and (iii) image generation performance.
2 Background
The stochastic diffusion framework [47] consists of two key components: 1) the forward-diffusion (i.e., data to noise) stochastic process, and 2) a learnable score-function that can then be used for the reverse-diffusio (i.e., noise data) stochastic process.
The forward diffusion stochastic process (xe)e)0 starts at data, Xo, and ends at noise, x7. We let
(xx denote the probability density of x time such that (x(x) the data distribution, and pr(x) denotes the noise distribution. The diffusion is defined with a
cinistti-iiffeeetttllleeee dx f(x, t) dt + g(t dw,
(SDE): (I)
where w denotes a standard Wiener process, f(x, t) is a drift coefficient, and g(t) is a diffusion coefficient. The drift and diffusion coefficients usually specified manually such that the solution to
'Code available https: / //github. http:/oore-lb/ddffffe

---

the SDE with initial value Xg is time-varying Gaussian distribution pa (x( whose mean p( t) and standard deviation (t) can be exactly computed.
To sample from po(x) starting with samples from pr(x), we solve the reverse diffusion SDE [1]:
dx = [x t) v(t)²Vx log (x)] dt + g(t) dw.
where dw is a standard Wiener process when time flows from T to O, and di is an infinitesimal negative timestep. In practice, the score function Volog Pt (x) is estimated by a neural network sa sa(x, t). parameterized by 0, trained using score-matching loss [47].
Denoised Examples. Given (xo: y) Po and Pe(x(xx) NN(x 1(xx, t), (t))), we can compute the denoised image &, using the pretrained score network sø as:
& x + (1)so((X,
Intuitively. x, is an expectation over all possible images m, μ(xo, t) that are likely to have been perturbed with (0, (t)l) to generate x and the denoised example X2 can be written as
x = E m, x = /
m, Pr (m,, (x)
We note that the mean does not change with diffusion time t in variance-exploding SDEs while the mean decays to zero with diffusion time for variance- prreerrinn SDEs (DDPMs).
3 DiffAug: Difluse-and-Denoiss Augmentation
In this section, we describe Diffuse-and-Denol Augmentation (DiffAug, in short) and then pro- vide an analytical and qualitative discussion on the role of denoised examples in training clas- sifiers. While we are not aware of any previous study on training classifiers using partially de- noised examples, Diffusion-denee -enooseed smoothing (DDS) [5], DiftPure [36] and Diffusion Driven Adaptation (DDA) [18] are test-time applica- tions of single-step (DDS) and multi-step (DiftPure/DDA) - denoised examples to pro- mote robustness in pretrained classifiers.
X
0
Figure I: DiffAug Augmentations. The leftmost col- umn shows four original training examples (Xo): to the right of that, we display random augmentations (X.) for each image hetween = 350 and = 700 in steps of size 50. Augmentations generated for t < 350 are closer the input image while the augmentations for f >> 700 are farther from the input image. We observe that the diffusion denoised augmentations with larger values l do not preserve the class label introducing noise in the training procedure. However, we find that this does not lead to empirical degradation of classification accuracy but instead contributes improved robustness. Also, see Fig. 6 in appendix for toy 2d example.
As implied by its name, DiffAug consists of two key steps: (i) Diffuse: first, we diffuse train example Xn to a uniformly sampled time t ~ T) and generate
Pa (x(x)); (ii) Denoise: then, we denoise x using a single ap- plication of trained score network se shown in Eq. 3 to generate &,. We assume that the class label does not change upon augmentation (see discussion below) and train the classifier to minimize the following cross-entropy loss:
L E, where, ~ (0, T). (xo, u) ~ Po px(x), and pa denotes the classifier parameterized by Φ. In this work, we show the effectiveness of DiffAug as standalone augmentation technique, as well as the further compounding effect of combining it with robustness-enhancing techniques such Augmix and DecpAugment, showing that DiffAug is achieving a robustness not captured by the other approaches. When combining DiffAug with such novel augmentation techniques, simply include Eq. 5 as an additional optimization objective instead of stacking augmentations (for example, we can alternatively apply DiffAug to images augmented with Augmix/DeepAugnent or vice-versa). Also, our preliminary analysis on stacking augmentations showed limited gains over simply training the network to classify independently augmented samples likely because training on independent augmentations implicitly generalizes to stacked augmentations.
(v)(())
(5)
Qualitative Analysis and Manifold Theory. When generating augmentations, it is important to ensure that the resulting augmentations lie on the image manifold. Recent studies [10, 38] on
theoretical properties of denoised examples suggest that denoised examples can be considered to
3

---

be on the data manifold under certain assumptions lending theoretical support to the idea of using denoised examples as augmentations. We can interpret training on denoised examples as a type of Vicinal Risk Minimization (VRM) since the denoised examples can be considered to lie in the vicinal distribution of training samples. Previous works have shown that VRM improves generalization: for example, Chapelle et al. [8] use Gaussian perturbed examples (x) as the vicinal distribution while MixUp [57] uses a convex sum of two random inputs (and their labels) as the vicinal distribution. From Eq. 4, we can observe that a denoised example is a convex sum over m, and we can interpret & as being vicinal to examples m, that have a non-trivial likelihood, Pr (m:[x), of generating x. The distribution Pa (m/|x) is concentrated around examples perceptually similar u(xx, t) when is closer to Xa (i.e,, smaller a (t)) and becomes more entropic as the noise scale increases: we can qualitatively observe this in Fig. 1.
Diffusion denoised augmentations generated from larger (4) can introduce label-noise into the training since the class-labels may not be preserved upon augmentation - for example, some of the diffusion denoised augmentations of the dog in Fig. resemble architectural buildings. Augmenta- tions that alter the true class-label are said to cause manifold intrusion [21] leading to underfitting and lower classification accuracies. In particular, accurate predictions on class-altered augmented examples would be incorrectly penalised causing the classifier to output less confident predictions on all inputs (i.e., underfittin). Interestingly, however, diffusion denoised augmentations that alter the true-class label are also of lower sample quality. The correlation between label noise and sample qual- ity allows the model to selectively lower its prediction confidence when classifying denoised samples generated from larger perturbations applied to X, (we empirically confirm this in Section 4). In other words, the classifier learns to observe important details in & to determine the optimal prediction estimating the class-membership probabilities of Xg- On the other hand, any augmentation that alters class-label by preserving the sample quality can impede the classifier training since the classifier cannot rely on visual cues to selectively lower its confidence (for an example, see Fig. 7 in appendix). Therefore, we do not consider multi-step denoising techniques to generate augmentations - despite their potential improve sample quality- since this would effectively decorrelate label-noise and cumple-quality necessitating additional safeguards - e.g.. we would then need to determine the maximum diffusion time we could use for augmentation without altering the class-label or scale down the loss terms corresponding to samples generated from larger (t). We leave this exploration to future work and include a preliminary analysis in Appendix B.2.2.
Test-time Augmentation with DiffAug. Test-time Augmentation (TTA) [30] is a technique to improve classifier prediction using several augmented copies of a single test example. A simple yet successful TTA technique is to just average the model predictions for each augmentation of a test sample. We extend DiffAug to generate test-time augmentations of test-example wherein we apply DiffAug using different values of diffusion times / and utilize the average predictions across all the augmentations to classify the test example Xy:
p(y)xo)= Xo)
1
Σ Po(U(R.) EES
where, S denotes the set of diffusion times considered. We refer to this as DiffAug Ensemble
(DE). A forward diffusion step followed by diffusion denoising can be interpreted as projecting a test-example with unknown distribution shift into the source distribution and forms the basis of DDA, a diffusion-basse based image adaptation technique. Different from DE, DDA uses a novel multi-step denoising technique to transform the diffused test example into the source distribution. Since DE uses single-step denoised examples of forward diffused samples, we observe significant improvement in terms of running time while either improving over remaining on par with DDA.
Experiments I: Classifier Robustness
In this section, we evaluate classifiers trained with DiffAug in terms of their standard classification accuracy as well as their robustness to distribution shifts and adversarial examples. We primarily conduct our experiments on Imagenet- 1k and use the unconditional 256x256 Improved-DDPM [14, 35] diffusion model to generate the augmentations. We apply DiffAug to train the popular ResNet-50 (RN-50) backbone as well as the recent Vissoo-Transformmr (ViT) model (ViT-B-16 16, in particular). In addition to extending the default augmentations used to train RN-50/ViT with DiffAug. we also combine our method with the following effective robustness-enhanci augmentations: (i) AugMix (AM), (ii DeepAugment (DA) and (ii) DeupAugment+Auuggix (DAM). While we train

---

Table 1: Top-1 1 Accuracy (%) on Imagenet-C (5everity=5) Imagenet-Test Test. We summarize the results for each combination of Train-augmentations and evaluation modes. The average (avg) accuracies for each classifier and evaluation mode is shown.
ImageNet-C (severity 5)
ImageNet Test
Train Augmentations
DDA (SF)
DDA
DDA
DE
Def.
Avg
DDA
DE
Def.
Avg
(SF)
AM
33.18
36.54
34.08
26.72
32.63
62.23
75.98
73.8
77.53
72.39
AM+DifIAug
34.64
38.61
38.58
29.47
35.33
63.53
76.09
75.88
77.34
73.21
DA
35.41
39.06
37.08
31,93
35.87
63.63
75.39
74.28
76.65
72.49
DA+DifAAu
37.61
41.31
40.42
33.78
38.28
65,47
75.54
75.43
76,51
73.24
DAM
40.36
44.81
41.86
39.52
41.64
65.54
74.41
73.54
75.81
72.33
DAM-DifAAug
41.91
46.35
44.77
41.24
43.57
66.83
74.64
74.39
75.66
72.88
RN50
28.35
30.62
27.12
17.87
25,99
58.09
74.38
71.43
76.15
70.01
RNSO+DifTAug
31.15
33.51
32.22
20.87
29.44
61.04
74.87
75.07
75.95
71.73
ViT-B/I6
43.6
52.9
48.25
50.75
48.88
67.4
81.72
80.43
83.71
78.32
ViT-B//6-DiffAug
45.05
53.54
51.87
52.78
50.81
70.05
81.85
82.59
83,59
79.52
Avg
37.13
41.73
39.63
34.49
38.24
64.38
76.49
75.68
77.89
73.61
Arg (No-DifTAug)
36.18
40.79
37.68
33.36
37.00
63.38
76,38
74.70
77.97
73.11
Arg (DiffAug)
38.07
42.66
41.57
35.63
39.48
65,38
76.60
76.67
77.81
74.12
the RN-50 from scratch, we follow DeIT-III recipe[50] for training ViTs and apply DiffAug in the second training stage; when combining with AM/DA/DAM, we finetune the official checkpoint for 10 epochs. More details are included in Appendix B.1. the following, we will evaluate the classifier robustness to (i) covariate shifts, (ii) adversarial examples and (iii) out-of of-disribittttinnnnnn examples.
Covariate Shifts To evaluate the classifiers trained with/without DiffAug in terms of their robustness to covariate-shifts, we consider the following evaluation modes:
(a) DDA: A diffusion-based test-time image-adaptation technique to transform the test image into the source distribution.
(b) DDA-SE: We consider the original test example well the DDA-adapted test-example by averaging the classifier predictions following the self-ensemble (SE) strategy proposed [18]. (c) DiffAug-Ensemble (DE): We use set of test-time DiffAug augmentations to classify test example as described in Eq. (6). Following DDA, we determine the following range of diffusion times S = f0, 50, ..... 450]. In other words, generate 9 DiffAug augmentations for each test example.
(d) Default: In the default mode, directly evaluate the model the test examples.
We evaluate the classifiers on Imagenet-C, a dataset of synthetic corruptions applied to Imagenet- test and summarize the results across all evaluation modes in Table 9. We summarize observations as follows:
(i) DiffAug introduces consistent improvements Classifiers trained with DiffAug consistently improve over their counterparts trained without these augmentations across all evaluation modes. The average relative improvements across all corruptions range from 5.3% to 28.7% in the default evaluation mode (see Table in Appendix). On clean examples, we observe that DiffAug helps minimize the gap between default evaluation mode and other evaluation modes while effectively preserving the default accuracy.
(ii) DE improves over DDA On average, DiffAug Ensemble (DE) yields improved detection rate as compared to direct evaluation on DDA images. Furthermore, DiffAug-traince classifiers evaluated using DE improve average
their counterparts (trained without DiffAug) evaluated using DDA-SE. This experiment interestingly reveals that a set of one-step diffusion denoised images (DE) can achieve improvements comparable to multi-step diffusion denoised images (DDA) at a substantially faster (~ 10x) wallclock time (see Table 10 in Appendix).
DiffAug vs Extra Synthetic Data: Bansal and Grover [4] demonstrate that foundation models such as Stable-Diffusion can used to generate additional synthetic data to improve classifier robustness to covariate shifts. We note that extra synthetic data with diffusion models trained exclusively on ImageNet do not help enhancing classifier robustness (see Appendix B.2.1). particular, Bansal and Grover [4] utilise diverse prompts to generate a synthetic clone of Imagenet consisting of 1.3M
5

---

Table 2: Top-1 Accuracy (%) across different types of distribution shifts when additional high-quality synthetic data from training and DiffAug- Ensemble (DE) inference. For reference, we also include the results for the corresponding ResNet50 models without extra synthetie data.
is available (denoted by +Synth). We show the net improvement obtained by DiffAug
ImageNet-C (Severity=5)
ImageNet Sketch
Model
ImageNet-R
ImageNet-S
ImageNet-A
ImageNet-D
Average
RN50
17.87
36.16 41.61 (+5.4))
7.12 12.52 (+5.40)
24.09 26.67 (+2.56)
0.03 1.09 (+1.06)
11.36
16.10
+DiffAug/DE
32.22 (+14.85)
11.37 (+0.01)
20.90
RN50+Synth
17.58
49.28
7.68
35.45 37.39 (+1.94)
0.63
17.52
21.35
+DiffAug/DE
30.06 (+12.48)
54.71 (45.43)
13.57 (+5.89)
1,53 (+0.9)
21.41 (+3.89)
26.45
images. We utilize their open-soureed synthetic dataset and fine-tune the torchvision resnet-50 model for 10 epochs using both real and synthetic images and call this RN50+Synth. Next, we repeat this finetuning process with DiffAug and call this RN50+Synh++DffAuug In Table 2, we summarize the results across many datasets and interestingly observe that DiffAug - using an ImageNet-only diffusion model- - offers improvements over and beyond additional synthetic data using Stable- Diffusion. This is surprising due to the following reasons: (i) SD trained on LAION-5B, a much larger dataset that also subsumes ImageNet. (ii) Additional synthetic data requires more compute per each sample (e.g., 50 reverse-diffusion steps) whereas DiffAug just uses one reverse-diffusion step.
While the improvements offered by the extra synthetic data can be largely attributed to the upstream training dataset. DiffAug augmentations offer complementary regularization benefits. To understand this, we first note that DiffAug augmentations are qualitatively distinct from high-quality synthetic images and can be interpreted as lying on the image manifold in regions between nigh-quality samples: depending on the diffusion time, the DiffAug augmentation can vary greatly in quality (as shown in Fig. 1). As a result, classifying some of these augmented images is more challenging as compared to the original examples producing a regularizing effect that leads to empirical robustness improvements.
We include detailed evaluation across different datasets including ImageNet- R/S in the Appendix (Table 7). Overall, we observe that DiffAug training and DE inference help significantly enhance robustness to covariate-shifts in several cases.
hut-of-Distribution (OOD) Detection. Test examples whose labels do not overlap with the labels of the train distribution are referred to as out-of-distribution examples. To evaluate the classifiers in terms of their OOD-detection rates, we use the Imagenet near-OOD detec- tion task defined in the OpenOOD benchmark, which also includes an implementation of recent OOD detection algorithms such as ASH[15], ReAct[49], Scale[54] and MSP[23]. For context, while the torchvision ResNet-50 checkpoint is most commonly used to evaluate new OOD de- tection algorithms, AugMix provides the best OOD detection amongst the existing robustness- enhancing augmentation techniques and Aug- Mix/ASH is placed 3rd amongst. 73 methods on the OpenOOD Icaderboard (ordered by near- OOD performance). Yet in Table 3 we observe that DiffAug introduces featuceesfutterrrr improvement on the challenging near-OOD detection task across all considered OOD algorithms.
Table 3: AUROC on Imagenet Near- Near-OOD Detection.
Train Augmentation
ASH
MSP
ReAct
Scale
Avg.
AugMix(AM)
82.16
77.49
79.94
83.61
80.8
AM+DiffAug
83.62
78.35
81.29
84.81
82.02
RN50
78.17
76.02
77.38
81.36
78.23
RN50+DiffAug
79.86
76.86
78.76
82.81
79.57
Maximum Maximym__Etrropx
DA
Entropy (nats)
+ DiffAuu
AM
DiffAug DAM
DiffAug
200
400
600
800
Figure 2: Average prediction entropy on DiffAug sam- ples vs diffusion time measured with Imagenet- Test. We observe that the models trained with DiffAug correctly yield predictions with higher entropies (lower confi- dence) for images containing imperceptible details (i.c. larger f). Surprisingly, the classifiers trained without DiffAug do not also assign andom-uniform label distri- bution for DiffAug images at = 999, which have no clas-iiffomation by construction. Also, see Fig. 11.
Comparing our results to the leaderboard, we ob- serve Aug Dix--iifAug//Sale achieves an AU- ROC of 84.81 outperforming the second best method (84.01 AUROC) and comparable to the top AUROC of 84.87.

---

DiffAug training teaches the network to selectively lower its prediction confidence (higher prediction entropy) based on the image content (Fig. 2) and we hypothesize that this leads to improved OOD detection rates. We include the detailed OOD detection results in Appendix B.4. Interestingly, the combination of augmentations that improve robustness on covariate shifts may not necessarily lead to improved OOD detection rates: for example, AugMix+DeepAAumeet and Deep Augment on covariate shift but achieves lower OOD detection rates than either. On the other hand, we observe that combining with DiffAug enhances both OOD detection as well as robustness to covariate shift.
improves over both AugMix
Certified Adversarial Accuracy. Denoised smoothing [43] is certified defense for pretrained classifiers inspired from Randomized smoothing [12] wherein noisy copies of a test image are first denoised and then used as classifier input to estimate both the class-label and robust radius for each example. Using the same diffusion model as ours, DDS[5] already achieves state-of-the-ard certified Imagenet accuracies with pretrained 305M-parameter BelTT-L. Here, we evaluate the improvement in certified accuracy when applying DDS to a model trained with DiffAug and include the results in Appendix B.3. We speculate that finetuning the BelT-L model with Diff'Ag should lead to similar improvements but skip this experiment since it is computationally expensive.
Perceptually Aligned Gradients and Robustness. Classifier gradients (V, log (y z) where z is an image) which are semantically aligned with human perception are said to be perceptually aligned gradients (PAG) of typical image classifier are usually unin- telligible, gradients obtained from adversarially robust classifiers trained using randomized smoothing [28] or adversarial training [16, 44, 52] are perceptualll-aaignen Motivated by the sfate-of-the-art certified adver- sarial accuracy achieved by DDS, we analyse the classifier gradients of one-step diffusion denoised examples - i.e. we analyse Vx lo
[17]. While
Figure 3: PAG exampic using NiT+DidAAug We diffuse the Imagenet example (leff) to 300 vistalise the min-max nomalized classifter gradients (rihh, For easy viewing, we apply contrstt maximization. More examples are shown below.
pa (y)xx) where x = Pt (x (x) and & = x + o² (x)) t). We visualise the gra- dients in Fig. 3 and interestingly discover the same perceptual aligmment of gradients discussed in previous works (we compare with gradients of
classifiers trained with randomized smoothing later). To theoretically analyse this effect, we first decompose the input-gradient using chain rule as:
d log (v(x)) d d (x.) d&c dx
dx;
dx
Empirically, we find that the perceptual alignment is introduced due to transformation by di: and analyse it further:
dx
Theorem 4.1. Consider forward-diffusin SDE defined as in Eq. such that px(x)
N(x m,, (t)() where m, = :(xo:t). If x ~ Pe(x) and & = x + o² (t)ss) t), for opti- mal parameters O, the derivative of 8, w.r.t. x is proportional to the covariance matrix of the conditional distribution p(m||x). See proof in Appendix B.6.
0xz
J (t(
Cov(m: [x]
0x
This theorem shows us that the multiplication by 0%. in Eq (7) is in fact a transformation by 8x
the covariance matrix Cov pommm,// x). Multiplying a vector by Cov(m,le stretches the vector along the principal directions of the conditional distribution p(m/(x). Intuitively, since the conditional distribution p(m,) |x) corresponds to the distribution of candidate denoised images, the principal directions of variation are perceptually aligned (to demonstrate, we apply SVD to and visualise the principal components in Appendix B.7) and hence stretching the gradient along these directions will yield perceptually aligned gradients. We note that our derivation complements Proposition in Chung et al. [10] which proves certain properties (c.g.. J ") of this derivative. In practice, however, the score-function is parameterized by unconstrained, flexible neural architectures that do not have exactly symmetric jacobian matrices J. For more details on techniques to enforce conservative properties of fuore-functions, we refer the reader to Chao et al. [7]. Ganz et al. [17] demonstrate that training a classifier have perceptually aligned gradients also improves its robustness exposing the bidirectional relationship between robustness and PAGs. This works offers additional evidence supporting the co-occurrence of robustness and PAGs since we observe that classification of diffused-and-denoised images (e.g., DDS, DE, DDA) not only improve robustness but also produce PAGs.
Ablation Analysis. Appendix B.5 includes ablation study on the following:
7

---

(a) Extra Training, The pretrained DA, AM, and DAM classifiers are sufficiently trained for 180 epochs and hence, we compare the DiffAug finetuned model directly with the pretrained checkpoint. For completeness, we train AugMix for another 10 epochs and confirm that there is no notable change in performance as compared to results in Tables 3, and
(b) DiffAug Hyperparameters. In our experiments, we considered the complete range of diffusion time. We investigate a simple variation where we either use t E [o, 500) or t E [500, 999] to generate the DiffAug augmentations.
(c) DiffAug-Ensemble
Hlpeepaarameters. We analyse how the choice of diffusion times considered in the set S (Eq. (6)) affects DE performance.
(d) Conditional DiffAug. While our DiffAug experiments mainly utilize unconditional diffusion models, we can also utilize conditional diffusion models and explore an extension of AugMix with conditional DiffAug for various guidance strengths.
(e) Latent-Space Diffusion Models. We also tried the DiffAug augmentation method with Diffusion- Transformer (DiT): while we observed that the test- ast-accuracy is preserved even in this case, we only observe slight robustness improvements compared with pixel-space diffusion models. This may be explained by noting that the VAE-decoder is trained to output perceptually high- quality image and hence, does not accurately capture the expectation in Eq. (4).
5 Experiments II: Classifier-uuided Diffusion
Classifier guided (CG) diffusion is a conditional generation technique to generate class-conditional samples with an unconditional diffusion model. To achieve this, a time-conditional classifier is separately trained to classify noisy samples from the forward diffusion and we refer to this as a guidance classifier
(8) each step of the classifier-guidd reverse diffusion (Eq. (2)), the guidance classifier is used to compute the class-conditional score Vx (ogg:((]j) (xly) = Vx log Ps Pp(yy) x, () + A. vx log ())pp))) (A, is classifier scale[14)), which is used in place of unconditional score Vx (x).
LCE = Ex,l log (t)
where, t U(00 T). x ~ (x(xx) and (xo. y) ~ (x)...
The guidance classifiers participate in the sampling through their gradients, which indicate the pixel-wise perturbations that maximizes log-likelihood of the target class. Perceptually aligned gradients that resemble images from data distribution lead to meaningful pixel-wise perturbations that could potentially improve classifier- cassifier-guiiance and forms the motivation of Kawar et al. [29], where they propose an adversarial training recipe for guidance classifiers. With the same motivation, we instead build on Theorem 4.1 in order to improve perceptual alignment and propose to train guidance classifiers with denoised examples &, derived from x. While the obvious choice is to simply train the guidance-classfier on &, instead of x, we choose to provide both x as well as &, as simultaneous inputs to the classifier and instead optimize Lcr: = TF,,, [- log P& x, &r, O)] (compare with Eq. (8)). We preserve the noisy input since the primary goal of guidance classifiers is to classify noisy examples and this approach enables the model to flexibly utilize information from both inputs. We refer to guidance- cllassifiiss trained using both and & as denoising-aut augmented (DA) classifier and use noisy classifiers to refer to guidance-classifee
Denoising-Augmented (DA) Classifier.
trained exclusively on x.
Experiment setup. We conduct our experiments on CIFARIO and Imagenet and evaluate the advantages DA-Classifiers over noisy classifiers. While we use the same Imagenet diffusion model described in Section 4, we use the deep NCSN++ (continuous) model released by Song et al. [47] as the score-network for CIFARIO (VE-Diffusion). As compared to the noisy classifier, the DA-classifier has an additional input-convolution layer to process the denoised input and is identical from the second layer onwards. We describe our classifier architectures and the training details in Appendix C.I.
Table 4: Summary of Test Accura- cics for CIFAR1O and Imagenet: cach test example is diffused to random uniformly sampled dif- fusion time. Both classifiers are shown the same diffused example.
Classification Accuracy. We first compare guidance classifiers terms of test accuracies as measure of their generalization (Ta- ble 4) and find that DA-classifiers generalize better to the test data. Training classifiers with Gaussian perturbed examples often leads to underfitting [59] explaining the lower test accuracy observed with noisy classifiers. Interestingly, the additional denoised example helps address the underfitting for example, see Fig. 19 (in appendix). One explanation of this finding could be found in Chung et al. [10].
Method
CIFARIO
Imagenet
Noisy Classifier
$4.79
33.78
DA- Dl--Classffie
S7.16
36.11
8

---

(a) PAG: Noisy-classifer DA-Classifier
(b) Generated Samples: Noisy- Joisy-Classifier vs DA- AA-Classifier
Figure 4: Min- max normalized gradients on clean samples (left column) diffused to = 300 (T = 999). For easy comparison between Noisy classifier gradients (middle column) and DA-classifeer gradients (right column), we applied an identical enhancement to both images, i.e. contrast maximization. The unedited gradients are shown in Fig. 20. (b) Qualitative Comparison of Guidance Classifiers the Image Generation Task using DDIM-100 with same random seed. In each pair, the first image generated with the Noisy Classifier and the second image generated with the Denoising Augmeetted (DA) Classifier. We observe that the Denoising- Augmented (DA) Classifier improves overall coherence as compared to the Noisy Classifier. Also see Fig. 25 in appendix for more examples.
Table 5: Quantitative comparison of Guidance Classifiers on the Image Generation Task using 50k samples. We also show unconditional precision/recall (P/R) and the average class-conditional Preesionnecall//Daiiyyyovrrrrr
Method
CIFARIO
Imagenet
FID4
18
P1
R1
Pt
R1
D1
Čt
FID 1
sFID
IS
Pa
Noisy Classifier
2.81
0.64
0.62
0.57
0.71
5.44
5.32
194.48
0.81
0.49
DA-Classifier
2.34
9.88
0.65
0.63
0.63
0.64
0.92
0.77
5.24
5.37
201.72
0.81
0.49
where they distinguish between noisy examples and their corresponding denoised examples as being the ambient space and on the image manifold respectively, under certain assumptions. To determine the relative importance of noisy and denoised examples in DA- -laassssise we zeroed out one of the input images to the CIFARIU 10 classifier and measured classification accuracies: while zeroing the noisy input caused the average accuracy across all time-scales to drop to 50.1 1, zeroing the denoised input breaks the classifier completely yielding random predictions.
Classifier Gradients. In Fig. 4a, we qualitatively compare between the noisy classifier gradients (y/x, x, X, t)). We find that the gradients
xVx log x, t)) and the DA-classifier gradients(V log Po (o obtained from the DA-classifier are more structured and semantically aligned with the clean image as compared the ones obtained with the noisy classifier (see Figs. 20 and in Appendix for more examples). Gradients backpropagated through the denoising score network have been previously utilized (e.g., [3, 10, 11, 18. 26, 36)), but our work is the first to observe and analyze the qualitative properties of gradients obtained by backpropagating through the denoising module (also see Fig. 21 in appendix).
Image Generation. To evaluate the guidance classifiers in terms of their image generation, we generate 50k images each see Appendix C.2 for details on the sampling parameters. We com- pare the classifiers in terms of standard generative modeling metrics such as FID, IS, and P/R/D/C The P/R/D/C metrics compare between the manifold of gen- erated distribution and manifold of the source distribution in terms of searest-neighhours and can be computed conditionally (i.e., classwise) or unconditionally. Following standard practice, we additionally evaluate CIFAR 10 classifiers on
P/R/D/C. Our results (Table 5) show that our proposed Denoising- Manoiiing-Augmentee (DA) Classifier improves upon the Noisy Classifier terms of FID and IS for both CIFAR10 10 and Imagenet at roughly same Precision and Recall levels (see Appendix C.3 for comparison with baselines). Our evaluation of average class-conditional precision, recall, density and coverage for each CIFAR10 class also shows that DA-classifiers outperform

---

Noisy classifiers: for example, DA-classifiers yield classwise density and coverage of about 0.92 and 0.77 respectively on average compared 0.78 and 0.71 obtained with Noisy-Classifies We can attribute our improvements in the class-conditional precision, recall, density and coverage to the improved generalization of -lacssssifier. To qualitatively analyse benefits of the DA-classifier, we generated Imagenet samples using DDIM-100 sampler with identical random seeds and X 2.5. In our resulting analysis, we consistently observed that the DA-classifier maintains more coherent fore- ground and background as compared to the Noisy Classifier. We show examples in Fig. 4b. Overall, we attribute improved image generation to the improved generalization and classifier gradients.
Related Works
Synthetic Augmentation with Diffusion Models have also been explored to train semi-supervised classifiers [55] and few-shot learning [51]. Other studies on training classifiers with synthetic datasets generated with a text2image diffusion model include [4, 22, 56]. Apart from being computationally expensive, such text2image diffusion models are trained on large-scale upstream datasets and some of the reported improvements could also be attributed to the quality of the upstream dataset. Instead, we propose a efficient diffusion-basee augmentation method and report improvements using diffusion model trained with no extra data. Further, we also find that DiffAug is complementary to synthetic training data generated with large text2image diffusion models and leave further exploration of DiffAug with larger diffusion models as future work.
Synthetic examples have also been shown be useful for enhancing training adversarially robust classifiers (e.g.. [27, 53]) and extension of Diff'Agg for compute-efficien adversarial training is apt for exploration in future work.
Diffusion Models for Robust Classification. Diffusion-classifii [31] is a method for zero-shot classification but also improves robustness covariate shifts. Dift-TTA[3) is test-time adaptation technique to update the classifier parameters at test time and is complementary to classifier training techniques such DiffAug. In terms of OOD detection, previous works have proposed reconstruction- based metrics for ood detection [20, 32]. To the best of our knowledge, this work is the first to demonstrate improved OOD detection on ImageNet- 1k using diffusion models.
7 Conclusion
In this work, we introduce DiffAug to train robust classifiers with one-step diffusion denoised examples. The simplicity and computational efficiency of DiffAug enables us to also extend other data augmentation techniques, where we find that DiffAug confers additional robustness without affecting accuracy on clean examples. We qualitatively analyse DiffAug samples in an attempt to explain improved robustness. Furthermore, we extend DiffAug to test time and introduce an efficient test-time image adaptation technique to further improve robustness to covariate shifts. Finally, we theoretically analyse perceptually aligned gradients in denoised examples and use this to improve classifier guidded diffusion. Overall, present effective augmentation technique using diffusion models trained with no external datasets.
Acknowled leknowledgements
We thank the Canadian Institute for Advanced Research (CIFAR) for their support. Resources used in preparing this research were provided, in part, by NSERC, the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www. www.vectoring poctorinstitutee
ai/#part We thank Kry Yik Lui for their feedback and suggestion to explore connections with Senoosed-ssootthigg We thank Scott Lowe, Emily Napier, Marvin F. Silva and Felix Dangel for providing feedback on our figure/terminologie., Chandramouli Sastry is thankful to the teachings of Shankaracharyas and Guru Parampara for instilling the values of enquiry, patience and perseverence. Chandramouli Sastry is also supported by Borealis AI Fellowship. We sincerely thank the anonymous reviewers and area-chairs for their service and valuable feedback & suggestions to strengthen our paper.
References
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastie Processes and their Applications. 12(3):313-322 1982.
10

---

[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Trunsactions on Machine Learning Research, 2023.
[3] Arpic Bansal, Hong-Ming Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Universal guidance for diffusion models, 2023.
Hritik Bansal and Aditya Grover. Leaving reality imagination: Robust classification via generated datasets, 2023.
[5] Nicholas Carlini, Florian Tramèr, Krishnamurthy Dvijotham, Leslie Rice, Mingjie Sun, and Zico Kolter. (certified!!) adversarial robustness for free! International Conference on Learning Representations (ICLR). 2023.
[6] Chen-Hao Chao, Wei-Fang Sun, Bun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun- Lee. Denoising likelihood score matching for conditional score-based data generation. In International Conference on Learning Representations. 2022.
[7] Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, and Chun-Yi Lee. On investigating the conservative property of score-based generative models, 2023.
[8] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, 13, 2000.
[9] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi, Diffusion models for imperceptible and transferable adversarial attack. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-17, 2024.
[10] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffussio models for inverse problems using manifold constraints. In Advances in Neural Information Processing Systems, 2022.
[11] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023.
[12] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. CoRR, abs//192.022 2019.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, pages 8780-8794. Curran Associates, Inc., 2021.
[14] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis. CoRR, abs22105.05233, 2021.
[15] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. The Eleventh International Conference on Learning Representations, 2023.
[16] Andrew Elliott, Stephen Law, and Chris Russell. Explaining classifiers using adversarial perturbations on the perceptual ball. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 10693-10702 Computer Vision Foundation / IEEE, 2021.
[17] Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply adversarial robust- ness?, 2023.
[18] Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrel, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven test-time adaptation. arXiv preprint arXiv:2207.03442 2022.
[19] Scott Geng, Ranjay Krishna, and Pang Wei Koh. Training with real instead of synthetic generated images still performs better. In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024.
[20] Mark S. Graham, Walter H. L. Pinaya, Petru-Daniel Tudosiu, Parashkev Nachev, Sebastien Ourselin, and M. Jorge Cardoso. Denoising diffusion models for out-of-distribution detection, 2023.
[21] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear mun-ool- regularization, 2018.
[22] Ruifei He, Shuyang Sun, Xin Yu, Chubui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, 2023.
11

---

[23] Dan Hendrycks and Kevin Cimpel. A baseline for detecting misclassified and out-of-distribttinn examples in neural networks. In International Conference on Learning Representations, 2017.
[24] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 2019.
[25] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
[26] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
[27] Kai Hu, Klas Leino, Zifan Wang, and Matt Fredrikson. recipe for improved certifiable robustness. In The Twelfth International Conference on Learning Representations, 2024.
[28] Simran Kaur, Jeremy M. Cohen, and Zachary C. Lipton. Are perceptually-alignee gradients a general property of robust classifiers? CoRR, 1bs//190,00860,, 2019.
[29] Bahjat Kawar, Roy Ganz, and Michacl Elad. Enhancing diffusion- bbaed image synthesis with robust classifier guidance. Transactions on Machine Learning Research, 2023.
[30] Masanari Kimura. Understanding test-time augmentation, 2024.
[31] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. arXiv preprint arXiv:2303.16203 2023.
[32] Zhenzhen Liu, Jin Peng Zhou, Yufan Wang, and Kilian Q Weinberger. Unsupervised out-of- detection with diffusion inpainting, 2023.
[33] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, USA. 2024. Curran Associates Inc.
[34] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. CoRR, abs/2101.02388, 2021.
[35] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. CoRR, abs/2102.09672, 2021.
2021.
[36] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar, Diffusson models for adversarial purification. In International Conference on Machine Learning (ICML), 2022. 2022.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:211.009748 2022.
[38] Frank Permenter and Chenyang Yuan. Interpreting and improving diffusion models using the euclidean distance function. arXiv preprint 2xXi:2336604884. 2023.
[39] Mihir Prahhudesai, Tsung- Wei Ke, Alexander C. Li, Deepak Pathak, and Katerina Fragkiadaki. Test- time adaptation discriminative models via diffusion generative feedback. In Conference on Neural Information Processing Systems, 2023.
[40] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in neural information processing systems, 32, 2019.
[41] Robin Rombach, Andreas Blatmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CoRR, abs/2112.1177555 2021.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3666 36494, 2022.
[43] Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and Zico Kolter. Black-hox smoothing: provable defense for pretrained classifiers. CoRR, abs/2003.01 2020.
[44] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019.
2019, Vancouver, BC, Canada, pages
12

---

[45] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones, 2023.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 2rXiv:0010002502, 2020.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[48] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. 2023.
[49] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations, In Advances in Neural Information Processing Systems, pages 144-157. Curran Associates, Inc., 2021.
[50] Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.077 2022.
[51] Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference Learning Representations, 2024.
[52] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness may be at odds with accuracy. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[53] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. Proceedings of the 40th International Conference on Muchine Learning, pages 36246-36263. PMLR, 2023.
[54] Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc out-of- distribution detection enhancement, In The Twelfth International Conference on Learning Representations, 2024.
[55] Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi- supervised learners benefit mutually with few labels. In Thirth--eventth Conference on Neural Information Processing Systems, 2023.
[56] Jianhao Yuan, Francesco Pinto, Adam Davies, and Philip Torr Not just pretty pictures: Toward interven- tional data augmentation using text-lo-image generators, 2023.
[57] Hongyi Zhang. Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018.
[58] Guangcong Zheng, Shengming Li, Hui Wang. Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy- driven sampling and training scheme for conditional diffusion generation. In European Conference on Computer Vision, pages 754-769. Springer, 2022.
[59] Stephan Zheng, Yang Song, Thomas Leung, and Ian J. Goodfellow. Improving the robustness of deep neural networks via stability training. CoRR, abss/660..04322 2016.
13

---

Partl I
Appendix
Table of Contents
Statements of Contributions, Limitations & Broader Impact
14
A Appendix for Section 3
15
B Appendix for Section 4
16
B.1
Training Details
16
B.2 More Results on Covariate Shift Robustness
B.3
Certified Accuracy Experiments.
20
B.4
Detailed OOD Detection Results
21
B.5
Ablation Experiments
22
Derivation Theorem 4.1
26
B.7 Analysis with SVD Decomposition
27
C Appendix for Section 5
27
C.I Guidance Classifier Training: Experiment Details
27
C.2 Glaiiefier-Guided Diffusion: Sampling
30
Comparisons with DLSM, ECT and Robust Guidance
30
D Uncurated Samples
33
Contribution Statement
CSS and SHD are student contributors of this project and their contributions are as follows: CSS proposed the key research idea, designed and carried out all the experiments in this paper, worked on the theoretical derivations and wrote the first draft of this paper. SHD provided advice and constructive criticisms throughout the research and development phase.
Limitations and Future Work
In this work, we introduced DiffAug, simple diffusion-based augmentation technique improve classifier robustness. While we presented unified training scheme, follow previous works and evaluate robustness to covariate shifts, adversarial examples and out-of-distribution detection using separate evaluation pipelines (for e.g., we do not apply DDA OOD examples). A unified evaluation pipeline for classifier robustness is an open problem and out of scope for this paper. The key advantage of our method is its computational efficiency since it requires just reverse diffusion step to generate the augmentations: however, this is computationally more expensive than handcrafted augmentation techniques such as AugMix. Nevertheless, it is fast enough that can generate the augmentations online during each training step. With recent advances distilling diffusion Donsistency-Modee [48], it may be possible generate better quality synthetic examples within the training loop. Since the improved robustness introduced by DiffAug be attributed the augmentations of varying image quality (Fig. 1), believe that this complementary regularization effect can still be valuable with efficient high-quality sampling and leave further exploration to future work. Likewise, DiffAug is complementary to the of additional synthetic data generated offline with text2image diffusion models and the extension of DiffAug text2image diffusion models suitable for future investigation. We also extend DiffAug to test time and propose DE, wherein we demonstrate improvements comparable to DDA at 10x wallclock time on all classifiers except ViT on Imagenet-C; while this demonstrates limitation of single-step denoising (i.e., DE) vs. multi-step denoising(i.e., DDA), we also note that DE improves over DDA for all classifiers when considering Imagenet-R and Imagenet-S. While we demonstrate improvements over existing baselines of classifier -uuiiiier-guidance (e.g., [6, 29, 58]), we do not compare with classifier-free guidance [25], the popular guidance method, since our primary focus on the training of robust classifiers with DiffAug and demonstrating the potential of DiffAug. Training classifier-guided diffusion models for careful comparison requires additional computational resources and we leave this analysis to future work. Nevertheless, Mlassffirrguiddane is computationally
models for fast sampling [34] such
14

---

attractive alternative to perform conditional generation since this allows flexible reuse of a pretrained diffusion model for different note that computing dlassifiergrradients is slower for DA-classifiers as compared to noisy classifiers since it requires backpropagating through the score-network and shares this limitation with other recent works that utilize intermediate denoised examples for guidance (e.g., Bansal et al. [3], Chung et al. [10, 11], Ho et al. [26]) - the recent advances in efficient diffusion sampling could be extended to class-conditional sampling to reduce the gap.
by separately training small classifier model. We also
Compute Resources
For this paper, had access 40GB A40 GPUs conduct our training and evaluation. We used a maximum of 4 GPUs for each job and the longest training job was the 90-epoch RN-50 training followed by the 20-epoch ViT training. The evaluation of classifiers on Imagenet-C. Imagenet-R R and Imagenet-S using DE and Default evaluation modes are fairly fast. However, generating DDA examples for the entire Imagenet-C dataset and Imagenet-test dataset is computationally expensive and takes up to a week (even while using 8 GPUs in parallel) and also requires sufficient storage capacity to save the DDD-Irrasformedd images. Likewise, evaluation of certified accuracy with DDS is also computationally expensive since it uses 10k noise samples per example to estimate the certified radius and prediction. Training CIFAR 10 guidance classifiers are fairly efficient while finetuning the Imagenet guidance classifier can take up 3 days depending on the gradient accumulation parameter GPUs. On average, the 50k CIFARI0 10 and Imagenet images that we sample for evaluation of guidance classifiers can be done within a maximum of 36 hours depending on how we parallelize the generation.
(i.e., number of GPUs available) when available, we used a maximum
Broader Impact
The main contribution of this paper is to introduce new augmentation method using diffusion models to improve classifier robustness. With increasing deployment of deep learning models in real-world settings, improved robustness is crucial to enable safe and trustworthy deployment. Since we demonstrate potential diffusion models trained with no extra data as compared to the classifier, we anticipate that this will be useful in applications where such extra data is not easily available (e.g., medical imaging). We also extend this method to improve classifier- glassifier-guided diffusion. Improvements in classifier- llsddd diffusion can be used in developing myriad downstream applications, each with their own potential balance of positive and negative impacts. Leveraging and re-using a pre-trained model amplifies the importance of giving proper consideration to copyright issues associated with the data on which that model was trained. While our proposed model has the potential for generating deep-fakes or disinformation, these technologies also hold promise for positive applications, including coutivity-suppoor tools and design aids in engineering.
A Appendix for Section 3
betch fressed exang
batrh trAittig ccampler diatters of orgginal soampie
-06
-0A
20
3.0
20
12
Figure 5: A demonstration of the DiftAug technique using Toy 2D datasct.
15

---

diflosion process reverse difuuiion step individual training point dift,, sed training point. difiiug training points
0.
-02-
-0.
0.10
0.1
Figure 6: A zoomed-in view demonstrating the transfrmationn considering single train point.
Mastooi Intrus from Caror Augmentation
Diigraa
Figure 7: Example of Manifold Intrusion from Appendix C of Hendrycks et al. [24]. While DiffAug may alter class labels (Fig. 1), the denoised images are visually distinguishable from the original images allowing the model to also learn from noisy labels without inducing manifold intrusion. On the other hand, here is example manifold intrusion where the augmented image does contain any visual cues that enable the model to be robust to noisy labels.
B Appendix for Section
B.1 Training Details
In the following, describe the training details for the classifiers evaluated in Section 4. In general, we optimize sum of two losses: LToil Long + C where, Lonig is the classification objective on the original augmentation policy that we aim improve using DiffAug examples and L denotes the classification objective measured on DiffAug examples (Eq. (5)). Before applying DiffAug. we first resized the raw image such that least one of the edges is of size 256 and then use 224 x 224 center-crop as the test image since the diffusion model was not trained on random resized crops.
RN-50: We trained the model from scratch for 90 epochs using the same optimization hyperparameters used to train the official PyTorch RN-50 checkpoint.
ViT: We used the two-stage training recipe proposed in DeIT-III. In particular, the training recipe consists of 800-epoch supervised pretraining at lower resolution (e.g.. 192x 192) followed by 20-epoch finetuning at the target resolution (e.g., 224x224). Starting with the pretrained checkpoint (i.e., after 800 epochs), finetune the classifter exactly following the preseribed optimization and augmentation hyperparameters (e.g.. AutoAugment (AA) parameters and MixUp/CutMix parameters) except that we also consider DiffAug examples. We also included DiffAug examples when applying MixUp/CutMix since we observed significant drops in standard test accuracies when training directly DiffAug examples without subel-smoothing or MixUp. We briefly explored stacking DiffAug with AA and identified that this did not introduce any noticeable change compared to independent application of DiffAug and AA to train examples.
Aug To evaluate the combination DiffAug with these augmentations, we finetune the RN-50 checkpoint opensourced by the respective papers for 10 epochs with batch-size 256. We resume the training with the optimizer state made available along with the model weights and use a constant learning rate of 1e-6.
16

---

B.2 More Results on Covariate Shift Robustness
100
Figure 8: ImageNet-C Accuracy (Top-1) averaged across all severities. While t = 0 corresponds to the accuracy in default evaluation mode, other values of t correspond to the DiffAug-Ensennnn accuracy. See Fig. (1( for the corresponding graph on ImageNet-C (severity=5).
Table 6: ImageNet-C (severity=5) accuracy for each corruption type. Relative Improvements when additionally using diffusion denoised augmentations are computed with respect to the corresponding pretrained checkpoints and averaged across all corruption types. Overall, we observe improvements for each family of corruptions: in the default evaluation mode we observe an average absolute improvement of 2.5%, 4.9%, and 0.76% for the Noise, Blur, Weather and Digital corruptions respectively. Across all evaluation modes, we observe an average absolute improvement of 1.86%, 4.74%, 1.67%, 1.51% % for the Noise, Blur Weather and Digital corruptions respectively.
Noise
Blee
Westher
Diginl
Inference
Made:
Train
Gauss.
Shot
Impal.
Detoc.
Motion
Zocm
Snow
Frost
Bri.
Contr.
Elastic
Pisel
JPbCi
Imp-
Rel.
AM
50.64
52.30 33
50.8
18.18
25.2
18.76
34.29
22.67
33.04
5.35
40.56
11.12
40
$4.15
50.08
33.18
AM4DiTAAn
52
23.13
411.79
41.03
55.15
51.48
53:37
51.15
18.06
23.24
25:31
35,46
10.69
48.13
12.33
41.28
58.66
51.84
25.41
DA+DifAAgg
52:21
53.96
51.74
29.26
33.09
23.06
27.84
25.47
36.98
13.66
47.09
14.51
41.84
60.63
52.85
37.61
9:35
DDA
DAM+DilTAug
DAM
54.15
53.5
55
55.56
96
54.86
55.33
31.35
33.47
37.44
35.2
28.91
33.5
28.52
31.99
31.24
30.76
41.09
39.67
12.88
15.53
50.91
49.05
23.42
21.28
45.04
44.77
61.63
62.99
56
54.95
41,91
10,36
5.50
0
RN50
45.05
17.08
17.97
35.42
47.91
RNS04DMTAn
47.46
48.55
48.16
17.65
18.16
23.47
19:05
30.55
7.89
38.11
6.86
37.51
52.9
48.37
VTT-/16
54.6
55:75
54.79
32.65
20.41
33.19
30.38
36.25
42.5
21.91
51.9
28.7
48.55
64.01
5%.7
43.6
0
56.37
57.18
56.45
32.79
42.13
35.76
35.05
36.65
43.25
21.49
55.15
26.54
49.78
65.2
61.05
45,05
3.12
AM
49.59
51.23
50.07
20.61
23.03
24.43
32.79
25.76
36.25
20.05
56,14
14.16
39.15
54.62
52.24
36.54
AM-Di
51.41
52:34
51.79
24.45
27.3
20 83
36.97
26.
37.4
26.01
53.67
15.49
30.08
54 54.96
52.17
18.61
8.30
DA
53.36
54.71
$3.34
25.72
28.22
20.
36.37
30.57
40.11
28.
59.39
13.91
39.82
59.1
$2.36
39.06
DAIDITAA
53,92
53.99
DDA-SE
DAM+DilAAU
DAM
54.17
54.53
56.31
56.83
55.96
55.19
36.19
35.73
36.34
40.5
34.87
37.72
36.39
35.82
45,96
35.52
39.19
61.89
60.9
31.84
27.85
43.33
42.59
62.99
63:43
56.56
16,35
4.32
RN50
41.85
45.52
45.17
14.33
16.2
14:23
23.
20.54
30.4
19.56
51.67
6.61
33.2
46.45
46.56
30,62
0
47.34
48.45 48
48.11
21.26
21.49
21.54
28.00
20.98
31,96
20.98
51.72
7.26
34.02
51.05
48.06
13,51
14.01
WIT-/16
58
70,05
49.19
VIT-B//64DDAAA
59.11
59.55
59.07
37.2
40.35
43.26
41.35
52.68
55.53
70.36
45.03
45.23
66.54
64.9
53,54
32.32
35.1
33.01
21.02
31.16
25.57
32.64
25.83
38.74
16.89
54,76
7.17
42.98
54.85
58,89
AM+DifIAug
37.62
36.28
27.34
56.3
43.93
56.21
18.17
44.90
45.75
46.13
30.14
30.21
22.41
29.01
20
39.7
23,93
$7.59
4.5
43.9
57.15
60.94
17.08
DA
45.65
46.78
46
27.91
35.08
20.5
35.71
50.79
41.9
$2.60
59.93
10.17
43.71
58.56
61.64
40.42
19.42
DE
DAMAD
DAM
43.06
46.43
48.4
KM
47.34
48.52
25.15
37.9
32.41
35.55
34
44.27
28,
59,69
15:43
46.14
60.77
62.23
41,86
1004
RN50
26.63
28
27:23
11.26
16.26
24.85
20.3
49.09
36.51
45.63
53.13
27.12
RNSO+DifAAug
31.73
33:42
31.93
18.38
23.77
23.82
32.91
21.77
34.37
25.01
52.52
2.48
38.61
51.41
57.2
32,22
26 99
ViT-B/16
54.7
52:21
55. 16
29.2
38.02
36.58
36.30 26
51.95
$8.6
41.89
70.41
18.9
50.35
61.64
67.87
48.25
0
57.33
54-87
57.88
33.95
41.05
43.11
45.46
49.74
58.5 58
44.93
72.75
30.38
$1.27
67
69.81
51,87
10.84
AM
15:01
18:38
16.64
21.48
13.69
24.59
33.67
21.54
27.13
22.91
57.92
13.08
25.17
42.30
46.99
99
36.72
0
AMADITAAN
19.5
22.33
20.58
36.34
17.88
31.11
37.9
22.53
28.21
28
56.98
15.34
24.63
43.00
47.09
20.47
14.3
39.61
401
19.01
45.41
DAADINTAu
40.79
41.76
43.15
31.52
17.58
23.6
28.54
27.67
37.27
63.11
23.11
34.33
33,78
Def.
DAM+DifAAE
DAM
39.61
40.86
41.04
42:75
42.14
43
37.08
34.47
22.95
26.04
20.67
36.57
37.77
35.5%
34.04
35.17
39.85
40.57
41.24
38.75
63.95
64.25
31.13
25.6
29.02
23.8
57.06
56.44
50.51
50.88
39,52
11.24
5.1
RN50
5.69
6.49
6.45
15.04
8:23
13:29
22.85
15.59
20.44
22.22
55.64
4.23
14.31
23
34:55
[7.87
10.71
28.88
VIT-R/16
51.78
45.67
51.92
37.13
19.09
43.04
36.99
55.93
69.17
74.79
56.76
35.22
56.85
62.38
50.75
54.12
50.53
54.0
41.11
30.52
45.43
42.75
55 06
60.88
70.21
74.92
56.9
35.8
58.14
63.22
$2.78
6.64
17

---

Table Top-1 Accuracy (%) on Imagenet-S and Imagenet-R. summarize the results each combination of Train--augmentation and evaluation modes. The average (avg) accuracies for each classifier and evaluation mode is shown.
ImageNet-S
ImageNet-R
Train Augmentations
DDA (SF)
DDA (SF)
DDA
DE
Def.
Avg
DDA
DE
Def.
Avg
AM
11.53
13.74
15.73
11.17
13.04
36.39
42.02
42.22
41.03
40.42
AM+DifIAug
12.30
14.02
16.00
10.95
13.32
37.09
42.14
43.28
40.98
40.87
DA
12.18
15.69
17.16
13.84
14.72
37.82
43.11
43.50
42.24
41.67
DA+DifAAu
13.04
16.28
17.80
14.06
15,30
38.83
43.62
44.39
42.61
42.36
DAM
14.63
19.68
20.05
19.47
18.46
41.47
46,64
46.25
46.78
45.29
DAM+DifAAug
15.41
20.00
20.22
19.82
18.86
42.37
47.12
46.67
47.05
45.80
RN50
9.38
10.29
11.68
7.12
9.62
32.85
38.24
38.49
36.16
36.44
RNSO+DifTAug
10.25
10.63
12.52
6.99
10.10
34.76
39.65
41.61
37.55
38,39
ViT-B/I6
16.22
24.40
23,97
25.36
22.49
44.62
53.84
53,30
53.61
51.34
ViT-B//6-DiffAug
18.86
25.14
24.77
25.74
23.63
47.71
55.36
55.80
54.98
53.46
Avg
13.38
16.99
17.99
15.45
15.95
39.39
45.17
45.55
44.30
43.60
Arg (No-DifTAug)
12.79
16.76
17.72
15.39
15.66
38.63
44.77
44.75
43.96
43.03
Arg (DiffAug)
13.97
17.21
18.26
15.51
16.24
40.15
45.58
46.35
44.63
44.18
Table 8: Top-1 Accuracy (%) across different types of distribution shifts when additional high-quality synthetic data from SiihhlDDiffusin available (denoted by +Synth). For reference, we also include the results for the corresponding ResNet50 models without extra synthetic data.
Model
IisgeNes-Tist
TmageNNetCC (Severity-5)
ImageNet-RR
ImageNer-S
ImagseNet Skrich
ImageNet-A
ImageNes-D
Average
DE
Det.
D2
Def.
DE
Def.
DE
Det.
DE
Det.
DE
Det.
DE
Det.
DE
Def
Tooal
RN50
71.43
76.15
27.12
17.87
38.49
16.16
11:68
7.12
25.37
24.09
0.77
0.03
13.14
11.36
26
24.68
25
RNSIHDuLAug
75.07
75.95
32.22
20.87
37.55
26.67
11.37
28.65
25.30
26.97
Symhh
25.05
17.58
36.41
0.63
30.70
29,00
RNOO-Synl--DiAAng
75.01
54.71
50.85
33.22
31.58
Table 9: Top-1 Accuracy (%) on Imagenct-A A and Imagenet-D. We summarize the results for each combination of araienaugmenttiionn and evaluation modes. The average (avg) accuracies for each classifier and evaluation mode shown.
ImageNet-
ImageNet.D D
Augmentations
DE
Def
Avg
DE
Def
Avg
AM
3.00
3.67
3.34
12,43
11.16
11.80
AM-Dit'Ang
4.09
11.64
DA
3.09
3.39
3.24
13.19
11.2
12.20
DA-DiTTAg
3.45
3.52
13.03
12.05
DAM
3.15
3.84
3.50
14.64
12.11
13.38
DAM -iirAA
3.44
4.43
3.94
12.78
11.73
12.26
Synth
0.63
21.06
19:29
RN30-Symih-DirAAg
1.53
1.12
1.33
21
19.18
20.30
RN50
0.03
13.14
12.25
RN50-DiTAn
1.09
0.56
0.83
11.37
10.77
11:07
VIT
25.88
39.81
32.85
17.65
16.4
17.03
VIT-DifAAg
39.35
Avg
6.43
8.70
7.39
15.08
13.37
14.2T
Avg (No-DifTAug)
6.20
8.56
7.38
15.35
13.29
14.32
Avg (DiflAng)
6.76
8.85
7,80
14.75
13:45
14.10
Table 10: DDA vs DE in terms of wallclock times: We use 40GB A40 GPU for determining the running time. For each method, we determine the maximum usable batch-size and report the average wallclock time for processing a single example.
Method
Wallelock Time (6)
0.5
DDA
4.25
18

---

B.2.1 Extra Synthetic Data with ImageNet-only Diffusion models
In this section, we conduct preliminary analysis of extending training data with fully-synthesized examples generated with diffusion model trained only on ImageNet data. In particular, we generate 50k new examples using different techniques as shown in Table 11. From these experiments, we find that fully synthesized examples from ImageNet-only diffusion models do not enhance the robustness as compared to the synthetic examples from stable-diffusion. Overall, we find that DiffAug uses an ImageNet-only diffusion-mode to achieve robustness improvements although the sulte-synthesized examples from the same diffusion models do not enhance robustness.
Table 11: An evaluation of extra synthetic data with ImageNet-only diffusion models. We generate 50k new examples extend the ImageNet train-dataset and finetune the torchvision resnet-50 models for 10 epochs.
Diffasion Model
Guidance
ImageNet
Sampler
FID
1S
ImageNet-C
ImageNet-R
Def
DE
Def
DE
Classifer-Guidance
Improved DDPM
[Noisy Classifier]
DDPM
250
3.94
215.84
75.53
36.13
38.32
(Nichol Dhariwal (35))
Classifier Guiddacee
[Noisy Classifier]
DDIM
25
5.44
194.48
75.68
17.84
27.34
36.02
38.38
Classifie- Guidance
[DA Classifier)
DDIM
25
5.24
201.72
75.64
17.97
27.31
36.27
38.59
Classifier-Free Guidance
DIT-XL (Peebles and Xic (37)
DDPM
250
2.24
279.91
75.68
17.91
27.19
36.68
38.63
RN50
76.15
17.87
27.12
36. 16
38.49
RN50-DifAAg
B.2.2 DiffAug with Other Samplers
The single reverse-diffusion step used in DiffAug can be understood single reverse-diffusion step of the DDIM [46] sampler. Alternatively, we can consider improved diffusion samplers and this section, we explore J.M-oolver[33]. We illustrate the augmentations in Fig. 9.
Original Iraages Xo
DiñAAg DDIM
DiflAng DPM
Figure 9: An illustration of DiffAug with DDIM and DPM solvers: we show DiffAug augmentations at 500 for the examples in (a) applied using one reverse-diffusion step of the DDIM sampler (b) and the DPM-solver (c).
We evaluate single-step of reverse-diffusion of DPM-solver-2, both train-time as well at test-time. In particular, we first diffused the train example to random diffusion-tima t and used single reverse-diffusiee diffusion step of the order-2 DPM solver to integrate the diffusion ODE backwards from diffusion time to t with 0.01 instead of e = 0 for numerical stability. Based on the results shown Table 12, DDIM appears to the better sampling strategy DiffAug although resulting augmented images are somewhat of high visual quality when using DPM-Solver. While DiffAug using DPM-Solver may not beneficial for classification, there may be other applications of such augmentations and leave further exploration to future work.
19

---

Table 12: Evaluation other sampling methods for DiffAug: use to denote lower performance due to the use of DPM-Solver instead of DDIM.
Evaluation on ImageNet-C (severity=5)
Training Method
Dinseug-Ensemble DiflAuu-Ennembll (DPM-Solver-2)
Default
(DDIM)
AM AMHDifAAug/DDIM
26.72
34.08
31.774
29.47
38.58
35.564
AM-DDffAuggIDPM-Solvez2
22.964
29.694
25.16
B.3 Certified Accuracy Experiments
We follow DDS[5] and previous works on denoised smoothing and evaluate the certified accuracy on randomly selected subset of 1k Imagenet samples. The classifiers are generally evaluated with noise scales: d; E {0.25, 0.5, 1.0} and for each l2 radius and model pair, the noise that yields the best certified accuracy at that radius is selected and summarized in Table 13, following previous works. We also show the certified accuracy plots for each Gaussian perturbation separately Fig. 10.
Certified Accuracy (%) at la radius.
0.5
1.0
1.5
2.0
2.5
3.0
VIT
36.30
25.50
16.72
14.10
10.70
8.10
ViT+DiffAug
40.30
32.50
23.62
19.40
15.20
11.00
Table 13: Certified Accuracy for different la perturbation radius. As is standard in the literature, consider d; E {0.25, 0.5, 1.0) and select the best o, for each l2 radius.
Certified Accuracy
VT
Certified Accuracy
VIT
Certified Accuracy
VIT
0.8
0.8
0.8
DiffAug
DiffAug
DiffAug
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.5
1.0
2.0
2.5
3.0
0.0
0.5
1.0
2.0
2.5
3.0
0.5
1.0
2.5
3.0
Radius
Radius
1. Radlus
(a) = 0.25.
(b) o. 0.5.
(c) d. = 1.0.
Figure 10: le Radius vs Certified Accuracy for different values of or.
B.3.1 DiffAttack Adversarial Examples
We also explored robustness to DiffAttack[)] adversarial examples: these are unrestricted adversarial examples generated with stable-diffusion model. Here, we generate adversarial examples using their official code: in particular, we use the inception-và to generate the adversarial examples and evaluate the other classifiers. We tabulate the results in Table 14.
20

---

Table 14: DiffAttack Evaluation: We generate DiffAttack adversarial examples for ImageNet-comppaible dataset and evaluate the robustness enhancements introduced by DiffAug training and DiftAug-Ensemine inference. We consistently observe that DiffAttack succeeds in attacking all models albcit to different extents. In each of these cases, DiftAug- ttaiiinn followed by DE inference enhances the performance. Most notably, ViT+DiffAug/DE comes closest the original performance.
ImageNet-Compatible
Training Method
Original
Adversarial
DiffAug/DE Improvement
Def
Def
DE
DA
91.80
66.20
68.10
DA+DiffAug
91.70
66.40
72.40
6.20
AM
92.80
66.80
69.50
AM+DiffAug
92.50
67.20
73.20
6.40
DAM
92.70
71.50
72.80
DAM+DiffAug
91.70
70.60
74.60
3.10
ViT
95.70
78.70
77.80
ViT+DiffAug
96.40
80.80
87.20
8.50
RN50
92.70
61.10
62.10
RN50+DiffAug
91.90
60.20
71.00
9.90
RN50+Synth
92.20
58.60
58.90
RN50+Symth+DiffAuug
90.80
59.80
68.70
10.10
B.4 Detailed OOD Detection Results
OOD Detection results are mainly evaluated with with two metrics: AUROC and FPR@TPR95. The AUROC threshold-free evaluation of OOD detection while FPR @TR95 measures the false positive rate which OOD samples incorrectly identified in-distribution samples given that the true positive rate of detecting in-distribution samples correctly is 95%. The Near-OOD Imagenet task as defined by OpenOOD consists of SSB-Hard and NINCO datasets and also include the performance for each dataset in the following tables.
Maximum Entropy
6
VIT
Entropy (nats)
+ DifAAug
RN50
4
RN50 +1500+DifAAu
2
200
600
800
Figure 11: The entropy plots for ViT and RN50 are shown where we observe similar trends in Fig. 2.
21

---

Table 15: AUROC and FPR (lower is better) on ImageNet Near-OOD Detection.
Train
AUROC
FPR95
Augmentation
ASH
MSP
ReAct
SCALE
Avg.
ASH
MSP
ReAct
SCALE
Avg.
AM
82.16
77.49
79.94
83.61
80.8
59.14
64.45
62.82
57.2
60.9
AM+DiffAug
83.62
78.35
81.29
84.8
82.02
55.13
62.7
59.71
54.27
57.95
RN50
78.17
76.02
77.38
81.36
78.23
63.32
65.68
66.69
59.79
63.87
RNSO+DifAAug
79.86
76.86
78.76
82.8
79.57
60.21
64.91
62.84
56.15
61.03
DAM
74.16
75.2
75.14
77.07
75.39
66.34
67.42
67.72
63.67
66.29
DAM+DiffAug
75.73
75.65
75.87
78.56
76.45
64.99
66.56
66.28
61.94
64.94
DA
79.14
76.67
78.43
81.52
78.94
67.44
65.9
65.9
63.74
65.75
DA+DiffAug
79.54
76.92
79.1
81,42
79.25
66.52
65.41
64.25
63.19
64.84
Table 16: AUROC and FPR SSB-Hard Dataset of ImageNet Near-OOD Detection.
Train
AUROC
FPR95
Augmentation
ASH
MSP
ReAct
SCALE
Avg.
ASH
MSP
ReAct
SCALE
Avg.
AM
78.22
72.83
75.86
79.69
76.65
68.17
74.39
74.48
67.11
71.04
AM+DiffAug
80.48
73.81
77.2
81.7
78.3
63.31
72.88
72.43
63.25
67.97
RN50
72.89
72.09
73.03
77.34
73.84
73.66
74.49
77.55
67.72
73.35
RN5O+DifFAug
75.09
72.89
74.49
79.06
75.38
69.82
73.23
73.56
64.67
70.32
DAM
65.68
69.23
68.35
69.42
68.17
81.03
78.46
81.5
77.97
79.74
DAM+DiffAug
68.33
69.82
69.05
71.9
69.78
78.27
77.89
81.32
75.79
78.32
DA
76.65
72.35
75.28
78.59
75.72
72.26
75.27
75.27
73.3
DA+DiffAug
76.75
72.32
75.5
77.95
75.63
72.34
75.32
74.98
71.33
73.49
Table 17: AUROC and FPR on NINCO Dataset of ImageNet Near-OOD Detection.
Train
AUROC
FPR95
Augmentation
ASH
MSP
ReAct
SCALE
Avg.
ASH
MSP
ReAct
SCALEE
Avg.
AM
86.11
82.15
84.01
87.53
84.95
50.11
54.52
51.16
47.3
50.77
AM+DiffAug
86.75
82.88
85.39
87.91
85.73
46.95
52.52
46.98
45.3
47.94
RN50
83.45
79.95
81.73
85.37
82.62
52.97
56.88
55.82
51.86
54.38
RN5O+DifFAug
84.63
80.84
83.03
86.53
83.76
50.6
56.59
52.12
47.63
51.73
DAM
82.65
81.16
81.94
84.71
82.61
51.65
56.38
53.94
49.36
52.83
DAM+DiffAug
83.12
81.47
82.68
85.23
83.12
51.71
55.24
$1.23
48.1
51.57
DA
81.62
80.99
81.58
84.45
82.16
62.62
56.52
56.52
57.07
58.18
DA+DiffAug
82.32
81.52
82.7
84.9
82.86
60.71
55.49
53.52
55.06
56.2
Table 18: ImageNet Near-OOD Detection results with VIT using MSP. Here, observe comparable performance although we notice slight improvements in detection rates. We not show the results for other OOD al algorithms since we found those results to be significantly worse than simple MSP This may be because the OOD research is mainly focused on deep convolution architectures such as DenseNet and ResNet.
Train Augmentation
AUROC
FPR95
SSB-Ilard
NINCO
Avg.
SSB-Ilard
NINCO
Avg.
VIT
72.02
81.61 82.00
76,82
82.19
61.87
72,03
ViT+DiffAug
72.22
77.11
81.42
58.06
69.74
B.5 Ablation Experiments
We describe ablation experiments related to DiffAug and DiffAAg Ensemble in the following.
22

---

B.5.1 Extra Training
We train the pretrained AugMix checkpoint for extra 10 epochs to isolate the improvement obtained by DiffAug finetuning. In Table 19, we analyse the OOD detection performance as well as the robustness to covariate shift (Imagenet-C, Imagenet-R R and Imagenet-S) finding no notable difference as compared to the results in Tables 3, 7 and
B.5.2 DiffAug Hyperparameters
The time range used to generate the DiffAug train augmentations constitutes the key hyperparameter and we analyse and compare between weaker DiffAug augmentations (t E [0, 500 and stronger DiffAug augmentations E [500, 999). From Table 19, we find that both weak and strong DiffAug augmentations complement each other and contribute to different aspects of robustness. Overall, using the entire diffusion time range to generate DiffAug yields consistent improvements.
Table 19: Ablation Analysis.
(a) Top-1 (ccuracy(%) on ImageNet-C (severity=5) and ImageNet- We observe that extra AugMix training does not introduce any remarkable difference with respect to the pretrained checkpoint allowing us to clearly attribute the improved robustmess to DiffAug. Further, we also analyse the choice of diffusion time-range for generating the DiffAug augmentations and find that the stronger DiffAug augmentations generated with t [500, 999] enhances robustness Imagenet-C compared to DiffAug [0, 500) while obtaining slightly lower accuracy on Imagenet-Tess DE and DDA evaluation modes. Using the entire diffusion time-scale tends to achieve the right balance between both.
ImageNet-C: C (severity
Image Net-Test
Train Augmentations
DDA (SE)
DDA
DDA
Def.
Avg
DDA
Def.
Avg
(SE)
AM
38.18
36.54
34.08
26.72
32.63
62.23
75.98
73.8
77.53
72.39
AM-DiffAug
34.64
38.61
38.58
29.47
35.33
63.53
76.09
75.88
77.34
73:21
AM+Extra
33.22
36.83
34.48
27.14
32.92
61.96
75.98
73.74
77.57
72.31
AM-DiffAug(O.55
34.05
36.96
36.15
26.93
33.52
63.94
76.09
76.15
77.25
73.36
AM-DiffAug(5000999
34.53
39.04
38.61
30.29
35.62
62.67
76.05
74.74
77.38
72.71
(b) Top-1 Accuracy(%) ImageNet-R (severity=5) and ImageNet-S. As above, we find that extra Augmix training does not introduce any significant change. We also observe that DiffAug generated with [0,500]
contributes more to improve robustness to Imagenet-R and Imagenet-S highlighting the benefits of using the entire diffusion time range for generating augmentations.
ImageNet-R
ImageNet-S
Train
DDDA
DDAA
DDA
Def.
Avg
DDA
Def.
Avg
Augmentations
(SE)
(SE)
AM
36.39
42.02
42.22
41.03
40.42
11.53
13.74
15.73
11.17
13.04
AM-DiffAug
37.09
42.14
43.28
40.98
40.87
12.30
14.02
16.00
10.95
13.32
AM+Extra
36.05
41.66
41.84
40.70
40.06
11.40
13.56
15.46
10.91
12,83
AM+DifAAug00500
37.61
42.36
41.21
12.55
15.79
10.77
AM+DiffAug(500.999
35.94
40.91
41.51
40.11
39.62
11.49
13.29
15.48
10.62
12.72
(c) OOD Detection. As above, we find that extra Augmix training does not introduce any significant change. Here, we find that DiffAug(500,999) contributes more to the improved OOD detection.
TrainAngmentution
ASH
MSP
ReAct
Scale
Avg.
AM
$9.14
64.45
62.82
$7.2
60.9
AM+DiffAug
55.13
62.7
59.71
54.27
57.95
AM+Extr
58.17
64.30
62.77
56.57
60.45
AM+DiITAug(0.500
56.63
63.39
61.42
54.83
$9.07
AM+DifAAug(001.950
23

---

B.5.3 DiffAug Ensemble (DE) Hyperparameters
We use set S = (0.50.... 450) to compute the DE accuracy in Table 9. Here, we study the effect of step-size (the difference between consecutive times) and the maximum diffusion time used. First, we analyse the performance when using maximum diffusion time 999 instead of / 450 in Fig. 13. Then, using maximum diffusion time = 450, we study the effect of using alternative step-sizes of 25 (more augmentations) and 75 (fewer augmentations) in Fig. 12.
DA
AM
DAM
VIT-15166
HNS0
0.42
0.36
0.325
0.50
0.25
75
0.300
0.40
75
0.20
75
0.276
0.68
0.30
0.35
0.35
75
70
0.42
0.020
75
0.20
200
100
200
400
200
400
200
300
200
400
DA+DIfAA ug
AM- DiffAAg
DiEA-uiAAug
VIT-B/10-DDiAAuu
RN60+DDifAAug
Figure 12: Plots of t DE Accuracy Imagenet-C (severity=5) for different step-sizes: in general, we observe that the performance is largely robust to the choice of step-size although using = 25 gives slightly improved result.
VIT-8/16
RN50
DA DDIIAAg
AM Dirtuu
DAM + DiffAug
D
RN5O+DIITAug
0.50
0.40
0.35
025
400
800
200
600
1000
a)
Accuracy vs
b) DiffAug Accuracy vs
Figure 13: Plots of t vs DE Accuracy and DiffAug Accuracy on Imagenet-C (severity=5): in general, observe that the performance saturates beyond certain time-step although the corresponding DiffAug accuracy steadily decreases. These plots also highlight the robustness of straightforward averaging a test-time augmentation method.
B.5.4 Conditional DiffAug
In this section, study the application of DiffAug with conditioning. For these experiments, utilize two one clnsscconiiii and one unconditional - ImageNet256 diffusion models to apply the DiffAug augmentation. If sø, (xly) and se, (x) refer to the conditional and unconditional scores, we can combine these using an hyperparameter (similar to classifier free guidance) and use the following formulation of sq to apply the DiffAug augmentation (Eq. (3)):
sx(x,t) se, (x, t) A(se, (xly, t) se, (x, t))
(9)
24

---

where, À = 0 denotes unconditional DiffAug, explored in the main paper. In the following, we illustrate conditional DiffAug both qualitatively and quantitatively.
a) Original Imagge K
DiffAnglA m
DifiAnglA 0.2
d) DiffAnglA m0.)
DuffAngl A (77
0 DiffAnglA 1.0
e) DiffAnglA= = 2d)
h) DifAAggA = 3.d)
1) Dif'Agg 4.)
Figure 14: We illustrate the DiffAug augmentations for various values of À at 600.
Delonit
DE
AM/ Defoutt
AM/DE
3ocunnc
34
32
30
28
Figure 15:
We extend AugMix(AM) with DiffAAg using different values plot the ImageNet-C (severity=5) accuracy for both default and DE inference. We observe that conditional DiffAug can enhance performance for optimal values of A. Nevertheless, DiffAug can also be applied with unconditional diffusion models broadening its applications.
25

---

B.5.5 DiffAug with DiT
In this section, we explore the application of DiffAug with latent-space diffusion models such as the Diflusion-Transfrmmm (DiT). TranssoeonTransformmrr models the distribution of variational autoencoder (VAE) latent-space. To apply DiffAug with DiT, we (i) first apply the VAE encoder to encode the train image into the latent-space, (ii) apply DiffAug within the latent-space, and (iii) finally, decode the image using the VAE decoder. We refer to this as DiT-DiffAug. We finetune the torchvision ResNet-50 model for epochs using conditiona and unconditional DiT-DiffAug augmentation and summarize the results in Table 20.
Table 20: In this table, we explore DiffAug augmentation with DīT, latent-space diffusion model. While the test-accuracy preserved even when training upon partially synthesized DiT images, we observe a smaller robustness improvement as compared to the DiffAug with pixel-space models.
Method
ImageNet- Test
ImageNet-C
RN50
76.15
17.87
+DiffAug
75.95
20.87
+DiT-DifAAug
76.08
18.91
+DiT-DiffAug [Conditional]
76.09
19.27
B.6 Derivation of Theorem 4.1
For the forward-diffusion SDEs considered in this paper, the marginal distribution pe(x) can be expressed in terms of the data distribution p(xo):
Pt(x) = / )xx
(10)
where
(x(xx) = Nx(x ((Xo, t), (t)I).
we denote p(xg,t) f) by m., we can rewrite Pt (x) as
(x(x) /
pt (x mm ) )dm;
since y is linear and invertible, The optimal score-function sp. (x, 1) V log can simplified as:
(x(x) S.
m, 0²(t) pe (x(m:) p(m.)dm,
S0x (x,
(11)
Using Eq (11), we can rewrite the denoised example, x; = x (2)soo( 1), as:
x; 1
m; Pt (x( pmm))dm; / m, Pe (e(mm/))dm Emm[x}]
(12)
(x) m.
m, That is, the denoised example &; fact the expected value of the mean m, given input x. (See also Eq (4) in the main text).
26

---

To compute
we algebraically simplity Sm. m. Vx
p(me)dm, as follows:
8x
(x)
0x
VxPi (x/mm) Pr(x m2) Vx VxPe
δx
m
(x(
xx)
p(mm)dmm
m
m1
Pe(x m))
m. -x
Pi (x Vx log (c(x)) T
p(m,)dm;
Pt(x)
x2(1)
Px(x)
m.
(x()
m, -x
Vx Vxlogp((x) T
m1.
p(m:)ddm
pr(x)
(2(1)
m.
Pe(x(mm))
m. X ( log (x) T
m
p(m: )dm;
(x(x)
(²(
m.
m.
Pe(x((me)
m.
T x x+ x log
p(m,)dm:
(13)
(x(
(2
(²(1)
mm
m,m,
(
x
Pt (m(m(())dm)
m, (e (
(t)
o²(t)
m.m. Pr (m, x)dm, {
(t)
m.
(E[m,m? [x] [
(2
Cov[me/x
0²(()
B.7 Analysis with SVD Decomposition
Figure 16: CIFARIO examples used for SVD Analysis
Considering the CIFARIO images in Fig. 16, we compute the 3072x3072 jacobian matrix J and then apply SVD decomposition J USV using default settings in PyTorch. Then, we visualise after min-max normalization the columns of U and rows of V along with the (batch-averaged) value in the diagonal matrix S of the corresponding row/column. We find that the principal components of the jacobian matrix are perceptually aligned and provides additional intuition for Theorem 4.1. See Figs. 17 and 18.
C Appendix for Section 5
C.1 Guidance Classifier Training: Experiment Details
We use the pretrained noisy Imagenet classifier released by Dhariwal and Nichol [14] while we trained the noisy CIFARI0 classifier ourselves; the Imagenet classifier is the downsampling half of the UNET with attention pooling classifier-head while we use 2ideResNee-28-2 as the architecture for CIFARIO. For the DA-classifier, simply add extra convolution that can process the
27

---

a) Average Eigenvalue; 2.40 Component ID:
b) Average Eigenvalue: 1.11
c) Average Eigenvalue: 0.81
Component ID: 10
Component ID: 35
Average Eigenvalue: 0.51 Component ID:
Average Eigenvalue: 0.29
Eigenvalue; 0.17 Component 200
Component ID: 130
g) Average Eigenvalue: 0.10 Component ID: 285
h) Average Eigenvalue: 0.06
1) Average Eigenvalue: 0.04 Component ID: 500
Component ID: 385
Average Eigenvalue: 0.03 Component ID:
k) Average Eigenvalue: 0.02 Component ID: 775
Average Eigenvalue: 0.01 Component 935
m) Average Eigenvalue: 0.01 Component ID: 1110
n) Average Eigenvalue: 0.01 Component ID: 1300 28 Figure 17: Columns of U
a) Average Eigenvalue: 0.01 Component ID: 1505

---

a) Average Eigenvalue; 2.40 Component ID:
b) Average Eigenvalue: 1.11
c) Average Eigenvalue: 0.81
Component ID: 10
Component ID: 35
Average Eigenvalue: 0.51 Component ID:
Average Eigenvalue: 0.29
Eigenvalue; 0.17 Component 200
Component ID: 130
g) Average Eigenvalue: 0.10 Component ID: 285
h) Average Eigenvalue: 0.06
1) Average Eigenvalue: 0.04 Component ID: 500
Component ID: 385
Average Eigenvalue: 0.03 Component ID:
k) Average Eigenvalue: 0.02 Component ID: 775
Average Eigenvalue: 0.01 Component 935
m) Average Eigenvalue: 0.01 Component ID: 1110
n) Average Eigenvalue: 0.01 Component ID: 1300 29 Figure 18: Rows of V
a) Average Eigenvalue: 0.01 Component ID: 1505

---

Noisy Classitier Denolding Augmented Classitier
0.2
0.4
04
02
Dencllnng Augmented Cassofeer Cassiferr
Chssifer
10
10
10"
18
haise Level
Figure 19: CIFARIO: Test Accuracy vs. Noise Scale,
denoised input: for Imagenet, we finetune the pretrained noisy classifier by adding additional input-convolution module while we train the senoising-augmente CIFARIO classifier from scratch.
The details of the optimization along with the new convolution module (initialized with very small weights) using Adam W optimizer with learning-rate of 1e-5 and weight-decay of 0.05 for 50k steps with batch size of 128. (2) For CIFAR10, we train both noisy and DA-classifiers for 150k steps with a batch size 512 using Adam W optimizer with learning-rate of 3e-4 and weight decay of 0.05. For CIFAR 10 classifiers, we use the Exponential Moving Average of the parameters with decay-rate equal to 0.999.
follows: (1) for Imagenet, fine-tune the entire network
C.2 Classifier-Guided Diffusion: Sampling
We use PC sampler described in Song et al. [47] with 1000 discretization steps for CIFARI0 samples while we use a DDIM [46] sampler with 25 discretization steps for Imagenet samples. We the 256x256 class-conditiona diffusion model open-sourced by Dhariwal and Nichol [13] for our Imagenet experiments and set the classifier scale A = 2.5 following their experimental setup for DDIM-25 samples. The classifier-scale is set 1.0 for CIFAR10 100 experiments.
C.3 Comparisons with DLSM, ECT and Robust Guidance
Table 21: DLSM vs DA-Classifier; In this table, we compare between using DLSM - i.c., DLSM-LosS in addition cross-entropy loss in training classifiers on noisy images as input and DA-Classifiers wherein we use both noisy and denoised images as input but only used cross-entropy loss for training. Since Chao al. [6] use ResNet18 backbones for their CIFAR10 experiments, we train a separate DA- Classifier for these comparisons. We compare between FID. IS and also compare the unconditional Retrassioneeaaaleeeesst/CCerreee
precision and recall (P/R) and the average class-conditionss
We obtain our results for Noisy Classifier (CE) and Noisy Classifier (DLSM) from Table 2 of Chao al. [6]. While the FID and IS scores are comparable, we note that class- wise Precision, Recall, Density and Coverage metrics are either comparable or demonstrate significant improvement.
Method
FID4
IS t
P1
R
P1
R1
D 1
C
Noisy Classifier (CE)
4.10
9.08
0.67
0.61
0.51
0.59
0.63
0.60
Noisy Classifier (DLSM)
2.25
9.90
0.65
0.62
0.56
0.61
0.76
0.71
DA-Classifier
2.27
9.91
0.64
0.62
0.63
0.64
0.90
0.77
30

---

Min-max normalized gradients samples diffused f 300 (T = 999). Top panel: gradients obtained with noisy classifier. Bottom panel: gradients obtained with DA-classifier. Middle panel: corresponding clean Imagenet samples. We recommend zooming in see differences between gradients, e.g. the clearer coherence in DA-classifier gradients.
Figure 20:
Table 22: ED and Robust-Guiie Sabust-GGuidanee vs DA-Classifier: Zheng et [58] propose two complementary techniques to improve over vanilla classifier-guidance Entropy-Driven Sampling (EDS). ECT consists adding an additional loss term to the cross-entropy loss encouraging the predictions to be closer to uniform distribution (similar to the label-smoothing loss). EDS modifies the sampling to use diffusion-time dependent scaling factor designed to address premature vanishing guidance-gradients The sampling method (EDS/Vanilla) can be chosen independent of the training method (determined by the loss-objective and classifieriinputs) In the following, compare between ECT and DA-Clan DA-Classifiers using Vanilla Sampling method using the results Table 3 of Zheng et al. [58]. As the robust-classifier [29] was not evaluated for Imagenet-256, fine-tuned the open-source checkpoint using the open-source code provided by robust-guice abbust-guidance for 50k steps with learning rate=le-5 We observe that DA Classifier obtains better FID/IS than both
Entropy-Constraint Training (ECT) and
ECT and Gobust-Guidance.
Method
Loss-Objectio
Classifier-Inputs
FID
sFID
IS
P
R
Noisy-Classifier ECT-Classifier
CE
Noisy Image
5.46
5.32
194.48
0.81
0.49
CE+ECT CE + Adv. Training
Noisy Image
5.34
5.3
196.8
0.81
0.49
Robust-Classifier
Noisy Image
5.44
5.81
142.61
0.74
0.56
Noisy Image &
DA-Classifer
CE
5.24
5.37
201.72
0.81
0.49
Denoised Image
31

---

d log P
d log P
à logp&
d logp
0& θx
dx
dx
0%
0%
Figure 21: The figure shows the total derivative alog (1)
Blog p 6x with respect noisy input @ log the partial derivative with respect denoised input
Glog Pa 8k, the partial derivative
doc
Ok. 8x
Po
and
log
Figure 22: Min-max normalized gradients samples diffused to = 0.35 (T 1.0). Left panel: gradients obtained with noisy classifier. Right panel: gradients obtained with the DA-classifier. Middle panel: clean corresponding CIFARIO samples.
32

---

Airplanes
cj Birds
d) Cats
e) Deer
fi Dogs
g) Frogs
Horses
Ships
Trucks
Figure 23: Uncurated CIFAR10 Samples with Noisy- cisys-Caassifier.
D
Uncurated Samples
33

---

Airplanes
b) Cars
c) Birds
d) Cats
Deer
f) Dogs
Frogs
Horses
Ships
Trucks
Figure 24: Uncurated CIFARIO Samples with DA-classifier.
34

---

N
Figure 25: Uncurated generated samples (with images containing human faces removed) to compare between Noisy classifier (left) and DA-Classifier (right). Please zoom in the subtle improve- ments introduced by DA-Classifier guidance.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please find the contributions listed in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please find the discussion of limitations in Part
35

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We state Theorem 4.1 in the main paper and include proof in Appendix B.6.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided preliminary version of the code as part of this submission. We will open-source the trained checkpoints to enable easy adoption of our method and classifiers in future research.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We use popular open-source datasets for all experiments in this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /ttpps.ccppuiiiiie httpccccccppppceeeeeeeeooppppeee /ooeee/e/ooooeeooeeeeo wwweesuumissssionoo sesisssssssiiiiissie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
37

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use the standard data splits and demonstrate the robustness to hyperparam- eter choice in our ablation study (Appendix B.5).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It is computationally expensive and do not compute error bars. However, we highlight the significance our improvement by providing appropriate context, where necessary, to interpret our improvements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss compute resources and approximate execution time in Part
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics fttts://neurr/ wwtparriis.ccpppppppeeeee Ceabblic/Etiistttiee
Answer:
[Yes]
Justification:
We ensured conformance to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include short description in Part
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
39

---

Justification: We only release the classifier weights but it can be used for generation in conjunction with the diffusion model, open-sourced by OpenAI.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we have ensured that we give the due credit to code and models of other researchers that we use in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi whrrdee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide appropriate documentation for use of our code and ensure reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
40

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Does not include research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Does not include research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "sfvUjzMG6d", "reviewer": "Reviewer_uEtP", "summary": "This paper proposes a data augmentation method DiffAug for training robust classifiers. The method is very simple: first, add Gaussian noise to one training image, and then denoise the noisy images with a pre-trained diffusion model. They also propose methods for test-time augmentation using DiffAug. Experiments show that DiffAug is able to improve robust accuracy on a wide range of datasets such as ImageNet-C, R,S, OOD detection, and certification accuracy. This paper also shows that the proposed method can improve the generation quality of classifier-guided diffusion generation.", "questions": "1. I do not pretty understand the sentence in line 128: what does \"additional optimization objective\" refer to?\n\n```\nWhen combining DiffAug with such novel augmentation techniques, we simply include Eq. 5 as an additional optimization objective instead of stacking augmentations\n```\n\nand line 131: why stacking augmentations are not good and what is the difference between stacking augmentations the the proposed method?\n```\nour preliminary analysis on stacking augmentations showed limited gains over simply training  the network to classify independently augmented samples likely because training on independent  augmentations implicitly generalizes to stacked augmentations.\n```\n\n2. Why use unconditional diffusion models? Will using conditional diffusion models raise to some problem?\n\n3. Can authors provide some insights why the proposed method can improve both corruption robustness  and certificated adversarial robustness? Will this method also improve empirical adversarial robustness?", "limitations": "Yes, authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To the best of my knowledge, this method is new. Most of the existing methods use diffusion models to generate new training examples instead of augmenting existing ones. The proposed method is simple and easy to follow and implement. \n\n2. This paper provides extensive ablation experiments to demonstrate the effectiveness of the proposed method. The proposed method is able to achieve significant improvements even combined with SoTA augmentation for robustness, say AugMix and DeepAugment, which indicates the proposed method provides new information for the model to learn. DiffAug can also be used to improve the performance of guidance classifiers in diffusion generation, thus lead to better results.\n\n3. Models trained with this method have perceptually aligned gradients and this paper provides nice figures to demonstrate this.", "weaknesses": "1. **Missing related work and comparison**\n\nComparison with existing work that uses large synthetic data datasets with diffusion models to improve training is lacking. For example [1] uses diffusion models to generate extra training examples to improve empirical adversarial robustness and [2] for certificated robustness. Both two works show significant improvements for robustness.\n\nSpecifically, suppose DiffAug train the model with E epochs on a dataset of N images. SoTA diffusion models can generate new images with K steps (K can be small (K<= 10) using DDIM, even 1 step using some flow based methods). If we use the diffusion model to generate EN/K new images, and train the model on the original dataset with the generated images. The extra computational cost is the same as DiffAug, if I understand it correctly. Will DiffAug achieve better results over this method? If yes, what could be the motivation/theory?\n\n\n2. **Concerns regarding some numbers reported**\n\nIn the DeepAugment paper, they report results for ResNet50 on ImageNet-C ([3], Figure 5). The accuracy of DeepAugment+AugMix (DAM in this paper) is close to 60%, which is much higher than the numbers in this paper (around 40%). Similarly, the results of AugMix reported in the original paper are also much higher than the numbers in this paper. (I do not work in the corruption robustness field, so there may be errors in citing these numbers)\n\nCan the authors explain why the gap is so large? If it is due to different settings. I suggest following the original settings for a fair comparison.\n\nSimilarly, the results on Table 8 are also significantly higher than numbers in [4], and far from SoTA.\n\n\n\n\n\n[1] Better Diffusion Models Further Improve Adversarial Training. ICML23\n\n[2] A Recipe for Improved Certifiable Robustness. ICLR 2024\n\n[3] The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization\n\n[4] (CERTIFIED!!) ADVERSARIAL ROBUSTNESS FOR FREE!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful review with suggestions to strengthen the contribution. We thank you for your appreciation of our method’s novelty, simplicity and computational efficiency, our presentation/figures and extensive ablation experiments. In the following responses, we aim to address the weaknesses and answer your questions. \n\n> _**[W1-A] Comparison [...] is lacking.**_\n\nWe address your suggestion for a comparison in the experiment [E1] described in the global response and include the results in Table 1 of the global PDF. In summary, we find that DiffAug training and DiffAug-Ensemble (DE) inference offer performance improvements _over and beyond_ additional high-quality synthetic datasets. \n***\n> _**[W1-B] Missing Related Work.**_\n\nWe thank you for bringing these papers to our attention, and we will certainly include discussion in the final version. We identify important distinctions and similarities between our work and the suggested papers. While the mentioned works focus exclusively on adversarial robustness, we focus on regular non-adversarial classifier training with an aim to improve different aspects of robustness (e.g., covariate shifts, OOD detection and certified adversarial accuracy). Nevertheless, both example papers demonstrate use of synthetic data generated with diffusion models trained with no extra data to improve the adversarial training technique and this is in fact very close to one of our research objectives: exploring the augmentation potential of diffusion models trained with no extra data. On the contrary, previous generative data augmentation approaches for regular non-adversarial classifier training often rely upon synthetic datasets from large diffusion models such as Stable-Diffusion or Imagen. \n\n***\n\n> **_[W1-C] Specifically, [...] theory?_**\n\nThank you for this interesting question about performance and sample quality vs compute budget! We provide a detailed answers to this in the global response (please see answers to Q1 and Q2). In the context of adversarial training, we imagine that corresponding enhancements in performance are likely possible using DiffAug. Additional exploration on adapting DiffAug to adversarial training and empirical validation is out of scope, but is an interesting avenue for future work.\n\n*** \n\n**[W2]** We appreciate the reviewer’s dedication to provide a well-informed review of this submission! We use official checkpoints and confirm our reproduction is error-free; we will release both the source-code and model checkpoints for reproducibility. \n\nThe DeepAugment+AugMix result in the original paper was obtained by considering all 5 severities of ImageNet-C, while Table 1 is for severity=5 (please see Fig. 1 in the global PDF for results over entire ImageNet-C). Our reproduction of the ImageNet-test and ImageNet-R accuracies of these models do also match the official results. Since the ImageNet-C dataset contains 750k test samples for each severity level, considering all 5 severities is computationally expensive for some methods (e.g., DDA) and in those cases, the standard practice is to evaluate on severity=5 since this represents the most challenging examples. \n\nAs we have described in lines 259-267 and as correctly identified by this reviewer, DDS already achieves state-of-the-art certified robustness results by using a pretrained 305M-parameter BeIT-L network. It is intuitive that training a classifier with DiffAug enhances the DDS certified accuracy and we empirically confirm this in Table 8 for a 86.6M parameter ViT-B model since this is computationally feasible.\n\n***\n**[Q1]** Additional optimization objective refers to extra loss term. When extending previous methods with DiffAug, we modify their official code by simply adding $\\mathcal{L}$ in Eq. 5 to the original loss-terms. For example, AugMix involves 2 loss-terms: a cross-entropy loss term and a Jensen-Shannon divergence loss term. When combining it with DiffAug, we simply introduce Eq.5 as the third loss-term. \n\nStacking refers to the sequential application of distinct augmentations on the same image. While stacking augmentations may intuitively offer more robustness enhancements, our preliminary analysis showed no advantage of stacking augmentations over simply including the DiffAug loss in addition to the original training loss. Therefore, we choose to implement DiffAug as an additional loss term since this requires minimal change to the original code and fewer design choices (e.g., order of augmentations).\n***\n\n**[Q2]** Similar to many standard augmentation techniques, we do not rely upon labels for DiffAug broadening its applicability -- e.g., DiffAug can also be applied at test-time when labels are unknown. In theory, DiffAug with conditional diffusion models should not cause any problem. However, this introduces additional hyperparameters such as the guidance-strength and choice of prompt. Additionally, it requires two forward passes through the diffusion model (one unconditional and one conditional) per training step. If the guidance-strength is very high, the classifier could cheat by exploiting imperceptible image statistics (e.g., due to adaptive layer-norms). This is an interesting extension for future exploration.\n\n***\n**[Q3]** In summary, DiffAug generates augmentations of varying sample-qualities, each presenting a different level of challenge for the classifier. Intuitively, this produces a regularisation effect that helps improve various aspects of classifier robustness including covariate-shift. Since both DiffAug and DDS utilise a single reverse-diffusion step, we can observe improvements in certified adversarial accuracy. Yes, we can also improve empirical adversarial robustness against $l_2$ perturbations by using a majority vote following the Algorithm 2 in [1].\n\n[1] (CERTIFIED!!) ADVERSARIAL ROBUSTNESS FOR FREE!\n***\nWe hope we have addressed all your concerns and look forward to discussing any outstanding concerns in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5U6S0sXkXe", "reviewer": "Reviewer_gb3b", "summary": "The paper explores the use of diffusion as a data augmentation technique to train robust classifiers. Specifically, it investigates whether a diffusion model trained without additional data can be leveraged to enhance classifier performance. The study shows utilizing a diffusion model trained without extra data to improve classifier robustness on Imagenet-C by 2 points, while train improved classifiers with just a single step of reverse diffusion.\nThe proposed method is evaluated on challenging benchmarks such as Imagenet-C and Imagine, utilizing Vision Transformer (ViT) and ResNet models.", "questions": "How do the hyper parameters of the one-step diffusion set? If you add more noise, you may change globally, if only local noise, you can only handle local noise corruption.\n\nCan the author show an ablation study on this tradeoff if there is any?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality: The paper proposes an effective method of one-step diffusion augmentation for training robust image classifiers. While similar augmentation techniques using generative models have been explored previously, the utilization of a single-step diffusion process in this context is interesting. The idea is straightforward yet innovative, offering a novel approach to enhancing classifier robustness with minimal computational overhead.\n\n2. Clarity: the paper is clearly written.\n\n3. Quality: the paper includes ablation study and visualization. Say on diffusion time.", "weaknesses": "1. Soundness of results: the robustness is only evaluated on Imagenet-C (severity 5 only), which is only local corruption. Does the proposed method work when viewpoint, background, scale, style, texture change? What about the other severity?  Say can test on Imagenet-A [1], ImageNet-D [2], Imagenet-S [3], ImageNet-R [4]. Since it is diffusion-based augmentation, the reviewer wondered if it will help ImageNet-D the most since this one is diffusion-based testing. Evaluation on those will help understand the strengths and weakness of the proposed approach.\n\n2. Not clear how useful is test-time augmentation, can add the ablation study.\n\n3. Unclear why the proposed method can handle covariate shifts. Can the author explain intuitively?\n\nThe score will be updated based on whether evaluation concerns are addressed.\n\n[1] https://openaccess.thecvf.com/content/CVPR2021/html/Hendrycks_Natural_Adversarial_Examples_CVPR_2021_paper.html\n[2] https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_ImageNet-D_Benchmarking_Neural_Network_Robustness_on_Diffusion_Synthetic_Object_CVPR_2024_paper.html\n[3] https://arxiv.org/abs/1905.13549\n[4] https://openaccess.thecvf.com/content/ICCV2021/html/Hendrycks_The_Many_Faces_of_Robustness_A_Critical_Analysis_of_Out-of-Distribution_ICCV_2021_paper.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the detailed review with suggestions to strengthen the contribution. We thank you for  recognizing the strengths of our method’s novelty, simplicity and computational efficiency. We are happy to see that you found our paper clearly written with nice visualisations. In the following responses, we aim to address the weaknesses and answer your questions. \n\n> ***Evaluation on other severity***\n\nFollowing this reviewer’s suggestion, we evaluate the models on all severity levels and include the results in the global PDF. We notice similar overall trends as those that we observed for severity=5. On average, we find that the DiffAug training and DiffAug-Ensemble inference leads to improved performances. We will include this in the final version of the paper.\n\n***\n> **_Evaluation on Imagenet-A [1], ImageNet-D [2], Imagenet-S [3] and ImageNet-R [4]._**\n\nAs per this reviewer’s suggestion, we carried out evaluations on ImageNet-A and ImageNet-D. For this evaluation, we also considered the additional classifiers we trained during the rebuttal week: in particular, we trained classifiers using synthetic datasets generated with stable-diffusion (please refer to the global response for more details) and denote this as RN50+Synth. We use RN50+Synth+DiffAug when trained using DiffAug in addition to the synthetic data. We note that ImageNet-R and ImageNet-S results are already included in the Appendix (Table 6) – we will label the paragraph describing these results on page 5 (line 220) to ensure that these results are easily accessible. We summarize our findings as follows:\n* DiffAug training introduces slight improvements in the default evaluation mode (i.e., directly evaluating on test examples) for these datasets while DiffAug-Ensemble (DE) inference introduces notable improvements for Imagenet-R, ImageNet-D and ImageNet-S. \n* On ImageNet-S, the average performance improves from 15.45 (using default evaluation) to 17.99 (using DE). For DiffAug-trained models, the average performance improves from 15.51 (default) to 18.26 (DE). In particular, DiffAug training and DE inference improves the RN50 performance from 7.12 to 12.52. \n* On ImageNet-R, we find similar observations as above. For example, DiffAug training and DE inference improves the accuracy of RN50 from 36.16 to 41.61. On average, DE enhances the performance of DiffAug-trained models from 44.63 to 46.35. The best performance on ImageNet-R is obtained when using additional synthetic data: RN50+Synth has an accuracy of 49.28. Using both DiffAug training and DE inference, we can improve the performance to 54.71. \n* On ImageNet-D, we observe that extra synthetic data from stable-diffusion helps enhance the robustness most as you had predicted. Interestingly, DiffAug helps to further improve robustness in this case. For example, RN50+Synth obtains an accuracy of 17.52 on ImageNet-D. Using DiffAug training in addition to extra synthetic data improves the performance to 19.18. When using DE-evaluation, we observe further improvements up to 21.41. \n* On ImageNet-A, we generally find that all models except ViT struggle on this task. In this case, we find that DE inference leads to a reduction in performance on this task (on average). DiffAug training neither improves nor negatively affects the model performance in this case although we find slight improvements in many cases. \n\nIn summary, we find that DiffAug training improves classifier robustness in general and this can be further enhanced using DiffAug-Ensemble (DE) inference. \n***\n> **_[W2] Not clear how useful is test-time augmentation, can add the ablation study._**\n\n[A] This is a good question, and we acknowledge that while we included this ablation in appendix B.6, the reviewers are not expected to look at appendix materials. We will gladly add a comment in the main body of the paper that highlights and refers the reader to these results.\n\nTo summarize these ablation results here: we find in multiple experiments that the DiffAug-Ensemble (DE) inference technique does indeed improve robustness. DE has two hyperparameters: maximum diffusion time-step and step-size controlling the number of augmentations. Overall, we also find that DE is robust to hyperparameter choices.\n***\n> _**[W3] Unclear why the proposed method can handle covariate shifts. Can the author explain intuitively?**_\n\n[A] We include a detailed answer in the global response providing an intuitive explanation of DiffAug's performance (in Q2). In summary, DiffAug generates augmentations of varying sample-qualities, each presenting a different level of challenge for the classifier. Intuitively, this produces a regularisation effect that helps improve various aspects of classifier robustness including covariate-shift. Notably, the complexity introduced by DiffAug does not lower the test accuracy despite the explicit use of low sample-quality images for classifier training. \n***\n> _**[Q1] How do the hyper parameters [...] tradeoff if there is any?**_\n\n[A] This is a very good question, and the reviewer is correct that smaller diffusion times produce DiffAug examples with localised modifications whereas larger diffusion times produce global modifications. In all our experiments, we use the entire diffusion time range to generate the DiffAug examples. To understand the influence of diffusion time-range, we conducted an ablation study (see Appendix B.5.2) comparing between _weaker_ DiffAug augmentations ($t \\in [0,500]$) and _stronger_ DiffAug augmentations ($t \\in [500,1000]$). While stronger and weaker augmentations help enhance different aspects of classifier robustness — for example, stronger augmentations are more useful to enhance performance on ImageNet-C and OOD detection — using the entire diffusion time scale tends to work better across all tasks.\n\n***\nWe hope we have addressed all your concerns and look forward to discussing any outstanding concerns in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bcHUUu7WH", "reviewer": "Reviewer_3vHc", "summary": "This paper applies a diffusion-based data augmentation method to enhance the robustness of classifier. First, a gaussian perturbation is applied to train examples and then a single diffusion denoising step is applied to generate the augmentations. Besides, DiffAug can also be combined with other augmentations to further improve robustness. Empirically, DiffAug can achieve improvements in classifier generalization, gradient quality and image generation performance.", "questions": "As shown in Weakness 1, why adopt a single diffusion denoising step when the current sampling method, such as DPM-Solver, can generate good results within 10 steps? In theory, better sample generation can achieve better results", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. DiffAug can be combined with other augmentations to improve robustness. Besides, DiffAug can be also used to improve many other performance, such as classifier generalization, gradient quality and image generation performance\n2. DiffAug is simple, computational and easy to follow to improove robustness.", "weaknesses": "1. Absence of ablation study of sampling methods and sampling processes. Did the use of better sampling methods and more steps achieve better results?\n2. Absence of ablation study of diffusion model. Can other diffusion model such as DiT also be used to apply DiffAug", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed reviews. We thank you for appreciating the strengths of our method’s simplicity and computational efficiency and its ability to be combined with other augmentations to further improve robustness. In the following responses, we aim to address the weaknesses and answer your questions:\n\n> **_[Q1] Absence of ablation study of sampling methods and sampling processes. Did the use of better sampling methods and more steps achieve better results? Why adopt a single diffusion denoising step when the current sampling method, such as DPM-Solver, can generate good results within 10 steps? In theory, better sample generation can achieve better results._**\n\nThis is a good question! To answer this question, we performed additional experiments during the rebuttal week using high-quality synthetic data from stable-diffusion as described in our global response (see [E1]). Specifically, we use 1.3M synthetic imagenet images in addition to the original ImageNet training data to finetune the torchvision resnet-50 model for 10 epochs. We call this RN50+Synth. To understand the utility of DiffAug in this case, we finetune another instance of the torchvision resnet-50 model for 10 epochs using DiffAug as well as additional synthetic data. We call this RN50+Synth+DiffAug. \n\nFrom our experimental evaluation across several datasets, we find that DiffAug training and DiffAug-Ensemble (DE) inference offers complementary benefits to extra synthetic training data. We agree with you that, theoretically, DiffAug augmentations generated using a DDPM model trained on ImageNet should not provide further improvements when high-quality synthetic data from SD — trained on LAION-5B, that also covers ImageNet — is already available. Yet, we surprisingly observe that DiffAug improves over and beyond additional high-quality synthetic data! To explain this, we first note that DiffAug is qualitatively different from previous diffusion-based augmentation techniques. Depending on the diffusion time used to generate the DiffAug augmentation, the resulting image can vary greatly in quality (as shown in Fig. 1). As a result, classifying some of these augmented images is more challenging as compared to the original examples producing a regularizing effect that leads to empirical robustness improvements. Lastly, yet importantly, the complexity introduced by DiffAug training does not sacrifice test accuracy despite training on poor quality examples. This is an unusual and noteworthy property that can be understood by interpreting denoised examples to lie on the image manifold following recent theoretical studies [E.g., refs. 9 & 34 in the main paper]. \n\nIn summary, DiffAug can enhance robustness even when efficient sampling techniques are available to synthesize high-quality images since its performance improvement can be attributed to the regularizing effect from learning to classify partially synthesized train examples (i.e.,  diffused-and-denoised examples). We will include these results in the final version of the paper and supplementary materials.\n\n\n***\n> _**[Q2] Absence of ablation study of diffusion model. Can other diffusion model such as DiT also be used to apply DiffAug.**_\n\nWhile we consider the DDPM model (a variance-preserving (VP) SDE) for all our ImageNet experiments, we use a variance-exploding (VE) SDE diffusion in our classifier-guided generation experiments for the CIFAR10 dataset. We find that DiffAug introduces identical improvements for both DDPM and VE-SDE in terms of classifier generalization, perceptually aligned gradients, and improved image-generation performance. Yes, in theory, it should be possible to apply DiffAug with a diffusion-transformer model instead of the UNet architecture. \n***\n\nWe hope this resolves all concerns and look forward to resolving any outstanding concerns during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SNQ2QtkqiC", "reviewer": "Reviewer_QY34", "summary": "The paper proposes DiffAug, a new method for training image classifiers that are more robust. DiffAug is based on diffusion models and effective at improving classifier robustness in several ways, such as resistance to variations in the data. It also can improve the performance of classifier-guided diffusion models. Furthermore, DiffAug is computationally efficient and can be combined with other augmentation techniques.", "questions": "1.\tThe authors state on Line 106 that they are unaware of prior studies training classifiers with denoised examples. However, Line 20 mentions previous work using \"synthetic images generated using Imagen [38]\". Does reference [38] not qualify as a \"previous study on training classifiers with denoised examples\"? Perhaps the authors meant there's no prior work on training with diffused-and-denoised data?\n2.\tThe authors mention \"include Eq. 5 as an additional optimization objective\" (Line 128-129). Could they elaborate on why it's considered \"additional\"? How does Eq. 5 differ from the standard classification loss typically used during classifier training?\n3.\tIn Section 4, the experiments explore \"unconditional improved-DDPM.\" Have the authors investigated using DiffAug with \"conditional\" diffusion models? Can such text guidance improve DiffAug's performance, or would the \"conditional\" process limit augmentation diversity and hinder performance?\n4.\tIn Table 1, which model structure is trained on when evaluating “AM, DA, and DAM”? It seems the first column didn’t present enough information on that.\n5.\tTable 1 results (columns 3, 4 & 8, 9) consistently show DE underperforming compared to DDA(SE). What explains this difference? Could it be due to DDA's \"self-ensemble\" strategy (although DE also uses ensembles)? Does this suggest that using even multiple-step-denoised examples can enhance classifier robustness?\n6.\tWhile the paper focuses on improving classifier’s robustness based on diffusion models, there's a complementary area of research: diffusion-based attacks (DiffAttack [1], Diff-PGD [2]). How effective is DiffAug against these attacks? Given Diff-PGD's constraints with perturbation norms, the results in the paper on certified adversarial accuracy might generalize. However, DiffAttack is unrestricted. Can DiffAug still be applied in such cases? The authors should supplement these discussions in the paper for better evidence of DiffAug’s robustness.\n7.\tAbout the related work, the idea of leveraging a diffusion model for augmentation and incorporating denoised examples to improve classifier robustness was also mentioned in the discussion section of DiffAttack [1]. Citing this related work would be appropriate. \n8.\tBoth DE and DDS employ a single reverse diffusion step during testing. What are the key distinctions between these methods? There seem no comparisons between them or descriptions about their differences.\n\n[1] Chen, Jianqi, et al. \"Diffusion models for imperceptible and transferable adversarial attack.\" arXiv preprint arXiv:2305.08192 (2023).\n\n[2] Xue, Haotian, et al. \"Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability.\" arXiv preprint arXiv:2305.16494 (2023).", "limitations": "Yes. Limitations and social impacts have been involved in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing and the presentation of the work is good and easy to follow.\n2. The finding, that training with degraded images from the diffusion process won’t harm the classifier’s performance but even improve it, is interesting.\n3. The experiments are comprehensive and diverse, showing the effectiveness of the proposed method.", "weaknesses": "While I currently find the paper favorable and have no major weaknesses to point out, I do have some questions that the authors' responses could help clarify (see the part below). I am open to raising the score based on their explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your insightful reviews and affirmative evaluation of our work. We thank you for your appreciation of our presentation and method’s simplicity/computational efficiency. We agree with you that it is indeed interesting to learn that classifiers can be improved without sacrificing test accuracy through the use of degraded images from the diffusion process! We answer your questions in the following:\n\n**[Q1]** Thank you for this question. The reviewer is correct that we meant no prior work on training with diffused-and-denoised data, and in particular by “denoised” we meant “partially denoised”; we apologize for this confusion, and we will make sure to clarify this. In [38], the generated synthetic images are of high-quality since they use iterative denoising: in fact, to ensure better quality, they first fine-tune the Imagen model on ImageNet data and then use FID to select the model checkpoints and various hyperparameters of the sampling algorithm. \n***\n**[Q2]** Additional optimization objective refers to extra loss term. When extending previous methods with DiffAug, we modify their official code by simply adding $\\mathcal{L}$ in Eq. 5 to the original loss-terms. For example, AugMix involves 2 loss-terms: a cross-entropy loss term and a Jensen-Shannon divergence loss term. When extending AugMix with DiffAug, we introduce Eq.5 as the third loss-term. \n***\n**[Q3]**  Similar to many standard augmentation techniques, we do not rely upon labels for DiffAug broadening its applicability for different tasks and scenarios. For example, DiffAug can be applied at both train-time and test-time (when class labels are unknown). Furthermore, it enables direct use of unlabeled data or a mix of labelled and unlabeled data to train the diffusion model. In theory, it should be possible to achieve performance enhancements with DiffAug by using conditional diffusion models, especially with innovative choices of prompt. However, this introduces additional hyperparameters such as the guidance-strength and choice of prompt. Additionally, it requires two forward passes through the diffusion model (one unconditional and one conditional) per training step. The augmentation diversity may certainly be affected especially if the guidance-strength is very high: in such cases, it is also possible that the classifier could cheat by exploiting imperceptible image statistics (e.g., due to adaptive layer-norms).\n***\n**[Q4]** We use the official pretrained checkpoints for these models and all of them are trained using a ResNet-50 backbone. We will clarify this.\n***\n**[Q5]** As compared to DE, DDA is a more complex technique that utilises multiple-diffusion steps to transform a test-sample from an unknown distribution into the source distribution. We find that the multiple-step denoising can be beneficial when dealing with severe corruptions in ImageNet-C. For example, we find negligible difference between DDA and DE over uncorrupted ImageNet test examples when considering the DiffAug-trained models (76.60 & 76.67 respectively). Yes, an optimal multiple-step denoising approach that effectively transforms the image into the source distribution may further enhance classifier robustness on ImageNet-C. For other distribution shifts such as ImageNet-R and ImageNet-S, we observe that DE offers better test-time adaptation on average as compared to DDA.\n***\n**[Q6,7]**  We thank you for sharing these works on novel methods for diffusion-based adversarial example generation. We agree with your analysis that DiffAug’s certified adversarial accuracy results may generalise to Diff-PGD. From our understanding of DiffAttack, it produces transferable untargeted adversarial attacks at a high success rate and hence, we will need to conduct an empirical analysis to study DiffAug’s effectiveness against these examples. While we are interested in understanding the performance of DiffAug and DiffAug-Ensemble against DiffAttack, we were unable to perform this experiment during the rebuttal period due to time and resource constraints. We will include a discussion on these methods and aim to provide a preliminary empirical analysis in the final submission for completeness. \n\nIn the following, we describe additional results on ImageNet-D, another stable-diffusion generated dataset designed to evaluate robustness (similar to DiffAttack). From Table 2 (global PDF), we find that extra synthetic data (from stable-diffusion, as described in global response) offers the most improvements against ImageNet-D --– similar to the suggestion in the DiffAttack paper. Interestingly, DiffAug training and DiffAug-Ensemble (DE) inference can offer further improvements: for example, RN50+Synth achieves 17.52% accuracy on ImageNet-D while RN50+Synth+DiffAug achieves 19.18% accuracy and can be further improved to 21.41%. Since these results are encouraging, we are curious to evaluate the performance of DiffAug against DiffAttack.\n***\n**[Q8]** This is a good question! DE utilises a set of different diffusion time-steps ($\\\\mathcal{S}$ in Eq. 6) while DDS uses a single diffusion time-step. Crucially, DDS is based on the randomized smoothing theory in [1] whereas DE is inspired from test-time augmentation techniques [2]. Additionally, DE uses average voting for prediction whereas DDS uses majority voting. Further, the number of samples generated for each input is typically higher in DDS (e.g., 100 samples for ImageNet) as compared to DE (9 new samples when using $\\\\mathcal{S}=\\\\{0,50,100 … 450\\\\}$). In Figure 11, we compare the performance between DE and DiffAug: since DiffAug is equivalent to DDS using 1 sample, we expect DDS results to be similar to Fig. 11b. \n\n[1] Certified Adversarial Robustness via Randomized Smoothing\n\n[2] Understanding test-time augmentation.\n\n***\nWe hope this resolves all concerns and look forward to resolving any outstanding concerns during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful and thoughtful reviews with suggestions for improvements. We are happy to see that the reviewers appreciated the novelty (uEtP, gb3b) and simplicity/computational efficiency (uEtP, gb3b, 3vHc, QY34) of the method and found the paper to be well-presented (uEtP, gb3b, QY34) with extensive experiments and analysis (uEtP, QY34). 

Based on the reviews, we conducted additional experiments during the rebuttal week and include the results in the attached PDF. We describe the experimental details as follows:

**[E1] Experiments with additional high-quality synthetic data**  

For this experiment, we consider 1.3 M synthetic images --- released by [1] --- generated with Stable-Diffusion (SD) using diverse prompts aimed at enhancing classifier robustness. We fine-tune the torchvision resnet-50 model for 10 epochs using both real and synthetic images and call this RN50+Synth. Next, we repeat this finetuning process with DiffAug and call this RN50+Synth+DiffAug.  We evaluate these models across many datasets and include the results in ***Table 1*** of the global PDF. 

From Table 1, we immediately observe that large-scale synthetic data can improve performance on ImageNet-R, ImageNet-Sketch and ImageNet-D. However, in each of these cases, we can observe that both DiffAug training and also DiffAug-Ensemble (DE) inference each offer additional improvements. For example, DiffAug training and DE inference can improve performance on ImageNet-R from 49.28 (RN50+Synth, Default (Def)) to 54.71 (RN50+Synth+DiffAug, DE). Similarly, DiffAug training and DE inference improves performance on ImageNet-Sketch (35.45 to 37.39) and ImageNet-D (17.52 to 21.41). 

On ImageNet-C, we observe that extra synthetic data does not offer any improvement while DiffAug training and DE inference offer similar improvements both with and without extra synthetic data. 
***
**[E2] More Datasets**

We present further empirical evidence in favour of DiffAug-Training and DE inference in ***Table 2*** (ImageNet-A/D) and ***Figure 1*** (ImageNet-C, all severities). 

***

Now, we address some common questions:

**Q1. [uEtP, 3vHc] DiffAug vs Additional high-quality data generated with more steps.**

This is an interesting question! We answer this question based on the experiment [E1] by comparing between RN50+Synth and RN50+Synth+DiffAug. Based on our experimental results in Table 1, we interestingly observe that DiffAug using improved-DDPM offers improvements over and beyond additional synthetic data using Stable-Diffusion. This is surprising due to the following reasons:
1. Stable-Diffusion is trained on LAION-5B, a much larger dataset that also subsumes ImageNet while the improved-DDPM model is trained on ImageNet data alone.
2. Additional synthetic data requires more compute per each sample (e.g., 50 reverse-diffusion steps) whereas DiffAug just uses one reverse-diffusion step.

Even when using a diffusion model trained on a smaller dataset, DiffAug manages to offer performance improvements _complementary_ to high-quality synthetic examples. We imagine that a better diffusion model for generating DiffAug augmentations may provide further performance improvements. Ultimately, when using comparable diffusion models (e.g., same training data and similar parameter counts) for synthetic data augmentation as well as DiffAug, +Synth+DiffAug may likely be equivalent to +DiffAug allowing for a compute-efficient diffusion-based augmentation technique that combines the advantages of both extra training data as well as DiffAug. We leave this exploration for future work.

***

**Q2. [uEtP, gb3b] Intuitive Explanation of DiffAug's Performance**

We do have some intuition, which we present here, as to why DiffAug offers additional improvements, even beyond what we obtain by making direct use of 1.3M high-quality synthetic images. 

As a first step (and to state what might be obvious), the improvements in ImageNet-R, ImageNet-Sketch and Imagenet-D when fine-tuned with extra synthetic images can be attributed to the large-scale upstream dataset (i.e., LAION-5B) used to train Stable-Diffusion and various prompts designed to enhance data-diversity (e.g., Table 8 of [1]). To intuitively understand improvements from DiffAug training, we first note that DiffAug augmentation is qualitatively different from previous diffusion-based augmentation methods: depending on the diffusion time used to generate the DiffAug augmentation, the resulting image can vary greatly in quality (as shown in Fig. 1). As a result, classifying some of these augmented images is more challenging as compared to the original examples producing a regularizing effect that leads to empirical robustness improvements. Lastly, yet importantly, the complexity introduced by DiffAug training does not sacrifice test accuracy despite training on poor quality examples. This is an unusual and noteworthy property. We can explain this observation by interpreting denoised examples to lie on the image manifold following recent theoretical studies [E.g., refs. 9 & 34 in the main paper]. 
We can use this interpretation of denoised examples to explain why DiffAug-Ensemble enhances robustness: a diffuse-and-denoise transformation applied to a test example from a different distribution *projects it towards the manifold of the original distribution*. This data manifold intuition helps clarify how and why these examples are fundamentally different from examples augmented with pure gaussian noise.
***

[1] Bansal and Grover. (2023) Leaving Reality to Imagination: Robust Classification via Generated Datasets

Meta-review: This work demonstrates that data augmentation with a single diffusion step improves robustness to input corruptions and adversarial attack and it likewise improves OOD detection with small but consistent gains, and furthermore demonstrates the improvement compounds with other techniques.

Four reviewers are mostly in favor of acceptance (QY34: 7, gb3b: 6, 3vHc: 6) except there is one vote for borderline rejection (uEtP: 5). The authors provide a thorough rebuttal by answering questions and supplying additional results as requested (such as addtional benchmarks like ImageNet-A/D/R and further evidence of complementary effects vs. purely training on synthetic data). These scores follow the rebuttal by the authors, where three of four reviewers raised their scores while uEtP acknowledged the rebuttal and maintained their sore of 5. In more detail, uEtP acknowledged the rebuttal and maintained their score now that concerns are addressed and the results are as they seem ("the numbers are reported correctly and the proposed method should be better than using generated data to increase the dataset size").

The AC sides with acceptance in agreement with the majority of the reviews. The authors are encouraged to address the discussion of unconditional and conditional diffusion models (QY34, uEtP) and follow through on the rebuttal suggestion to include DiTs with class conditioning.
Decision: Accept (poster)