Improving Adversarial Robustness by Contrastive Guided Diffusion Process
Authors: Yidong Ouyang
Liyan Xie
Guang Cheng
Venue: Submitted to ICLR 2023 (2023)

Abstract: Synthetic data generation has become an emerging tool to help improve the adversarial robustness in classification tasks since robust learning requires a significantly larger amount of training samples compared with standard classification tasks. Among various deep generative models, the diffusion model has been shown to produce high-quality synthetic images and has achieved good performance in improving the adversarial robustness. However, diffusion-type methods are typically slow in data generation as compared with other generative models. Although different acceleration techniques have been proposed recently, it is also of great importance to study how to improve the sample efficiency of generated data for the downstream task. In this paper, we first analyze the optimality condition of synthetic distribution for achieving non-trivial robust accuracy. We show that enhancing the distinguishability among the generated data is critical for improving adversarial robustness. Thus, we propose the Contrastive-Guided Diffusion Process (Contrastive-DP), which adopts the contrastive loss to guide the diffusion model in data generation. We verify our theoretical results using simulations and demonstrate the good performance of Contrastive-DP on image datasets.

Under review as a conference paper at ICLR 2023
IMPROVING ADVERSARIAL ROBUSTNESS BY CON- TRASTIVE GUIDED DIFFUSION PROCESS
Anonymous authors Paper under double-blind review
ABSTRACT
Synthetic data generation has become an emerging tool to help improve the ad- versarial robustness in classification tasks since robust learning requires a sig- nificantly larger amount of training samples compared with standard classifica- tion tasks. Among various deep generative models, the diffusion model has been shown to produce high high-quality synthetic images and has achieved good perfor- mance in improving the adversarial robustness. However, diffusion-type methods are typically slow in data generation as compared with other generative models. Although different acceleration techniques have been proposed recently, it is also of great importance to study how to improve the sample efficiency of generated data for the downstream task. In this paper, we first analyze the optimality condi- tion of synthetic distribution for achieving non-trivial robust accuracy. We show that enhancing the distinguishabilty among the generated data is critical for im- proving adversarial robustness. Thus, we propose the fusion Process (Contrastive- -DP),,,,tvveD,,,, which adopts the contrastive loss to guide the diffusion model in data generation. We verify our theoretical results using simula- tions and demonstrate the good performance of Contrastive-DD on image datasets.
Seerrastve-GGiider Dif-
INTRODUCTION
The success of most deep learning methods relies heavily on massive amount of training data, which can be expensive to acquire in practice. For example, in autonomous driving (O (O'Kelly al.] 2018) and the medical diagnosis (Das et al. 2022) type applications, the number of rare scenes usually very limited in real data. Moreover, it may be expensive label the data supervised learning. These challenges call for methods that can produce additional training data that satisfy two essential properties: (i) the additional data should help improve the downstream task performance; (ii) the additional data should be easy to generate. Synthetic data generation based on deep gener- ative models has shown promising performance recently to tackle these challenges (Sehwag et al,, 2022 Gowal et al. 2021 Das et al. (022).
In synthetic data generation. one aims to learn a synthetic distribution (from which we generate synthetic data) that is close the true date-genest date-generating distribution based training data available, and most importantly, can help improve the downstream task performance. Synthetic data generation highly related generative models. Among various kinds of generative models, the score-based model and diffusion type models have gained much success in image generation recently (Song & al. 2015 Nichol & Dhariwal 2021 Bao et al. 2022 Rombach et al. 2022). As validated in image datasets. the prototype of diffusion models, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al.,
Ermon 2019 Song et al. 20216 2020 [Song & Ermon 2020 Sohl-Dickstein
2020), and many variants can generate high-quality image data as compared with classical generative
models such as GANs (Dhariwal & Nichol 202T).
This paper mainly focuses on the adversarial robust classification of image data, which typically requires more training data than standard classification tasks. In Gowal et al. (002)). 100M high- quality synthetic images are generated by DDPM and achieve the state-of-the-ar performance on adversarial robustness on the CIFAR-10 10 dataset, which demonstrates the effectiveness of diffusion models in improving adversarial robustness. However, a major drawback of diffusion-type methods is the slow computational speed. More specifically, DDPM is usually 1000 times slower than GAN (Song et al. 2021a) and this drawback is more serious when generating large number of samples,

---

Under review as a conference paper at ICLR 2023
e.g., it takes more than 99 GPU days for generating 100M image data according to Gowal et al. (202I). Moreover, the computational costs will also increase dramatically when the resolution of images increases. which inspires plentiful of works studying how to accelerate the diffusion models (Song et al., 2021a, Watson et al. 2022 Ma et al., 2022, Salimans & Ho 2022 |ao et al. [022] Cao et al. 2022 Yang al. 2022). In this paper, aim to study the aforementioned problem from different perspective "how to generate effective synthetic data that are most helpful for the downstream task?". We analyze the optimal synthetic distribution for the downstream tasks to improve the sample efficiency of the generative model.
We first study the theoretical insights for finding the optimal synthetic distributions for achieving adversarial robustness. Following the setting considered in Carmon et al.] (2019), we introduce a family of synthetic distributions controlled by the distinguishability of the representation from different classes. Our theoretical results show that the more distinguishable the representation is for the synthetic data, the higher the classification accuracy we will get when training a model on such synthetic data sets. Motivated by the theoretical insights, we propose the Contrastive- Guided Diffusion Process (Contrastive-DPP for efficient synthetic data generation, incorporating the contrastive learning loss (van den Oord et al.) 2018 Chuang et al., 20200 Robinson al. 2021) into the diffusion process. We conduct comprehensive simulations and experiments on real image datasets demonstrate the effectiveness of the proposed Contrastive-DP.
The remainder of the paper is organized as follows. Section presentt the problem formulation and presents preliminaries on diffusion models. Section 3 contains the theoretical insights of optimal synthetic distribution under the Gaussian setting. Section4prros proposes a new type of data generation procedure that combines contrastive learning with diffusion models, as motivated by the theoretical insights obtained in Section 3 Finally, Section 3 conducts extensive numerical experiments to validate the good performance of the proposed generation method on simulation and image datasets.
PROBLEM FORMULATION AND PRELIMINARIES
We first give brief overview of adversarial robust classification, which is our main focus, but the whole framework is widely applicable to other downstream tasks in general. Denote the feature space as X, the corresponding label space as y, and the true (joint) data distribution as D Pxxy- Assume we have labeled training data Drrain == (( Y1) " We aim to learn robust classifier fo : X + Y. parameterized by a learnable 0 that can achieve minimum adversarial loss:
min Ladio (0) = B
max f(x + +5, BEA
where f(x, y. 0) = 1y + ʃo(x)} is the 0-1 loss function, 1(-) is the indicator function, and = {6: |8|| S c} is the adversarial set defined using for Intuitively, the solution to (I is robust classifier that minimizes the worst-case loss within e-neighborhood of the input features.
In the canonical form of adversarial training. we train the robust classifier fe on the training set Dtrain = ((x,, }i) by solving the following sample average approximation of [
min Eudr (f) = n2
max {(1; + 5,, Vi, 0).
(2)
S.EA
2.1 ADVERSARIAL TRAINING USING SYNTHETIC DATA
Synthetic data generation is one way to artificially increase the size of the training set by generating a sufficient amount of additional data, thus helping improve the learning algorithm's performance Gooal et al.] 2021). The mainstream generation procedures can be categorized into two types: (i generate the features (œ) first and then assign pseudo labels to the generated features; (ii) or perform conditional generation conditioned on the desired label. Our analysis is mainly based on the former paradigm, which can be easily generalized to the conditional generation procedure, and our proposed algorithm is also flexible enough for both pipelines. Denote the distribution of the
generated features as Dxx and the generated synthetic data as Dayn = {(ē,, H.) Here the feature
Running on RTX 4x2080Ti GPU cluster.
2

---

Under review as a conference paper at ICLR 2023
values Ø; generated from the synthetic distribution Dx. and 8/s pseudo labels assigned by classifier learned on the training data Duain- Combining the synthetic and real data, we will learn the robust classifier using a larger training set Danl Dirin J Payn which now contains + n samples:
max {(o; + 8, Me. 01 0)} (3) o.ED
max l(a; + o,, 1.: + - B,
Σ
min 0
1
72
where 1 E (0, 1) is parameter controlling the weights of synthetic data.
2.2 DIFFUSION MODEL FOR SYNTHETIC DATA GENERATION
We build our proposed generation procedure based the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al.
2020) and its accelerated variant Denoising Diffusion Implicit Model (DDIM) a aa) In the following, we briefly review the key components of DDPM.
Song et al.
The core DDPM a forward Markov chain with Gaussian transitions q( (x,-1) to inject noise to the original data distribution qzx)). More specifically, Ho et al. (2200) model the forward Gaussian transition as:
q (x, xt-1) = N' ( (yaro:-1, - (1-aa) I),
where 02 t = 1.2.... 7 is a decreasing sequence to control the variance of injected noise, and Il is the identity covariance matrix. The joint likelihood for the above Markov chain can be written as 7 (mr)r)
(mo) IT'= (1))) DDPM then assumes we have po (xo:r) = pe (xx) II== (Ht-1(z;; for the reverse process, where pa pa( is parameterized using neural network. The training objective is minimize the Kullback- Leibler (KL) divergence be- tween the forward and reverse process, DRI. (q (IO:T), Pa (2o.r)), which can be simplified as:
min Er, |||e - -ee (Vārzo + -I- v1- -,e,
where Vo ~ g(xo), a₂ = II a, for 1,..., T. (O. and cø denotes the neural network parameterized by 0 to be learned. We refer to Ho et al. (2020) for the detailed algorithms.
After learning the time-reversed process parameterized by 0, the original generation process in Ho et al. (2020) is time-reversed Markov chain follows:
1 Vai
1 04
co (x1, + 0121, t=
I1-1
Et VT
-T-ar at
where 2, ~ *, II) if 1 and 2, 0 if 1. DDIM (Song et al. 2021a) speeds up the above procedure by generalizing the diffusion process to non-Markovian process, leading to sampling trajectory much shorter than T. DDIM carefully designs the forward transition q( [z,, 2D) such that g (2)) = A (Va,zo, (1 ax) IT) for all 1, ..... T. The great advantage of DDIM that it admits the same training objective DDPM, which means we can adapt the pre-trained model of DDPM and accelerate the sampling process without additional cost. The key sample-generate step in DDIM is follows:
It-1 = Var-1
Ft
area (xr, t)
/I - at-1 Eg (#t, 1),
(4)
Vat
pointing
predicted 2p
in which we can generate x;-1 using and Øp Also, the generating process becomes deterministic.
THEORETICAL INSIGHTS: OPTIMAL SYNTHETIC DISTRIBUTION
In this section, we consider conerete distributional model used Carmon al. 2019 Schmidt et al. (2018), and demonstrate the advantage of refining the synthetic data generation process using the optimal distribution for synthetic data generation can help reduce the sample complexity needed for robust classification. This provides theoretical insights and motivates the proposed generation method to be introduced in Section 4

---

Under review as a conference paper at ICLR 2023
3.1 THEORETICAL SETUP
Consider a binary classification task where X' = R [-1,1}. The true data distribution D yxxyy is specified as follows. The marginal distribution for label y is uniform in V. and the conditional distribution of features is æly ~ N NNye,, o²Ia), where E I2d is non-zero, and Ta is the dimensional identity covariance matrix. Assume we generate set of synthetic data from another synthetic distribution D.
We focus on learning robust linear classifier under such setting. The family of linear classifiers is represented as f0) = sign(0 x²). Recall that we first generate features and then assign pseudo labels to the features. Therefore, self-learning paradigm is adopted here (Wei et al. 2020). Given set of unlabeled features {ã1, Ø22 ..... 3s we apply an intermediate linear classifier parameterized by Oiner = + >i= Milt, learned from real data Dumin to assign the pseudo-label. Then, the syn- thetic data Payn {(01 ......... (za, }a)}, where D sign ( Di), I,..., n. We combine the real data and synthetic data Dan == Diran Dsyn {( y:) y;)]E ((),, ū.)]==) to obtain an approximate optimal solution 2 iimal as:
0 O
y + Σ Cイ
(S)
Note that the final linear classifier Onnal depends on the synthetic data generated from D. We aim to study which synthetic distribution D can help reduce the adversarial classification error (also called robust error) eorrt (Se..) P (a.))>>(33 A, Séme (xc + δ) + y). where A {6
S
<). And we similarly define the standard error as erTstandad (Som.) ui (Somm) (x) # y) which will be used later.
Remark 1 (Comparison with existing literature). In Carmon et al. (2019) Deng et al. 2021), sample complexity results are proposed based on the same Gaussian mixture setting. The major difference is that they all assume the learned linear classifier Ojmat is only learned from synthetic datu Day rather than the combination of the real and synthetic data Dan. In general, our theoretical setup matches well with the practical algorithms.
3.2 THEORETICAL INSIGHTS FOR OPTIMAL SYNTHETIC DISTRIBUTION
We first study the desired properties of the synthetic distribution D that can lead a better adversar- ial classification accuracy when the additional synthetic sample Dayn is used in the training stage. In
Carmon al. 2019)), the case D D is studied, i.e., they consider the case that additional unlabeled available, and they characterize the usefulness of those additional training data. Compared with from Carmon et al. 2019)) we consider general distributions D which does not necessarily equal to D.
data from the true distribution
First note that by the Bayes rule, the optimal decision boundary for the true data distribution is given by _ x O. Therefore, we restrict our attention synthetic data distributions that satisfy: (i) the marginal distribution of the label ý is also uniform in V, same as D; (ii) the conditional probability densities p(ø 1) and p( ( y = 11 1) of the synthetic data distribution are symmetric around the true optimal decision boundary f x = O. More specifically, we start with special case of the synthetic data distribution Dx 0.5.V I²I) 0.5.' (I)) (note that when çs cu for some constant c, the above two conditions are all satisfied).
In the following proposition, we present several representative scenarios of synthetic distributions in terms of how they may contribute to the downstream classification task. Figure gives a pictorial demonstration for different cases.
Proposition 1. Consider a special form of synthetic distributions Dx 0.5N o2I) +
0- and assume {2 (ø1,... ...,, Dā} } are samples from Dx. We follow the self-learning paradigm described in Section 3.7 to learn the classifier fo when ñn is sufficiently large we have:
Case 1: Inefficient Dx. When (jz, 11) = 0, the standard error eTT aundard (So,..) achieves the maxi- mum and when (jt, μr =lg) O. the robust error erT (Fon.) achieves the maximum.
4

---

Under review as a conference paper at ICLR 2023
i ep, enor achieves minimum / = eroor
p
p))
maximmza x
Figure 1: Demonstration of Proposition 1
Case 2: Optimal Dx for clean accuracy. When j cp for O, stnnaandard (Sowa) achieves the minimum, and the bigger the c is, the smaller the sIadurd (f
Case 3: Optimal Dx for robust accuracy. When ja c(u Eld) for c. > O, the robust eтror eTT robst ( achieves the minimum, and the bigger the e is, the smaller the mtust (
Remark 2 (Comparison with the existing characterization of the synthetic distribution). We briefly comment on the main differences and similarities with Deng el al. (2027), 1) in which similar result was presented in Theorem 4. In Deng et al. (2027), the optimal solution of 0* was given for minimiz- ing robust error erT wobss (S and they provides a specific unlabeled distribution p - - that achieves asymptotic optimality under certain condition. In this paper, we propose a general family of optimal distribution controlled by scalar c, which represents the distinguishabiiity of the fea- ture. The optimal 0 proposed in Deng et al. (202I) can be recovered when e - oc. Therefore, our conclusion points out the optimality condition of unlabeled distribution and inspires a line of work to improve the performance of jpmall by making the feature of unlabeled distribution distinguishable.
Table 1: Simulation results validating findings in Proposition [ d = and d 100 denotes the dimension of æ, representing the low-dimensional and high-dimensional cases, respectively. For d 2, set | 1 2, E 0.5, and for d 100, set ||| 4, E 0.1. We "Real" to denote the real data distribution and n to denote the number of data from the real distribution, while we use "c" to denote different synthetic distributions with cu and use denote the number of synthetic data. The results and the standard deviation in the bracket are averaged over 50 independent trials.
2=2
d=100
Accuracy
Robust Accuracy
Accuracy
Robust Accuracy
Real
n
0.9201 (0.0012)
0.7593 (0.0020)
n 10
0.9023 (0.0192)
0.6843 (0.0359)
n 100
0.9205 (0.0004)
0.7608 (0.0006)
n 100
0,9682 (0.0014)
0.8239 (0.0028)
i 10
0.9159 (0.0099)
0.7541 (0.0096)
ñ 10
0.7562 (0.0564)
0.4611 (0.0694)
c 0.5
100
0.9213 (0.0011)
0.7601 (0.0009)
ñ 100
0.9505 (0.0047)
0.7848 (0.0111)
ñ 10
0.9133 (0.0066)
0.7502 (0.0061)
ñ 10
0.8866 (0.0273)
0.6557 (0.0487)
0
ñ 100
0.9165 (0.0005)
0.7528 (0.0006)
ñ = 100
0.9695 (0.0012)
0.8239 (0.0031)
0.9209 (0.0038)
0.7523 (0.0025)
ñ 10
0.9400 (0.0100)
0.7603 (0.0233)
c 1.5
100
0.9232 (0.0003)
0.7538 (0.0005)
ñ 100
0.9743 (0.0008)
0.8343 (0.0011)
Simulation results. To verify the findings in Proposition]] I we conduct extensive simulation ex-
periments for Gaussian example with varying data dimensions, sample sizes, and the position of a. In Table I we demonstrate the clean and robust accuracy learned on synthetic distribution with the angle between μ and elg equals 0". More experiment results (with 30 60". and 90%) can be found in Table / g 9 and 10 in Appendix B In most cases, the classifier learned from the synthetic
distribution with J Cft. with achieves better performance even than the iid samples.
5

---

Under review as a conference paper at ICLR 2023
4 DIFFUSION MODELS GUIDED BY CONTRASTIVE LOSS
Proposition 1 and the corresponding simulation results in Table 1 show that the synthetic data can help improve the classification task especially when the representation of different classes is more distinguishable in the synthetic distribution. Therefore, contrastive loss (van den Oord et al. 2018) can be adopted to explicitly control the distances of the representation of different classes. Therefore, propose variant of the classical diffusion model, named Contrastive Contrastive-Guided Diffiusion Process (Contrastive DP), to enhance the sample efficiency of the generative model. In this section, we first present the overall algorithm of the proposed Contrastive-DP procedure in Section 4.1 then describe the detailed design of the contrastive loss in Section
4.1 CONTRASTIVE-GUIDEE DIFFUSION PROCESS
The detailed generation procedure of Contrastive-DP is given in Algorithm | We highlight below some major differences between the proposed Contrastive-DD and the vanilla DDIM algorithm. In each time step t of the generation procedure, given the current value x;"), we add the gradient of the contrastive loss (1) (5) (6) ;)) with respect xx() to the original diffusion generative process, Tp
(5) is the positive pair of æ;) (will be explained in detail later), is the temperature for softmax, and À the hyperparameter balancing the contrastive loss within the diffusion process.
here Up
This modification ensures that the generated data will be distinguishable among data in the same batch. The construction of the contrastive loss (-omtr(-) )) is very flexible we can adopt multiple forms of contrastive loss together with different selection strategies of positive and negative pairs, which will discussed in detail in the following.
4.2 CONTRASTIVE LOSS FOR DIFFUSION PROCESS
Algorithm Generation in Contrastive-guided Diffusion Process CrncesssContrriie (Contrastive-DP)
1: XT {-
1=1 1=
(8) (0,
2: t T
3: while # do
4: for 1:m do
Choosing 2 as the positive pair of an (6)
5:
6:
Az!") (6)
x V feoara ( x r) + en {m) (0) .t)
(8)
Sa) )+ + vI a--1 Ar
7:
a1
8:
t t=
t
9:
end for
10:
end while
11:
return Xo
(
)
(1)
1
Let X = 21, ... Im be minibatch of training data. We apply the contrastive loss the embedding space. Assume f()) is the feature extractor that maps the input data in X onto the embedding space.
In general, we adopt two forms of the contrastive loss feontra (zr (i) (i)
xp (.:)) used in Algorithm 1
First is the InfoNCE loss: LIiNNCE (æa, Zpi T) = -- log(a=(xa. xpp/ ΣmL 1xzufr (ataar(aa, 2xk) )), where is the batch size, is the temperature for softmax, Ta. Tp denote the anchor and the positive pair, respectively, 9- (z, x²) exp(f(x) f(x'//T, and all images except the anchor x, in the minibatch X is negative pairs. InfoNCE loss is an unsupervised learning metric and does not explic- itly distinguish the representation from different classes, which implicitly regards the representation from the same class as negative pair.
Second is the hard negative mining loss: bHNM (wa, (pi)) m// (Ez..a9 [[rr (æa, x.)]] - -+Eu-gg [(gr (2., v)]))}, where m denotes the batch size, 7
log(gr (x., xp)) (xa, xp)) +
1 -- + denotes the probability of observing any different class with æn and qa is an unnormalized von Mises-Fisher distribution (Jammalamadakal 2011), with mean direction f(x) and "concentra- tion parameter" (3 control the hardness of negative mining; gs and 93 be easily approximated

---

Under review as a conference paper at ICLR 2023
by Monte-Care importance sampling techniques. We refer to Chuang et al. (2020); Robinson al. (202T) for detailed descriptions of hard negative mining contrastive loss. Compared with the In- foNCE loss that does not consider class/label information, the hard negative mining (HNM) loss enhances the discriminative ability of different classes in the feature space.
It is worth mentioning that the Contrastive-DD enjoys the plugin-type property it does not modify the original training procedure of diffusion processes and can be easily adopted to various kinds of diffusion models.
Numerical Validations. We first demonstrate the effectiveness of contrastive-DP in Figure[juing simulation example. Consider the binary classification problem as in Section 3.1 and the real data for each class are generated from a Gaussian distribution. Figure 2(a) demonstrates the synthetic data generated by the vanilla diffusion model, which recovers the ground-truth Gaussian distribution well. When using the contrastive-DP procedure with HNM loss, we obtain the generated synthetic data as shown Figure which more distinguishable with much smaller variance.
(a) DDPM
(b) Contrastive-DD
Figure 2: An illustration of the effectiveness of synthetic distribution guided by contrastive loss.
In addition, FigureBland FigureHin Appendix C.2 demonstrate the synthetic data distribution guided by different kinds contrastive loss mentioned above. It can be shown that InfoNCE loss and hard negative mining method cannot explicitly distinguish the data within the same class and thus form circle within each class to maximize the distance between samples, while the conditional version of contrastive loss (given the oracle class information) can make two classes more separable.
5 REAL-WORLLD IMAGE DATASETS
In this section, we demonstrate the effectiveness of the proposed contrastive guided diffusion process synthetic data generation in adversarial classification tasks. We first compare the performance of Contrastive-D DP with the vanilla DDIM methods in Section 5.1 Then, present comprehensive
ablation study on the performance of Contrastive-DP to shed insights on how to adopt the contrastive loss functions the diffusion model in Section 5.2 especially which contrastive loss gives the best performance on the diffusion process and how to choose the hyperparameter À to control the strength of the guidance of the contrastive loss.
5.1 EXPERIMENTAL RESULTS
We test the contrastive-DP algorithm on two image datasets, the CIFAR-10 dataset (Krizhevsky 2009) and Traffic Signs dataset (Houben et al., 2013). CIFAR-10 dataset contains 50k training images in 10 classes and 10K images for testing, while Traffic signs dataset contains 39252 training images 43 classes and 12629 images for testing. For CIFAR-10 dataset, we generate 50K, 200K, and 1M additional images together with the original training images for adversarial training, while for Traffic signs dataset, we synthetic 50k images. To demonstrate our Contrastive-DD algorithm is flexible to be adopted to various kinds of diffusion models and make use of the existing pre- trained models, we establish our Contrastive-DP algorithm on the unconditional DDIM for CIFAR- 10 datasets and on the conditional DDPM for Traffic signs dataset. The detailed description of the pipeline for generating data and the corresponding hyperparameter can be found in Appendix D.2
Table 2 demonstrates the effectiveness of our contrastive-De DP algorithm the CIFAR-10 dataset which achieves better robust accuracy on all data regimes than the vanilla DDIM. All of the results
SSinee the Pytorch Implementation of Gowal et al. (202)) is not open source, we utilize the best unofficial implementation reconduct all the experiments for fair comparison.

---

Under review as a conference paper at ICLR 2023
higher than the baseline result without synthetic data by large margin (+4.37% in 50K setting and +7.3% in 1M setting). Table [ demonstrates the effectiveness of our contrastive- algorithm on the Traffic Signs dataset. Our contrastive-DD DD achieves better clean and robust accuracy than the vanilla DDPM model and is also higher than the baseline result without synthetic data by a large margin (+9.96%)
The clean and adversarial accuracy on the CIFAR-10 dataset. The robust accuracy is reported by the worst accuracy obtained by either AUTOATTACK (Croce & Hein| 2020) or AA+MT (Gowal et al. 2020) with coc. 8/255 and WRN-28-10. 50k, 200k, and 1M denote the number of synthetic used for adversarial training.
Table 2:
No additional data clean mh
50K
200K
IM
clean
rb acc
clean ace
rob
clean acc
rob acc
WRN-28-10 (DDIM)
84.65% 83.66%%
52.46% 53,91 %
85.86 % 85.71 %
54. 44.99% 55.79% %
85.37% 86.30%
56.61% 56.84 %
81.09%
49.54%
WRN-28-100 10 (Contrastive-DP)
Table 3: The clean and adversarial accuracy on the Traffic Signs dataset. he results and the standard deviation in the bracket are averaged over 3 independent trials.
clean acc
rob acc
No additional data DDPM
78.52% (0. 16%)
46.03%
86.79%(0. (0.1%% 86.5 (0.62 %)
56.01 (0.14%% 56.27 % (0.25 )
Contrastive-DD
5.2 ABLATION STUDIES
Sensitivity of A. Table shows the influence of the strength of the contrastive loss. 100k: gives consistently better results than smaller À = 50k or larger À = 200k on robust accuracy on all settings. Moreover, we find the larger the À is, the better performance we get on clean accuracy when the additional data is small (50K case), while the smaller the À is, the better performance we get on clean accuracy when the additional data large (IM case).
Table 4: The performance of Contrastive-DD DP under different À values.
50K
200K
1M
clean acc
rob acc
clean acc
rob acc
clean acc
rob acc
À 50k
84.41% %
53.78%
85.45%
55.24%
86.35%
56.83%
A = 100k
83.66%
53.91 %
85.71% %
55.79 % 55.33%
86.30%
56.84 %
A 2004
84.51 %
53.55%
85.51 %
85.98%
56.69%
The effectiveness of different contrastive losses. Table 5
demonstrates the performance of dif- ferent design of the contrastive loss. We find out that applying the hard negative mining together with the embedding network achieves better clean and robust accuracy when the additional data is small (50K and 200K setting), while the infoNCE loss achieves better clean and robust accuracy when the additional data is large (1M setting). This result shows that we can improve the sample efficiency of the generative model by carefully designing the contrastive loss.
Table 5: The performance of Contrastive-DD under different contrastive loss: infoNCE and HNM losses, and w/wo embedding denote with/without an embedding network.
50K
200K
1M
clean acc 83.40%
rob ace
clean acc
rob acc
clean acc
rob acc
DDIM+infoNCE DDIM-HNM(W embedding)
52.74%
84.18%
54.75%
85.64%
56.28%
84.20 %
53.19% %
85.71 %
54.92 %
85.29%
56. 16%
DDIM-HNM(wo embedding)
83.97%
52.89%
85.65%
54.83%
85.38%
55.95%
Data selection for synthetic data. Data selection methods are worthy of study since, in practice, we would like to know whether we can achieve better performance by generating a large number of samples and applying some selection criteria filter out some samples. Therefore, several data selection criterion and evaluate corresponding effectiveness Table 6 All of the se- lection methods on Contrastive-DP are higher than vanilla DDIM plus selection methods, which
propose
8

---

Under review as a conference paper at ICLR 2023
demonstrates the superiority of using the contrastive learning loss the guidance rather than using selection methods on the images generated by the vanilla diffusion model.
Table 6: Comparison of different data selection criteria. The detailed explanation of each selection method can be found Append D.3
D.3
50K
200K
1M
clean ace 79.93%
rob acc
clean acc
rob acc
clean acc
rob acc
DDIM (Separahility)
49.49%
85.09%
54.90%
84.87%
56.08%
Contrastive-DP (Gradient norm)
83.92%
55.09%
84.64%
55.17%
86.36%
57.11% 1%
Contrastive-DD DP (Gradient norm-rob)
83.91%
55.23% %
84.78%
55.42%
85,93%
57.18%
Contrastive-DD (Entropy)
84.17% %
55.08%
85.71% %
55.79%
86.30%
56.84%
RELATED WORK
Using generative models to improve adversarial robustness has attracted increasing attention re- cently. Gowal et al. (2021) uses 100M high-quality images generated by DDPM together with the original training set to achieve state-of-the tofofffth-artt performance on the CIFAR-10 dataset. They propose important metric for measuring the efficacy of the synthetic data. In Sehwag et al. 22022). it was claimed that the transferability of adversarial robustness between two data distributions is measured by conditional Wasserstein distance, which inspires criterion for selecting samples. Our work follows the same line, but we investigate how to generate the samples with high information rather than applying the selection to the data generated by the vanilla diffussin model. Below we also summarize some closely related work in different lines.
to use Complementary
Sample-efficient generation. We view the sample-efficient generation problem Bi-level optimization problem. We can regard how to synthesize data as the meta objective and the perfor- mance of the model trained the synthetic data as the inner objective. For auttentugmentaionnn based methods, Ruiz et al. (2019) adopt reinforcement learning based method for optimizing the genera- in order to maximize the training accuracy. For active learning based methods, [Taan et al. (2019) use an Auto-Encoder to generate new samples based on the informative training data selected by the acquisition function. Besides, Kim et al. (2020) combines the active learning criterion with data augmentation methods. They use the gradient of acquisition function after one-step augmentation as guidance for training the augmentation policy network.
Theoretical analysis of adversarial robustness. In Scchiidtt et al. (2018), the sample complexity of adversarial robustness has been shown be substantially larger than standard classification tasks in the Gaussian setting. Carmon et al. (2019) bridges this gap by using the self-training paradigm and corresponding unlabeled data. Deng et al. (202T) further extends the aforementioned conclusion by leveraging out-of-domain unlabeled data. None of the works mentioned above investigates the optimal distribution for unlabeled synthetic data.
Contrastive learning. Contrastive learning algorithms have been widely used for representation learning (Chen al. 20200 He al., 2020; Grill et al. (020). The vanilla contrastive learning loss, InfoNCE (van den Oord et al. 2018), aims to draw the distance between positive pairs and push the
negative pairs away. To mitigate the problem that not all negative pairs may be true negatives, the negative hard mining criterion was proposed in (Chuang et al. 2020 Robinson et al. 2021).
7 CONCLUSION
In this paper, we delve into which kind of synthetic distribution is optimal for the downstream task, especially for achieving adversarial robustness in image data classification. We derive the opti- mality condition under the Gaussian setting and propose the Contrastive-guidde Diffusion Process (Contrastive-DD), plug-in algorithm suitable for various types of diffusion models. We verify our theorem on the Gaussian simulation and demonstrate the superiority of the Contrastive-DD algorithm on the image datasets.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-dpm: an analytic estimate of the optimal
reverse variance diffusion probabilistic models. ICLR, 2022.
Li. A Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Am Heng, and Stan Li. survey generative diffusion model. arXiv preprint wrXiv:2209002646, 2022.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Ching-Yar Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. In NeurIPS, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee attacks. ArXiv, abs/2003.01694 2020.
Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey H. Tison, Alberto L. Sangiovanni- Vincentlel, and Costas J. Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In AAAI, 2022.
Zhun Deng, Linjun Zhang, Amirata Ghorbani, and James Y. Zou. Improving adversarial robustness via unlabeled out-of-domain data. In AISTATS, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, Abs/2105.05233. 2021.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy A. Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against corm-bounded adversarial examples. ArXiv, abs/2010.03593, 2020.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy Mann. Improving robustness using generated data. NeurlPS, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised learning. ArXiv, 0tt.22006.077333 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian enror linear units (gelus). arXiv: Learning, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detec- tion of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference Neural Networks, number 1288, 2013.
Pavel Izmailov, Dmitrii Podoprikhin, Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. ArXiv, abs//b03.000 1803.05407, 2018.
Sreenivasa Rao Jammalamadaka. Directional statistics, i. 2011.
Yoon- Yeon-Yeong Kim, Kyungwoo Song, Joonho Jang, and II-Chul Moon. Lada: Look-ahead data ac- quisition via augmentation for active learning. ArXiv, abs/201 1.04194, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2023
Hengyuan Ma, Li Zhang, Xiatian Zhu, Jingfeng Zhang, and Jianfeng Feng. Accelerating score- based generative models for high-resolution image synthesis. arXiv preprint arXiv:2206.04029, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, John C. Duchi, and Russ Tedrake. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In NeurIPS, 2018.
Joshua Robinson, Ching-You Yao Chuang, Suvrit Sra, and Stefanic Jegelka. Contrastive learning with hard negative samples. In ICLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz. Samuel Schulter, and Manmohan Chandraker. abs/1810.00 11.102200,,,..1 2019.
Learning to simulate. ArXiv,
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. In NeurlPS, 2018
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve ad- versarial robustness? In ICLR, 2022.
Jascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based based generative models. Advances in neural information processing systems, 33:12438- 12448, 2020.
Yang Song. Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011. 13456,, 2021b.
Toan Tran, Thanh- Toan Do, Ian D. Reid, and G. Carneiro Bayesian generative active deep learning. ArXiv, abs/1904.1 11643, 2019.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, ArXiv, abs/1807 707478377.84 2018.
Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Sample-efficien neural architecture search by learning action space. ArXiv, abs/1906.06832 2019.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In ICLR, 2022.
11

---

Under review as a conference paper at ICLR 2023
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-traing self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2020.
Ling Yang, Zhilong Zhang, and Shenda Hong. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint wrXiv:22090007%. 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/ (bss/665500716,, 2016.
Hongyang R. Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jor- dan. Theoretically principled trade-off between robustness and accuracy, ArXiv, abs/ 119119917777 2019.
12

---

Under review as a conference paper at ICLR 2023
A THEORETICAL DETAILS
A. ERROR PROBABILITIES IN CLOSED FORM.
Here, we briefly recapitulate the closed form formulation for the standard and robust error probabil- ities as detailed in Carmon et al. (2019) Deng et al. (2021).
The standard error probability can be written as
(N
p T
stanadd (ʃo) = P ( x 0 < 0) =
< <0)
=
(6)
σ 0
where
(
1 12 dt
2
the Gaussian eтror function and is non-increasing. Clearly the standard error probability mini-
mized when T [ i.e., op for some scalar 0. We may impost ||0||2 1 to ensure
the unique solution 0 = /a/ ||
The robust error probability under the foo adversarial set A {ō 110 sc is
Tobust (fo)
inf (y (x- 1x- v) T) < |I
= (y xx To -e|0011 <0) = (N 1, < <=((0]))
Q μ θ
=|10111
01 a
In the following part, simpler notation erT (fo) for the robust error err robust (fo) without ambiguity. The closed-form of the optimal O" that minimizes the above robust error eTT can be shown to be (Deng al. 2021):
0" (p(
(T)
where Tg (p)) is the hard-thresholding operator with (T)) sign (Ht) max {lu e, 0)- Under the mild assumption μ4 > 2. (1.2.....d the optimal solution can be simplified as:
0" / =1
μ =1
Remark 3. Note that when μ c1 for some constant C >> e, the optimal solution 0" #-o1 for minimizing the robust error is the same the optimal solution l| for minimizing the standard error. Otherwise, these two solutions are different, representing a trade-off between robustness and accuracy.
A.2 DETAILS FOR THE THEORETICAL ANALYSIS SECTION 3
Overall, we would like to design an appropriate synthetic distribution D that can help optimize the adversarial classification accuracy in the downstream task. First note that by Bayes rule, the optimal decision boundary for the true distribution x fly A (yy,, o²1) is given by μ = 0. Therefore, we restrict our attention to synthetic data distributions that satisfy the following two conditions:
1. The marginal probability density p(ū) of the synthetic distribution matches p(y) of the real data distribution well.
The conditional probability densities p( = 1) and p = -1) of the synthetic data distri- bution symmetric around the optimal decision boundary μ 0.
More specifically, we consider a special case of the synthetic data distribution Dx = 0,, (-.5((1,,0')
13

---

Under review as a conference paper at ICLR 2023
Proof of Proposition 7 We follow the proof strategy in Carmon et al. (2019). Let b be the indicator that the 2-th pseudo-label Yi assigned to a, is incorrect, so that we have 2; ~ N ((L 2b,) Bift, o?1). Let 7 == 2/2 - (1 - 2bx) E [-1,1] and a ;= ū+n"
a
Note that the
true data samples az N (Vila, IlI) thus we may write the final estimator as
1
Ornal
yj
7
= aa + n+ nt
Ú.g. + (1 a)p +
Σ yiei
+ 7t
km
an + (1 a
Ui +
U.E:),
i=1
where E, ~ N (0, independent of each other, and the marginal probability density p(ū) matches p(y) well. Defining 8 mmall - ayp (1 (1-aa) a) Note that 8 ~ -N(0, =²
By (6), we have that the standard error fe... is a non-increasing function of 1r O'mea Note that al|0mat
when ń is large enough, have the direction of iial approach the direction of j.. Therefore, the
statement in Case holds as a consequence, and similarly for the robust error according to 7
The remaining proof Case 2 and Case 3 is based detailed discussion for the squared inverse of the term / Omn [0mm II
[mall ||
11 on 1
(8)
(4 Onall )2 (ary(1, (3) + 1 + (1 a 11|2?
Note that the larger the quantity in (8) is, the larger the standard error of fan
Case 2. Assume ja cjs. Then we have 8 reduces to:
O 112 || (1 a e7
(9)
(H nnnal )2
(11 a cya))l TT) 2
| a + cya) 1/ (1-a+ + 4
|||||
(1- - (1-a+ + + + T8)
|| 2
|| (11 5)²
(10)
||
(1- a eya') 1 + 1 TT))
which demonstrates that the bigger the c is, the smaller the standard error erTstandard (fomm) is, which verifies the second part of Case 2.
Case 3. Assume j = c(ps =lla. Similar to Case 2, we rewrite the term inside the robust error
function 7 as:
Ornall 2|
(1aa =1a)
(1 eld) Tl1))Tmia Oimall
2
(11 + eyaa elal + (μ eld) T8)
(1 c1a)
1
1/4 elaد +
2: (11)
(1 - a cya)ll elal? (# Ela) 18)
which demonstrates the bigger the is, the smaller the robust error erT (ffm..) is, which proves the second part of Case 3.
14

---

Under review as a conference paper at ICLR 2023
B MORE SIMULATION RESULTS UNDER GAUSSIAN SETTING IN SECTION 3
In this section, we present more detailed simulation results under the Gaussian setting in Section
3 demonstrate different scenarios in Proposition [ Table 7 and Table show the clean and robust accuracy learned on synthetic distribution f with different angles between and elg- c(f =la) with different angles between Il and eld- Recall that fa is (one of) the optimal linear classifier that maximize the clean accuracy under the true distribution considered in Section E similarly μ ela is the optimal solution for robust accuracy. Therefore, different angles between l4 and c1, represent different trade-offs between the clean and robust accuracy. For example, when the angle between l and e1, is O degrees, i.c., μ cla. we have that the optimal solution for clean accuracy and robust accuracy are the same. In most cases, the classifier learned from the synthetic distribution that is most separable achieves better performance than the iid samples, which verifies Proposition T
Table I0 shows the clean and robust accuracy learned on synthetic distribution
Table 7: The clean and robust accuracy learned synthetic distribution = eu when d = 2 and the angle between fr and с degrees and 90 degrees. "Real" denotes the real data distribution, and n denotes the number of data from the real distribution, while we use "c" denote different synthetic distributions and use îi to denote the number of synthetic data. The results and the standard deviation in the bracket are the results of 50 independent trials.
degree
90 degree
acc (std)
rob acc (std)
acc (std)
rob acc (std)
n 0.9201 (0.0012)
0.7593 (0.0020)
0.9171 (0.0046)
0.7552 (0.0040)
n 20
0.9204 (0.0007)
0.7598 (0.0016)
0.9186 (0.0017)
0.7563 (0.0012)
Real
n = 50
0,9206 (0.0004)
0.7605 (0.0007)
0.9196 (0.0009)
0.7566 (0.0006)
n 100
0.9205 (0.0004)
0.7608 (0.0006)
0.9199 (0.0006)
0.7565 (0.0007)
ñ = 10
0.9159 (0.0099
0.7541 (0.0096)
0.9104 (0.0121)
0.7492 (0.0122)
ñ 20
0.9179 (0.0047)
0.7562 (0.0050)
0.9161 (0.0052)
0.7546 (0.0054)
c 0.5
n 50
0.9200 (0.0023)
0.7586 (0.0024)
0.9183 (0.0022)
0.7570 (0.0022)
n = 100
0,9213 (0.0011)
0.7601 (0.0009)
0.9193 (0.0012)
0.7576 (0.0010)
n
0.9133 (0.0066)
0.7502 (0.0061)
0.9161 (0.0048)
0.7598 (0.0048)
ñ 20
0.9155 (0.0020)
0.7516 (0.0019)
0.9180 (0.0017)
0.7612 (0.0020)
1
ñ 50
0.9161 (0.0009)
0.7525 (0.0006)
0.9186 (0.0010)
0.7620 (0.0006)
π 100
0.9165 (0.0005)
0.7528 (0.0006)
0.9189 (0.0005)
0.7622 (0.0003)
n 10
0,9209 (0.0038)
0.7523 (0.0025)
0.9221 (0.0017)
0.7583 (0.0015)
i 20
0.9228 (0.0010)
0.7536 (0.0006)
0.9226 (0.0013)
0.7588 (0.0013)
C= 1,5
ñ 50
0.9229 (0.0008)
0.7538 (0.0005)
0.9232 (0.0005)
0.7594 (0.0006)
ñ 100
0.9232 (0.0003)
0.7538 (0.0005)
0.9233 (0.0005)
0.7595 (0.0005)
15

---

Under review as a conference paper at ICLR 2023
Table 8: The clean and robust accuracy learned on synthetic distribution ji epe when d = 2 and the angle between l and is degrees and 60 degrees. "Real" denotes the real data distribution, and n denotes the number of data from the real distribution, while we use "c" to denote different synthetic distributions and use ń to denote the number of synthetic data. The results and the standard deviation in the bracket are the results of 50 independent trials.
30 degree
60 degrec
acc (std)
rob acc (std)
ace (std)
rob acc (std)
10
0.8307 (0.0123)
0.6343 (0.0283)
0.8348 (0.0117)
0.6378 (0.0293)
n 20
0.8353 (0.0055)
0.6404 (0.0234)
0.8391 (0.005)
0.6433 (0.0222)
Real
n 50
0.8371 (0.0022)
0.6450 (0.0168)
0.8410 (0.0017)
0.6494 (0.0134)
100 71
0.8385 (0.0010)
0.6461 (0.0097)
0.8413 (0.0013)
0.6522 (0.0102)
Ā 10
0.8265 (0.0184) )
0.6282 (0.0418) )
0.8338 (0.0132)
0.6303 (0.0335 )
7 20
0.8299 (0.0129)
0,6352 (0,0325)
0.8365 (0.0132)
0.6393 (0.0316)
0.5
A 50
0.8372 (0.0046)
0.6483 (0.0215)
0.8414 (0.0034)
0.6489 (0.0199)
100
0.8402 (0.0015)
0.6466 (0.01 10)
0.8431 (0.0012)
0.6510 (0.0135)
â 10
0.8383 (0.0158)
0.6439 (0.0319)
0.8377 (0.0074)
0.6396 (0.0267)
20
0.8425 (0.0060)
0.6480 (0.0218)
0.8416 (0.0034)
0.6513 (0.0178)
1
â = 50
0.8455 (0.0023)
0.6553 (0.0128)
0.8432 (0.0020)
0.6503 (0.0122)
i 100
0.8457 (0.0021)
0.6535 (0.0100)
0.8435 (0.0014)
0.65011 (0.0096)
â 10
0.8431 (0.0045)
0.6542 (0.0173)
0.8368 (0.0073)
0.6446 (0.0213)
71 20
0.8447 (0.0021)
0.6542 (0.0142)
0.8393 (0.0022)
0.6479 (0.0150)
e 1.5
á 50
0.8455 (0.0006)
0.6556 (0.0082)
0.8404 (0.0005)
0.6488 (0.0089)
100
0.8457 (0.0004)
0.6547 (0.0057)
0.8404 (0.0007)
0.6486 (0.0082)
Table The clean and robust accuracy learned on synthetic distribution jı == ep when d = 100 and the angle between a and is 0 degrees. "Real" denotes the real data distribution, and n denotes the number of data from the real distribution, while we use "c" to denote different synthetic distributions and use ñ to denote the number of synthetic data. The results and the standard deviation in the bracket are the results of 50 independent trials.
acc (std)
rob acc (std)
n1 10
0.9023 (0.0192)
0.6843 (0.0359)
n 20
0.9341 (0.0128)
0.7519 (0.0267)
Real
50
0.9599 (0.0028)
0,8078 (0.0061)
100
0.9682 (0.0014)
0.8239 (0.0028)
Ā 10
0.7562 (0.0564)
0,4611 (0.0694)
ñ 20
0,8566 (0.0307)
0.6047 (0.0491)
c 0.5
ñ 50
0.9261 (0.0117)
0.7328 (0.0227)
ñ = 100
0.9505 (0.0047)
0.7848 (0.0111)
ñ
0.8866 (0.0273)
0.6557 (0.0487)
ñ 20
0.9371 (0.0091)
0.7555 (0.0201)
1
ñ 50
0.9620 (0.0028)
0,8085 (0.0060)
ñ 100
0.9695 (0.0012)
0.8239 (0.0031)
ñ 10
0.9400 (0.0100)
0.7603 (0.0233)
20
0.9591 (0.0037)
0.8031 (0.0080)
c= 1.5
71
0.9710 (0.0013)
0.8280 (0.0028)
100
0.9743 (0.0008)
0.8343 (0.0011)
16

---

Under review as a conference paper at ICLR 2023
Table 10: The clean and robust accuracy learned synthetic distribution j dlu =la) when = 2 and the angle between la and 30 degrees and 60 degrees. "Real" denotes the real data distribution, and n denotes the number of data from the real distribution, while use "c" to denote different synthetic distributions and use ň. to denote the number of synthetic data. The results and the standard deviation in the bracket are the results of 50 independent trials.
30 degree
60 degree
acc (std)
rob acc (std)
acc (std)
rob acc (std)
0,9152 (0.0049)
0.7633 (0.0111)
0.9211 (0.0034)
0.7702 (0.0094)
20
0.9170 (0.0030)
0.7642 (0.0075)
0.9225 (0.0020)
0.7714 (0.0068)
Real
50
0.9183 (0.0009)
0.7653 (0.0040)
0.9232 (0.001 1)
0.7711 (0.0050)
100
0.9185 (0.0006)
0.7658 (0.0027)
0.9235 (0.0009)
0.7724 (0.0027)
n 10
0.9089 (0.0183)
0.7563 (0.0310)
0.9111 (0.0114)
0.7638 (0.0172)
20
0.9144 (0.0068)
0.7659 (0.0107)
0.9138 (0.0068)
0.7694 (0.0066)
= 0.5
50
0.9174 (0.0029)
0.7680 (0.0068)
0.9161 (0.0038)
0.7714 (0.0033)
100
0.9183 (0.0016)
0.7681 (0.0053)
0.9165 (0.0031)
0.7727 (0.0014)
0.9135 (0.0116)
0.7642 (0.0194)
0.9069 (0.0111)
0.7677 (0.0109)
ñ 20
0.9178 (0.0046)
0.7710 (0.0073)
0.9042 (0.0098)
0.7676 (0.0072)
c= 1
ñ
50
0.9183 (0.0042)
0.7728 (0.0042)
0.9073 (0.0047)
0.7702 (0.0017)
n 100
0.9196 (0.0017)
0.7733 (0.0036)
0.9059 (0.0039)
0.7698 (0.0016)
n 10
0.9181 (0.0079)
0.7747 (0.0104)
0.9034 (0.0079)
0.7704 (0.0053)
20
0.9209 (0.0053)
0.7770 (0.0052)
0.9077 (0.0059)
0.7716 (0.0056)
c= 155
50
0.9218 (0.0029)
0.7788 (0.0028)
0.9073 (0.0030)
0.7722 (0.0014)
7 100
0.9222 (0.0017)
0.7793 (0.0023)
0.9077 (0.0024)
0.7729 (0.0011)
17

---

Under review as a conference paper at ICLR 2023
C THE DETAILED CONSTRUCTION OF THE CONTRASTIVE LOSS
In this section, we first give a detailed description of several possible ways to design contrastive loss, especially in constructing positive and negative pairs. Then, we give visualization of the synthetic data distributions generated under different contrastive losses.
C.1 POSITIVE AND NEGATIVE PAIR SELECTION STRATEGY.
In this subsection, we give several possible ways to construct positive and negative pairs.
1. Vanilla version: Using all the samples in the minibatch is the common strategy for contrastive learning. In the diffusion process, since for each time step i, we want to distinguish each image from other images in the minibatch the same time step. straight-forward forward strategy is to use all the samples in the minibatch other than æ; at time step t to be the negative pairs. For the positive pairs, we can simply adopt #/+1 to be the positive pairs rather than augmentation of æ
Real data as positive pairs: A possible improvement upon the vanilla version is considering aim to generate images similar to real data. Therefore, we can directly adopt the real data as the positive pairs.
3. Real data negative pairs: Another improvement upon the vanilla version is considering the other images in time step in the minibatch not as high quality as the real data. Therefore, we directly adopt the real data the negative pairs.
4. Class conditional version: When minibatch is available, a further improvement can be adopted is to use all the samples with dif- ferent class label y in the minibatch at time step to be the negative pairs.
conditional diffusion, and the class label of æ; in the
C.2 VISUALIZATION THE SYNTHETIC DATA DISTRIBUTION GENERATED BY DIFFERENT DESIGNS OF THE CONTRASTIVE LOSS
In this subsection, we demonstrate the synthetic distributions generated by different designs of the contrastive loss mentioned in Section C on the Gaussian setting mentioned in Section 3.1 Fig-
ure 3 shows the synthetic distribution generated by using (0,11) as initialization, while Figure shows the synthetic distribution generated by using (0, 41) as initialization. In all figures, all of the contrastive loss except for conditional hard negative mining form circle within each class, which means these algorithms cannot explicitly distinguish the data within the same class and thus maximize the distance within each class, while the guidance from conditional hard negative mining generate samples that are more distinguishable.
18

---

Under review as a conference paper at ICLR 2023
(a) InfoNCE
(b) Hard negative mining
(c) Hard negative mining (real data as positive (d) Hard negative mining (real data as negative pair)
pair)
2.0
1.5
0.
0.5
-3.5
-35
120
-20
-1.3
-3.9
-05
9.0
03
1.0
1.3
2.0
(c) Conditional inforNCE
(f) Conditional hard negative mining
Figure 3: A comparison of the synthetic distribution guided by different contrastive loss with initial- ization A 1). Real data positive pair means using the mixture of oracle distribution ( 1+111) and the data in the same batch as negative pair, while real data as negative pair means using the data in the same batch as positive pair and using the mixture of oracle distribution negative pair.
19

---

Under review as a conference paper at ICLR 2023
(a) Diffusion
(b) InfoNCE
(c) Hard negative mining
(d) Hard negative mining (true data as positive pair)
(e) Hard negative mining (true data as negative pair)
(0) Conditional hard negative mining
Figure 4: A comparison of the synthetic distribution guided by different contrastive loss with initial- ization A 4Il).
20

---

Under review as a conference paper at ICLR 2023
D THE EXPERIMENTAL RESULTS IN THE HEAL-WORLD SETTING
D.1 EXPERIMENTAL SETUP FOR CIFAR- 10 DATASET
We describe the pipeline of synthetic data generation for adversarial robustness and corresponding setting for the CIFAR- 10 dataset this subsection.
Dataset. CIFAR-10 10 dataset Krizhevsky (2009) contains 50K 32x32 color training images in 10 classes and 10K images for testing.
Overall training pipeline We follow the same training pipeline as Gowal et al. (2021 i.e., syn- thesizing data by using the diffusion model, assigning pseudo-label for synthetic data and aggregate the original data and the synthetic data for adversarial training. We give a careful explanation of these three components as follow.
Synthetic data generation by the diffusion model. Considering the advantage of DDIM gen- eration speed, we base on the official implementation of the DDIM model (Song et al. 202 Ta and add the guidance of the contrastive loss. We generate images with 200 steps with batchsize=512, and use the quadratic version of sub-sequence selection 3) For the guidance of the contrastive loss, we try different designs of the contrastive loss mentioned in Section 4.2 We set the temperature = 0.1 and the strength of guidance of the contrastive loss X = 20 in the InfoNCE loss, while 10, the strength of guidance of the contrastive loss À 100k, the probability of the same class in the minibatch ,+ = 0.1 and the hardness of negative mining 3 1 in hard negative mining loss. These corresponding hyperparameters are chosen based on some preliminary experiments on image generation. The detailed ablation studies can be found in Section 5.2 Moreover, also delve into the representation used by contrastive loss. The default setting is to use the pre-traines Wide ResNet-28- 10 model Gowel et al. 2021) get the representation for applying the contrastive loss, which is named as (without embedding) in Section [5.2] A further improvement is to apply a 2-layer feed-forward neural network to encode the representation after the pre-trained model, which is named as (with embedding). The advantage of the latter design is we can adopt the contrastive loss optimize the encoding network rather than fixed encoder.
LaNet for assigning pseudo-label. Since the DDIM an unconditional generator, we need to assign the pseudo-label to the generated sample. We follow the same choice adopted by [Sehwag] al. (2022), i.e., using state-of-the-art LaNet Wang et al. (2019) network for assigning the pseudo- label for the synthetic data.
Adversarial Training. We follow the same setting Gowal et al. (2021). i.e., use Wide ResNet-28-115 Zagoruyko & Komodakis (2016) with Swish activation function Hendrycks & Gim- pel (2016), adopt stochastic weight averaging Izmailov et al. (2018) with decay rate 0.995 and use TRADES ZZang et al. (2019) with 10 Projected Gradient Descent steps and &xc 8/255 for 400 epochs with batch size 1024
Evaluation setup We follow the same training pipeline as Gowal et al. 202 i.c.. for each trained
model, we adopt either AUTOATTACK Croce Hein 2020) or AA+MT Gowal et al. (2020) with
8/255 and report the worst accuracy as robust accuracy.
D.2 EXPERIMENTAL SETUP FOR TRAFFIC SIGNS DATASET
We describe the pipeline of synthetic data generation for adversarial robustness and corresponding setting for the Traffic Signs dataset in this subsection.
We refer to Appendix D.2 for a detailed explanation of the quadratic version. *oor Table in the ablation studies subsection, we batch size with 256.
21

---

Under review as a conference paper at ICLR 2023
Dataset. Traffic Signs dataset Houben al. (2013) contains 39252 training images in 43 classes and 12629 images for testing, and the image sizes vary between 15x15 to 250x250 pixels.
Synthetic data generation by the difftusion model. To utilize the pre-trained diffusion model | we use a conditional DDPM for generating samples for Traffic Signs dataset. We adopt the hard 10, the strength of guidance of the contrastive loss
negative mining loss with probability of the same class in the minibatch r* 0.1 and the hardness of negative mining & = 1. We also use the pre- pre-trained Wide ResNet-28- 10 model get the representation for applying the contrastive loss and use a 2-layer feed-forward neural network to encode the representation after the pre-trained model.
5k, the
Adversarial Training. We follow the same setting the CIFAR-10 dataset, except the training epochs are reduced to 50. We also extend the training epochs to 400 but do not find significant improvement.
D.3
THE DETAILED EXPLANATION THE DATA SELECTION METHODS
Below we summarize different data selection methods:
DDIM (Separability): We adopt the separability of the data as criterion to make the selection of the data generated by vanilla DDIM. For each data, we use a pre-trained WRN-28-10 model to encode them into the embedding space. Then, we compute the 1.2 distance between each sample and the centroid of all classes (which is easily computed the mean of all samples in this class) and add them together. To select subset of samples that are most distinguishable, we choose the top samples that have the smallest distance each class.
Contrastive-DP (Gradient norm): We the gradient norm with respect to pre-trained WRN- 28- 10 model as a criterion to make the selection on the data generated by Contrastive-DP. The larger the gradient norm is, the more informative the sample is for learning downstream model. Therefore, we select the top K samples that have the largest gradient norm in each class.
Contrastive-DD (Gradient norm-rob): Similar to Contrastive-DD (Gradient norm), we use the gra- dient norm of the robust loss rather than standard classification loss criterion to make the selection on the data generated by Contrastive-DP Therefore, we select the top K samples that have the largest gradient norm in each class.
Contrastive- DP (Entropy): We the entropy of each sample with respect to LaNet a criterion to make the selection on the data generated by Contrastive- The smaller the entropy is, the higher likelihood this image has good quality. Therefore, select the top K samples that have the smallest entropy in each class.
D.4 COMPARISON OF THE IMAGE GENERATED BY CONTRASTIVE-GP WITH THE VANILLA DDIM
In this subsection, we visualize the image generated by Contrastive- Gatrrasstvee-Paaa. and the vanilla DDIM on the CIFAR- 10 dataset. We find the guidance of the contrastive loss changes the category of the synthetic images makes the synthetic images realistic (colorful).
D.5 T-SNE ON THE CLASSIFIER LEARNED ON SYNTHETIC DATA.
In this subsection, we visualize the t-SNE of the finial classifier learned on different synthetic data. We find with the guidance of the contrastive loss, the finial classifier learns better representation that makes the feature of the images from different classes more separable than the finial classifier learned on the images generated by the vanilla DDIM.
22

---

Under review as a conference paper at ICLR 2023
(a) DDIM
(b) Contrasrive-DP
Figure 5: Comparison of the Contrastive-GP with the vanilla DDIM. The image in the same position on subfigures (a) and (b) has the same initialization. With the guidance of the contrastive loss, the category of the synthetic images changes, the synthetic images become more realistic (colorful), which demonstrates the effectiveness our Contrastive DP algorithm.
50
arplames
-50
dter
53
doge
Abo
Trag
frag
-100
-10
DDIM
(b) Contrastive
Figure 6: A comparison of the T-SNE of the finial classifier learned different synthetic data the CIFAR-10 dataset.
23


Reviews:
[{"review_id": "vsLOyo_y0s", "reviewer": "Reviewer_H9p7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a diffusion model-based data augmentation method. The proposed method combines the contrastive loss with the diffusion model generation process to generate data with a large diversity between different classes. It shows that using the proposed method generates data and trains a classifier, the trained classifier is more robust to adversarial attacks than DDIM, the SOTA diffusion model.    ", "summary_of_the_review": "This paper proposes a new defense to train adversarially robust classifiers. As mentioned above, the descriptions of the proposed method are not entirely clear, and the technical contribution is not that significant. Besides, the empirical improvement is marginal and is not well justified. ", "strength_and_weaknesses": "Strengths\n+ The paper provides a clear description of the background. \n+ The theoretical analysis part is clearly described. \n\nWeaknesses:\n- Regarding the theoretical analysis, as mentioned above, I appreciate that the analysis part is clear. However, I have the following two questions: (1) For the simulation test, the authors mainly focus on the case $\\hat{\\mu}=c\\mu$. The case that improves the clean accuracy. I am wondering why not verify the other case $\\hat{\\mu}=c(\\mu - \\epsilon 1_d)$ that improves the robust accuracy. (2) It would be great if the authors could provide more justification for the conclusion drawn from the theoretical analysis: \"The synthetic data can help improve the classification task especially when the representation of different classes is more distinguishable in the synthetic distribution.\" From the analysis, the most obvious conclusion one could draw is under what situation the clean accuracy is the best ( $\\hat{\\mu}=c\\mu$), and under what situation, robust accuracy is the best. This situation is based on the difference between the synthetic distribution and the original distribution. It is not that obvious between the connection between the above situations to the difference between the representations of different classes in the synthetic data.  \n\n- The proposed technique is not entirely clear to me. The proposed data augmentation is clear. But the overall training process is not clear. I am not sure what the procedure would be like. It either generates data using the proposed method and uses the augmented dataset to train the model or generates data using the proposed method and applies adversarial training on the augmented data, or others. My understanding is the second one. It would be great if the authors could clearly describe the overall procedure. If my understanding is correct, it brings a new question, whether the robustness comes from data augmentation or adversarial training. The paper does not justify this. \n\n- The evaluation setup is not clear enough, mainly how to generate attacks. My understanding is training a model using the proposed method and then generating attacks based on the trained model. Another way is to generate attacks on a vanilla model and use those adversarial samples to test models trained by the proposed method. It would be great if the authors could make this setup clear. The improvement over DDPM and DDIM is marginal (BTW, what is the reason for using DDIM for CIFAR-10 and DDPM for Traffic sign data). I would also suggest reporting the mean and standard errors of multiple runs to show the improvement is not because of randomness.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nJ4VFehjhF8", "reviewer": "Reviewer_96gf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the setting where supplemental synthetic data is used to augment the training sample and improve generalization. Specifically, authors focus on the adversarial setting where prediction errors consider worst case perturbations within a neighbourhood of test points. In particular, in a simplified setting, authors verified necessary properties of the synthetic data source in order fo it to yield performance gains: synthetic class conditionals must be far from their real counterparts, while still being separable by the same the decision boundary as the original data. They then use this insight and define image generation procedures that generate samples that, while still being perceptually close to the data distribution, it is as far as possible from it. Empirical assessment shows improvements when using the proposed \"guided\" generation approach combined with adversarial training.", "summary_of_the_review": "The paper tackles a relevant problem and proposes practical approaches to guide generative processes towards samples that do not trivially match the training data. However, it's unclear to me whether the observed improvements are due to simply increasing the sample size or to the specific properties of the generated data. Moreover, I'm not confident that results on CIFAR-10 are at least on par with recent work under similar settings.", "strength_and_weaknesses": "Pros:\n\n+The paper tackles relevant problems from a practical perspective: how to best leverage synthetic data in order to obtain robust classifiers.\n\n+The practical approach is motivated from findings obtained formally in a simplified setting, rather than solely based on intuition.\n\n+The approach used to sample synthetic data might be more generally applicable in settings other than the ones covered in the paper.\n\nCons:\n\n-It's unclear whether observed improvements are due to the properties of the synthetic data or the simple fact the training sample became bigger. I would suggest an experiment where you train models on a subsample of the training data (say half of it) supplemented by synthetic data up to the same size as the original sample. Comparing this model with the one training on the full training sample would help control for the contribution of the sample size.\n\n-Results aren't contextualized with past work in that numbers reported in previous work are not reported. In particular, baselines results without additional data in table 2 seem weak (entries 7 and 11 in this leaderboard seem so use the same architecture and threat model: https://robustbench.github.io/#div_cifar10_Linf_heading).\n\n-It's also unclear to what extent the results observed in the simplified setting transfer to real data. It seems to me the results are a direct consequence of the choice of model class, so it seems the optimal synthetic data source is the one that introduces points lying far from the decision boundary. I wonder how conclusions would change had a different model class been chosen, e.g., SVMs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TunQIIhe7J", "reviewer": "Reviewer_4Hr9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, Contrastive-Guided Diffusion Process (Contrastive-DP) for robust training is presented. Using Contrastive-DP, a diffusion model to generate new data could be created. For a classification task, using a contrastive setup, better results than DDIM are obtained. The article is heavily inspired by Gowal et al. (2021) and (Song et al., 2021a). The most prominent point is using both real data and synthetic data as in (Carmon et al., 2019) and the effort to select discriminative features. Thus, it can be viewed as an extension to the previous work.", "summary_of_the_review": "As the method is not explained very clearly, the paper lacks necessary experiments, and explanations, which are explained above, and the results seem to produce incremental gains or none, the contribution of the paper is not adequate for ICLR.", "strength_and_weaknesses": "(+)\n- The idea of generating more discriminative images to obtain more robustness is powerful and valid.\n- The reasoning for selecting new synthetic data in a contrastive framework is valid.\n\n(-)\n- The inspiration comes from previous work, the idea is incremental.\n- In Table 1, no better results for robust accuracy on “Real data/d=2” is obtained. Also same values are repeated for “Real” and “c=0.5”. \n- There is a problem in Algorithm 1. For Lines 5-6, a more clear explanation is needed. At Line 5, z is created and it is not clearly shown how it is used.\n- The results are presented for only WRN-28-10. The results should be presented for recent neural network architectures that achieve state-of-the-art performance values on the used datasets.\n- Song et al.’s 12th formula is used as the 4th formula in the paper. However no random noise is applied to make it deterministic here. Ablation study for nondeterministic setups should also be included. \n-DDIM also used CelebA, LSUNBedroom and LSUNChurch datasets, i.e. not only CIFAR-10 and Traffic Signs. In this work, the authors  should also demonstrate the performance on those relatively more complex datasets.\n- Results given in CIFAR-10 are incremental. Also in Table 3, there is nearly no difference between DDPM and Contrastive-DP. Also no std. values are given in the tables. Multiple initializations and training runs should be run and the standard deviations should be reported in the tables.\n-Time complexity should be analyzed. Using Contrastive-DP instead of DDIM, there may be a tradeoff between time and accuracy. Is the given incremental results worthy of it?\n- Figure 1 is hard to understand. Should be explained clearly in its caption.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject