A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training
Authors: Yifei Wang
Yisen Wang
Jiansheng Yang
Zhouchen Lin
Venue: ICLR 2022 Poster (2022)
Keywords: Generative Models
Energy-based Models
Sampling
Adversarial Training

Abstract: Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently researchers notice that robust models with AT have good generative ability and can synthesize realistic images, while the reason behind it is yet under-explored. In this paper, we demystify this phenomenon by developing a unified probabilistic framework, called Contrastive Energy-based Models (CEM). On the one hand, we provide the first probabilistic characterization of AT through a unified understanding of robustness and generative ability. On the other hand, our unified framework can be extended to the unsupervised scenario, which interprets unsupervised contrastive learning as an important sampling of CEM. Based on these, we propose a principled method to develop adversarial learning and sampling methods. Experiments show that the sampling methods derived from our framework improve the sample quality in both supervised and unsupervised learning. Notably, our unsupervised adversarial sampling method achieves an Inception score of 9.61 on CIFAR-10, which is superior to previous energy-based models and comparable to state-of-the-art generative models.

Published as a conference paper at ICLR 2022
A UNIFIED CONTRASTIVE ENERGY-BASED MODEL FOR UNDERSTANDING THE GENERATIVE ABILITY OF ADVERSARIAL TRAINING
Yifei Wang' Yisen Wang2.3 Jiansheng Yang' Zhouchen Lin
1 School of Mathematical Sciences, Peking University
2 Key Lab. of Machine Perception (MoE), School of Artificial Intelligence, Peking University
3 Institute for Artificial Intelligence, Peking University
4 Pazhou Lab, Guangzhou, 510330, China
ABSTRACT
Adversarial Training (AT) is known effective approach to enhance the ro- bustness of deep neural networks. Recently researchers notice that robust mod- els with AT have good generative ability and can synthesize realistic images, while the reason behind it yet under-explored. In this paper, we demystify this phenomenon by developing unified probabilistic framework, called Contrastive Energy- Models (CEM). On the one hand, we provide the first probabilistic characterization of AT through a unified understanding of robustness and gener- ative ability. On the other hand, our unified framework can be extended the unsupervised scenario, which interprets unsupervised contrastive learning as an important sampling of CEM. Based these, we propose a principled method to develop adversarial learning and sampling methods. Experiments show that the sampling methods derived from our framework improve the sample quality in both supervised and unsupervised learning. Notably, our unsupervised adver- sarial sampling method achieves an Inception score of 9.61 on CIFAR-10, which is superior previous energy-based models and comparable to state-of-the-art generative models.
INTRODUCTION
Adversarial Training (AT) is one of the most effective approaches developed so far to improve the robustness of deep neural networks (DNNs) (Madry et al. 2018). AT solves minimax optimization problem, with the inner maximization generating adversarial examples by maximizing the classifica- tion loss, and the outer minimization finding model parameters by minimizing the loss on adversarial examples generated from the inner maximization (Wang et al., 2019). Recently, researchers have noticed that such robust classifiers obtained by AT are able to extract features that are perceptually aligned with humans (Engstrom et al. 2019). Furthermore, they are able synthesize realistic im- ages on par with state-or-the-art generative models (Santurkar et al.) 2019). Nevertheless, it is still mystery why AT is able learn more semantically meaningful features and turn classifiers into generators. Besides, AT needs the labeled data (x,,y:) for training while canonical deep genera- tive models do not, e.g., VAE (Kingma & Welling 2014) and GAN (Goodfellow al. 2015) only require {x, Thus, it is worth exploring if is possible to train a robust model without labeled data.
Several recent works (Jiang et al. 2020 Kim et al. 2020 Ho & Vasconcelos 2020) have proposed unsupervised AT by adversarially attacking the InfoNCE loss (Oord et al.) 2018) (a widely used objective in unsupervised contrastive learning), which indeed improves the robustness of contrastive encoders. However, depth investigation and understanding for unsupervised AT is still missing.
To address the above issues, in this work, we propose a unified probabilistic framework, Contrastive Energy-baset Models (CEM), that provides principled understanding the robustness and the generative ability of different training paradigms. Specifically, we make the following contributions:
Demystifying adversarial training and sampling. We firstly propose a probabilistic in- terpretation for AT, that is, it is inherently (biased) maximum likelihood training of the
"Corresponding author; Yisen Wang (yisen.wang@e @ pku.edu.cn).

---

Published as a conference paper at ICLR 2022
corresponding energy-based model, which explains the generative ability of robust models learned by AT. Inspired by this, we propose some novel sampling algorithms with better sample quality than previous methods.
A unified probabilistic framework. Based on the understanding above, we propose Con- trastive Energy-based Model (CEM) that incorporates both supervised and unsupervised learning paradigms. Our CEM provides a unified probabilistic understanding of previous standard and adversarial training methods in both supervised and unsupervised learning.
Principled unsupervised adversarial training and sampling. Specifically, under our proposed CEM framework, we establish the equivalence between the importance sampling of CEM and the InfoNCE loss of contrastive learning, which enables us to design principled adversarial sampling for unsupervised learning.
Notably, show that the sampling methods derived from framework achieve state-of- the-art sample quality (9.61 Inception score) with unsupervised robust models, which is comparable to both the supervised counterparts and other state-of-the-art generative models.
2 RELATED WORK
Robust generative models. Researchers recently notice that features extracted by robust classifiers are perceptually aligned with humans, while standard classifiers are not (Engstrom al. 2019 Kaur et al. 2019 Bai et al. (021). Santurkar et al. (2019) show that we can also generate images of high quality with robust classifiers by iterative updating from randomly sampled noise, where the resulting sample quality is comparable to the state-of-the-aa generative models like BigGAN (Brock et al. 2018).
Contrastive learning. Oord et al. (0188) firstly propose unsupervised contrastive learning by max- imizing a tractable lower bound on mutual information (MI), i.e., the negative InfoNCE loss. How- ever, later works find that the lower bounds degrade lot with large MI. and the success of these methods cannot be attributed to the properties of MI alone (Poole et al., 2019 Tschannen et al.) 2020). Our work provides alternative understanding of unsupervised contrastive learning im- portance sampling of an energy- based model, which also enables us to characterize the limitations of existing methods from a new perspective. In fact, contrastive learning also be seen general learning framework beyond the unsupervised scenarios. For example, SupContrast (Khosla et al.) 2020) extends contrastive learning to supervised scenarios. Our work further bridges supervised, unsupervised and adversarial contrastive learning with a unified probabilistic framework.
CEM: A UNIFIED PROBABILISTIC FRAMEWORK
Inspired by previous work that bridges discriminative models with energy- based models (Grathwohl et al. 2019), in this work, we propose a unified framework, called Contrastive Energy-based Model (CEM), that incorporates both supervised and unsupervised scenarios.
Our proposed CEM is a special Energy-basee Model (EBM) that models the joint distribution (u, v) over two variables (u, v) with a similarity function So v) defined in a contrastive form,
exp (u, exp(fo(u,v) v))
(u v)
Z(0)
where Z(0) = exp (u, v)) dudv is the corresponding partition function. In other words, in CEM, a pair of samples (u, v) has higher probability if they are more alike. In particular, it can be instantiated into the two following variants under different learning scenarios.
Parametric CEM. In the supervised scenario, we specify the Parametric CEM (P-CEM) that models the joint distribution (x, y) of data and label y in the following form,
pe(x, y) = exp(So(x. w)) exp(go(x) ww) 2(0)
(2)
Z
where ge: : Rnn - m denotes the encoder, g(x) E Th is the representation of x, and W E IRm refers to the parametric cluster center of the k-th class. Denote the linear classification weight as
2

---

Published as a conference paper at ICLR 2022
W [W...... wg] and the logit vector hxx) g(x) T WW, we see the equivalence between P-CEM and JEM (Grathwohl et al., 2019) as
fo y) go (x) Wy = hx(x)
(3)
Pon-Parametric CEM. In the unsupervised scenario, we do not have access to labels, thus we instead model the joint distribution between natural samples (x,x') x')
(x, cxp(fo(x,x')) exp (x) T (x())) Z(0)
px(x, =
(4)
Z(0)
and the corresponding likelihood gradient of this Non-Parametric CEM (NP-CEM) is
VoEpa(xxx) log Po(, x') = (x
Vofo( x').
(5)
In contrastive P-CEM that incorporates parametric cluster centers, the joint distribution of NP- CEM is directly defined based on the feature-level similarity between the two instances (x, x'). We define the joint data distribution pa(x, x') Pa(x)pa(x') through re- parameterrizzinnn.
x (o((x)))
- T, x ~~ <x)xx
(6)
where u.a.r. denotes sampling uniformly at random and T refers to set of predefined data aug- mentation operators {t: IR. ]R]. ). For the ease of exposition, we assume the empirical data distribution (x) is uniformly distributed over finite (but can be exponentially large) set of natural samples X.
SUPERVISED SCENARIO: REDISCOVERING ADVERSARIAL TRAINING AS MAXIMUM LIKELIHOOD TRAINING
In this section, we investigate why robust models have good generative ability. The objective of AT is to solve the following minimax optimization problem:
lcE yı: 0) SI
min (x.(v.))
max
where (CE(8, y: 0) - log (o(y)))
The inner maximization problem is to find an adversarial example & within the f--norm e-ball around the natural example x that maximizes the CE loss. While the outer minimization problem is to find model parameters that minimize the loss on the adversarial examples x.
4.1 MAXIMIZATION PROCESS
For the inner maximization problem. Projected Gradient Descent (PGD) (Madry et al. 2018) is the commonly used method, which generates the adversarial example & by maximizing the CE loss (i.e., minimizing the log conditional probability) starting from &n = x:
xn+1 Sx, + a ((xn, y; 0) X, VVx. log po(((,)))))
x, + V
log
exp (op(fo(Xn..))) b))
VV, fo(nn. v),
while the Langevin dynamics for sampling P-CEM starts from random noise xo ô and updates with
X x Xn + a V log (x V
Xn + V
log exp([o(Xxn k)) + /2a e.
Eqns. 8 and 9 both have a positive logsumexp gradient (the second term) to push up the marginal
probability Po (x). As for the third term, PGD starts from a data point (x, such that it requires the repulsive gradient be away from the original data point and do the exploration in local
Note that omit the projection operation and the gradient re-normalization steps.
3

---

Published as a conference paper at ICLR 2022
region. Langevin dynamics instead starts from random noise and an additive noise is injected for exploration.
Comparing PGD and Langevin. Following the above analysis, the maximization process in AT can be seen as (biased) sampling method that draws samples from the corresponding probabilistic model po ()) Compared to Langevin dynamics, PGD imposes specific inductive bias for sampling. With the additional repulsive gradient and e-ball constraint, it explicitly encourages the samples be misclassified around the original data points. In practice, adversarial training with such adversarial examples is generally more stable than training JEM with Langevin samples, which indicates that PGD attack is a competitive alternative for the negative sampling method for JEM training.
4.2 MINIMIZATION PROCESS
To begin with, the gradient of the joint log likelihood for P-CEM can be written as follows:
Vol (x..)) log px y)
Bpa(x)))) Vofo y) (x) (xf
(10)
-B pe(x,))) Vefo(x, v) -E) (x) x) Vefo(X, 0),
where (x. y) px(x, 3) denotes the positive data pair, and (x. 9) px 9) denotes the negative sample pair. As discussed above, the adversarial examples x generated by the maximization process can be regarded as negative samples, and ÿ ~ po (ū)()) denotes the predicted label of x. To see how the maximum likelihood training of P-CEM is related to the minimization process of AT, we add an
interpolated adversarial pair (&, y) into Eq. and decompose it as the consistency gradient and the contrastive gradient:
Po log po Xx v) px & (K) Vole ( 0
Vo
-Epa(x.y (o(k.))) Vefe -
(11)
consistency gradient
contrastive gradient
Next, show that the two parts correspond to two effective mechanisms developed in the adver- sarial training literature.
AT loss. As the two sample pairs in the contrastive gradient share the same input X, that the contrastive gradient can be written equivalently as
Epu(x.)) po (R.
VoJo Vofo(x 0)1 Vofo y) Eno(d x) Vofo 0)]
-Eps(x,) (x) (x
-Epo(x. @ pa (R) Ve log (o()))),
(12)
which is exactly the negative gradient of the robust CE loss (AT loss) in Eq. other words, gradient ascent with the contrastive gradient is equivalent to gradient descent w.F.f the AT loss.
Regularization. As for the consistency gradient, original AT (Madry et al. 2018) simply ignores it. Its variant TRADES (Zhang et al. 2019) instead proposes the KL regularization (L (()p(())llp(( [[]p((())) that regularizes the consistency of the predicted probabilities on all classes, whose optimum implies that p(-1x (+) = p(-1x) + fo x, 3) = fx(
Comparing AT and JEM training paradigms. The above analysis indicates that the minimization objective of AT is closely related to the maximum likelihood training of JEM (Grathwohl et al. 2019). Compared to JEM that decomposes the joint likelihood into an unconditional model po [ and discriminative model Po ()) the decomposition of AT in Eq. iO stabilizes training by introducing an intermediate adversarial pair (x. y) that bridges the positive pair (x, y) and the nega- tive pair (x. f). Besides, it inject the adversarial robustness bias by regularizing the consistency gradient. Together with our analysis on the maximization process, we show that AT is a competitive alternative for training JEM (a generative model) with more stable training behaviors. That explains why robust models with AT are also generative.
4.3 PROPOSED SUPERVISED ADVERSARIAL SAMPLING ALGORITHMS
Our interpretation also inspires principled designs of sampling algorithms for robust classifiers.
4

---

Published as a conference paper at ICLR 2022
Targeted Attack (TA), Previously, to draw samples from robust classifier, (annturar et al. (2019)
utilize targeted attack that optimizes an random initialized input Xo towards a specific class y:
k Xa+1 = &n + VV (o) = X. + a V (Xo, 9) - a log
exp((f((x,
(13)
Compared to PGD attack in Eq. 8 while pushing & towards ÿ. TA has a negative logsumexp gradient that decreases the marginal probability (x)) This could explain why TA is less powerful for adversarial attack and rarely used for adversarial training.
Conditional Sampling (CS). To overcome the drawback of targeted attack, natural idea would be dropping the negative logsumexp gradient. In fact, we can show that this is equivalent sampling from the conditional distribution:
p 9) (xp(fo(x.0)) Zx|r(0) / exp(fo(x, dx)
Zxlx(0)
and its Langevin dynamics takes the form:
En+1 X,, + a V log (x,,)))))) 10) + 22a aa fo + 22 E.
(14)
Samples drawn this way essentially follow an approximated model distribution, po po 0) Pa( Thus, CS can be seen debiased targeted attack algorithm.
Reinforced Conditional Sampling (RCS). Inspired by the above analysis, we can design biased sampling method that deliberately injects a positive logsumexp gradient:
8x+1 =Xn + a 2o, 9) + VVR.. &
log exp(fo(Xx: k)) + 22 E.
(15)
With our designed bias, RCS will sample towards the target class by maximizing pe (xll) (with the (S !) term), and at the same time improve the marginal probability (x) (with the logsumexp
term). As shown in experiment, RCS indeed obtains improved sample quality.
4.4 DISCUSSION ON STANDARD TRAINING
In the above discussion, we have explained why adversarial training is generative from the perspec- tive of CEM. In fact, it can also help characterize why classifiers with Standard Training (ST) are not generative (i.e., poor sample quality). key insight that if we replace the model distribution (x( with the data distribution pa(x) in Eq. 10 we have
Volpu(x..) log Pe (x. y) Bx)(x.)) Vefe(x, px (ofo(x,
BPpa(x.. Volo(x, v) (x)p (+lx) VoJo (x, (x, 5) = VoRpa(x.N) log (y)xx,
(16)
which is the negative gradient of the CE loss in Eq. 7 Thus, ST is equivalent to training CEM by (x( with data samples x ~ pa(x). This
simply replacing model-base negative samples & approximation makes ST computationally efficient with good accuracy natural data, but signifi- cantly limits its robustness on adversarial examples (as model-based negative samples). Similarly, because ST ignores exploring negative samples while training, standard classifiers also fail to gen- erate realistic samples.
EXTENSION OF ADVERSARIAL TRAINING TO UNSUPERVISED SCENARIO
In this section, we show that with our unified framework, we can naturally extend the interpretation developed for supervised adversarial training the unsupervised scenario.
5.1 UNDERSTANDING UNSCPERVISED STANDARD TRAINING THROUGH CEM
InfoNCE. Recently, the following InfoNCE loss is widely adopted for unsupervised contrastive
learning of data representations (Oord al. 2018 Chen et al. 2020 He al. 2020
exp exp(fo(x,x'))
(NCE(x,x, K 0) = log KK exp(f(((((
(17)
5

---

Published as a conference paper at ICLR 2022
where fx(x, x) g0 g0(x)) (xx calculates the similarity between the representations of the two data samples, x, x' are generated by two random augmentations (drawn from 7) of the same data example, and {x, ) denotes K independently drawn negative samples. In practice, one of the
K negative samples is chosen to be the positive sample x'. Therefore, InfoNCE can be seen as an instance-wise K --class cross entropy loss for non-parametric classification.
Perhaps surprisingly, we show that the InfoNCE loss is equivalent to the importance sampling esti- mate of our NP-CEM (Eq. 4) by approximating the negative samples from (x with data samples from Pa(x), as what we have done in standard supervised training (Section 4.4
xx)(xx)) (ofo(x,x')) (xx(x*))) Vofo (x, x')
exp(So (X, x)))
= =pp((xx)) Vafo(x, x') (x)( E (x) exp(ff(8., x))
Vofo(xxX))
exp(So(x 8)))
pxxxx xx) (pa())na(8)) Bp)(() exp(fo(X x)) Vofo(8,x*)
(18)
=xp((xx)) Vo log exp(fo(x, x'))
exp(fo(x,, x,))
N Vo log
Epu(se) (x,,
Σ
x,))
which is exactly the negative gradient of the InfoNCE loss. In the above analysis, for an empirical estimate, we draw N positive pairs (x,, x() ~ pa(x, x'), and for each anchor X;, we further draw K' negative samples {K independently from pa(x').
Remark. As po (x, x') pe (N) (x). the negative phase of NP-CEM is supposed to sample x' from pe ( where samples semantically close the anchor sample x, a.k.a. hard negative sam- ples, should have high probabilities. However, InfoNCE adopts a non-informative uniform proposal pa(&') for importance sampling, which is very sample inefficient because most samples useless (Kalantidis et al., 2020). This observation motivates us to design more efficient sampling scheme for contrastive learning by mining hard negatives. For example, Robinson et al. (2021) 1) directly replace the plain proposal with po (x x') exp(3fo(* x)))
while keeping the reweighing term. From the perspective of CEM, the temperature / introduces bias that should be treated carefully. In all, CEM provides principled framework to develop efficient contrastive learning algorithms.
5.2 PROPOSED UNSUPERVISED ADVERSARIAL TRAINING
AT is initially designed for supervised learning, where adversarial examples can be clearly defined by misclassification. However, it remain unclear what is the right way to do Unsupervised Adver- sarial Training (UAT) without access to any labels. Previous works (Jiang et al. 2020 Ho Vas- concelos 2020 Kim et al. 2020) have carried out UAT with the adversarial InfoNCE loss, which works well but lacks theoretical justification. Our unified CEM framework offers principled way to generalize adversarial training from supervised to unsupervised scenarios.
Maximization Process. Sampling from pe (x) can be more difficult than that in supervised scenarios because does not admit a closed form for variable x'. Thus, we perform Langevin dynamics with K negative samples (Xg) } drawn from pa(x'),
&nt1 Xn + a x..
log (x) + V2a
(19)
K
1
Xn aVx, x.
log
po po X(i)
+ V2a.
K
K
Xn + Vx.
log Σ
+ V2aa.. (Xn, x(6)
km
While the PGD attack of the InfoNCE loss (Eq. 31
exp( (X
Xn+1 Xn a x, log
(20)
1= exp( (Xxn x(,)
=x, + aaax &n.
log
exp (ap(fo(Xxoo (xn Kai) - -aVofo(Xni
1=1

---

Published as a conference paper at ICLR 2022
resembles the Langevin dynamics as they both share the positive logsumexp gradient that pushes up po (8), and differs by repulse negative gradient ( x') away from the anchor x', which is direct analogy of the PGD attack in supervised learning (Section 4.1) Therefore, we believe that the PGD attack of InfoNCE is a proper way to craft adversarial examples by sampling from po px(x).
Minimization Process. Following the same routine in Section 4.2 with the adversarial example & (8))),
insert interpolated adversarial pair (x, x J and decompose the gradient of NP-CEM into the consistency gradient and the contrastive gradient,
p((xx)( Pogpo(x x')
E (xx(xx)) ② po (8.)
E pxx(xx)) @ ) xx
+ -
(21)
consistency gradient
contrastive gradient
In this way, we can directly develop the unsupervised analogy of AT loss and regularization
(Sec. 4.2). Following Eq. it is easy to see that the contrastive gradient is equivalent to the gradi-
of the Adversarial InfoNCE loss utilized in previous work (Jiang al. 2020 Ho & Vasconcelos,
2020 Kim et al. 2020) with adversarial example &,
E px(xx') @ (*) Vefe( x - (x)
Pe (x
E px(xxx)) X (8)
Volo(8, x') Epo (80) (2)
Valo(2,8))
po (x
VoJo(&,, x') - (x') pe p0 8')
pa(xx') & (2)
px(
exp(Jo x())
Vo log
(22)
2k=1 exp(Jo x(,)))
where (xx) denotes the adversarial negative samples from po (xx).
5.3 PROPOSED UNSUPERVISED ADVERSARIAL SAMPLING
In supervised learning, natural method draw sample x from robust classifier is maximize its conditional probability w.r.. a given class ÿ ~ Pa(ū). iee, maxş p( x), by targeted attack (Santurkar et al. 2019). However, in the unsupervised scenarios, we do not have access to labels, and this approach not applicable anymore. Meanwhile, Langevin dynamics is also not directly applicable (Eq. 19) because it requires access to real data samples.
Nevertheless, still find effective algorithm for drawing samples from unsuper- vised robust model. We first initialize a batch of N samples B = {x,,,,- from a prior distribution px(x) (e.g., Gaussian). Next, we update the batch jointly by maximizing the empirical estimate of
MaxEnt.
entropy, where we simply take the generated samples at B = {x as samples from (x
H(pe)= = (x) log x0
1 (o()) N
Σ
log Σ ex,( + log Z(0). (23)
Specifically, we update each sample x; by maximizing the empirical entropy (named MaxEnt)
N
X = x; + vx H(po) + V2a . e e x
aV
log
exp(Jo(x,,x)) + V2a e. (24)
As a result, the generated samples { are encouraged to distribute uniformly in the feature space with maximum entropy, and thus cover the overall model distribution with diverse semanties.
6 EXPERIMENTS
In this section, we evaluate the adversarial sampling methods derived from our CEM framework, showing that they can bring significant improvement the sample quality of previous work. Be- sides, in Appendix A we also conduct a range of experiments on adversarial robustness to verify probabilistic understandings of AT. We show adversarial training objectives derived from our
7

---

Published as a conference paper at ICLR 2022
Table 1: Inception Scores (IS) and Fréchet Inception Distance (FID) of different generative models. Results marked with are taken from Shmelkov et al. (2018).
Method
11)
FID (L)
Auto-regressive
PixeICNN++ (Salimans et al. 2017
5.36
119.5
GAN-based
DCGAN* (Radford al. 2016
6.69
35.6
WGAN-GP (Gulraann cf al. 2017
7.86
36.4
PresGAN (Dieng al. 2019
52.2
StyleGANZ-ADA (Krras et al.
2020
10.02
Score-based
NCSN (Sog & Ermon 2019
8.87
25.32
DDPM (Ho et 2020
9.46
3.17
NCSN++
Song ct al. 2020
9.89
2.20
EBM-based
JEM (Grathwohl et al.
2019)
8.76
38.4
DRL (Ga al. 202T
8.58
9.60
AT-based
TA (Santurkar al. 2019) (w/ ResNet50)
7.5
Supervised CEM (w/ ResNet50)
9.80
55.91
Unsupervised CEM (w/ ResNet18) (ours)
8.68
36.4
Unsupervised CEM (w/ ResNet50) (ours)
9.61
40.25
CEM can indeed significantly improve the performance AT in both supervised and unsupervised scenarios, which helps justify our interpretations and our framework.
Models. For supervised robust models, adopt the same pretrained ResNet50 checkpoin on CIFAR-10 as Santurkar et al. (2019) for a fair comparison. The model adversarially trained with l2-norm PGD attack with random start, maximal perturbation norm 0.5, step size 0.1 and 7 steps. As for the unsupervised case, we are the first to consider sampling from unsupervised robust models. We train ResNet18 8 and ResNet50 (He et al., 2016) following the setup of an existing unsupervised adversarial training method ACL (Tiang al. (020). The training attack is kept the same as that of the supervised case for fair comparison. More details are provided in Appendix.
Sampling protocol. In practice, adversarial sampling methods take the following general form as a mixture of the PGD and Langevin dynamics,
x x,, S0 X,, + aggi nex]. X2 = 6, E WXx''ro, ero, 1), k 0.1...., K,
where g: is the update gradient, ε is the diffusion noise, IIs is the projector operator, and 8 is the (conditional) initial seeds drawn from the multivariate normal distribution whose mean and covari- ance are calculated from the CIFAR-10 test set following Santurrar et al. (2019). We evaluate the sample quality quantitatively with Inception Score (IS) (Salimans et al. (016)) and Fréchet Inception Distance (FID) (Heusel et al. 2017). More details can be found in in Appendix C.T
Comparison with other generative models. In Table T compare the sample quality of adver- sarial sampling methods with different kinds of generative models. We analyze the results in terms of the following aspects:
Our adversarial sampling methods outperform many deep generative models like Pixel- CNN++ WGAN-GE and PresGAN, and obtain ttate-of-the-ar Inception scores on par with StyleGAN2 (Karras et al. 2020).
Comparing our AT-based methods with previous methods for training EBMs (rathhohh
2019 Gao et al. 202T), see that obtains state-of-the-art Inception scores among al.
We download the checkpoint from the repository https://gitan cow/Madryaab/
robustness_appll iccatiooniiiiions
8

---

Published as a conference paper at ICLR 2022
the EBM-based methods. Remarkably, unsupervised CEM with ResNet18 obtains both better IS and FID scores than the original JEM, which adopts WideResNet-28-10 (Zagoruyko & Komodakis, 2016)) with even more parameters.
Compared with previous AT-based method (Santurkar al. 2019), our CEM-based meth- ods also improve IS by large margin (even with the unsupervised ResNet18), Remarkably, the supervised and unsupervised methods obtain similar sample quality, and the supervised methods are better (higher) at IS while the unsupervised methods are better (lower) at FID.
We obtain similar Inception scores state-of-the-ar score-based models like NCSN++, while fail to match their FID scores. Nevertheless, a significant drawback of these methods is that they typically require more than 1,000 steps to draw sample, while ours only require less than 50 steps.
Table Inception Scores (IS) and Fréchet Incep- tion Distance (FID) of different sampling methods for udversarially robust models. Cond: conditional. Uncond: unconditional.
Seed
TA
Langevin
Training
Sampling
Method
IS (1)
FID (L)
TA
9.26
56.72
Langevin
9.65
63.34
Supervised
Cond
CS
9.77
56.26
PGD
RCS
9.80
55.91
MaxEnt
PGD
5.35
74.27
Unsupervised
Uncond
MaxEnt
8.24
41.80
PGD
(w/ ResNet18)
PGD
5.85
68.
Cond
MaxEnt
Maxlin
8.68
36.44
PGD
5.24
141.54
Unsupervised
Uncond
MaxEnt
9.57
44.86
Figure 1:
Four groups of random sam- ples (top to bottom): initial, supervised (ResNet50), unsupervised (ResNet18), unsu- pervised (ResNet50).
(w/ ResNet50)
PGD
5.37
137.68
Cond
MaxEnt
9.61
40.25
Comparison among adversarial sampling methods. In Table 2 we further compare the sample quality of different ad adversarial sampling methods discussed in Sections 4.3 & 5.3] For supervised models, we see that indeed 'AA obtains the lowest IS, while CS can significantly refine the sample quality, and RCS can further improve the sample quality by the injected bias. For unsupervised models,
that MaxEnt outperforms PGD consistently by large margin. In particular, conditional sampling initialized with class-wise noise can improve a little on the sample quality compared to unconditional sampling. The average visual sample quality in Figure T is roughly consistent with the quantitative results.
The mismatch between IS and FID. A notable issue of adversarial sampling methods is the mis- match between the IS and FID scores. For example, Table [ DDPM and our unsupervised CEM (w/ ResNet50) have similar Inception scores, but the FID of DDPM (2.20) is significantly smaller than ours (40.25), a phenomenon also widely observed in previous methods (Santurkar et al.) 2019; Grathwohl et al. 2019 Song Ermon 2019). Through visual inspection of the samples in Figure we can see that the samples have realistic global structure, but as for the local structure, we can find some common artifacts, which could be the reason why it has a relatively large distance (FID) to the real data.
7 CONCLUSION
In this paper, we proposed a unified probabilistic framework, named Contrastive Energy-based Model (CEM), which not only explains the generative ability of adversarial training, but also pro- vides unified perspective of adversarial training and sampling in both supervised and unsupervised paradigms. Extensive experiments show that adversarial sampling methods derived from our frame- work indeed demonstrate better sample quality than state-of-the-art methods.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
Yisen Wang is partially supported by the National Natural Science Foundation China under Grant 62006153, Project 2020BD006 supported by PKU-Baidu Fund, and Open Research Projects of Zhejiang Lab (No. 2022RC0AB0S). Jiansheng Yang supported by the National Science Founda- tion of China under Grant No. 11961 141007. Zhouchen Lin is supported by the NSF China (No. 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606), Project 2020BD006 supported by PKU-Baidu Fund, and Qualcomm.
REFERENCES
Yang Bai, Xin Yan, Yong Jiang, Shu-Tao Xia, and Yisen Wang. Clustering effect of adversarial robust models. In NewrlPS, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:/809 1096, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI., 2020.
Adji Dieng, Francisco JR Ruiz, David M Blei, and Michalis Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv: 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness a prior for learned representations. arXiv preprint arXiv:1906.00944 2019.
Ruiqi Gao, Yang Song. Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. ICLR, 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In /CLR, 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. NeurlPS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. NeurIPS, 2017.
Chih-Hui Ho and Nuno Vasconcelos. Contrastive learning with adversarial examples. NeurIPS, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. NeurlPS, 2020.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. NeurIPS, 2020.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv: 1803.06333. 2018.
10

---

Published as a conference paper at ICLR 2022
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurIPS, 2020.
Simran Kaur, Jeremy Cohen, and Zachary Lipton. Are perceptually-aligned gradients a general property of robust classifiers? arXiv preprint arXiv: 151.0:9910006444 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11362 2020.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. NeurlPS, 2020.
Diederik Kingma and Max Welling, Auto-encoding variational Bayes. ICLR, 2014.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. ICML, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016.
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. ICLR, 2021.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. NeurlPS, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517. 2017.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with single (robust) classifier. In NeurlPS, 2019.
Konstantin Shmelkov, Cordelia Schmid, and Karteck Alahari. How good my GAN? ECCV, 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv: 1007:v/0000000000 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
Michael Tschanne,, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. ICLR, 2020.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In JCML, 2019.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. ICML, 2019.
11

---

Published as a conference paper at ICLR 2022
A EVALUATING ADVERSARIAL ROBUSTNESS
In Section we have shown that the new adversarial sampling algorithms derived from our CEM framework indeed obtain improved sample quality, which helps justifies our interpretation of AT from generative aspect. In this section, we further take complementary way verify our inter- pretation by studying its discriminative part. In particular, we develop new variants of AT regular- ization and verify their effectiveness improving the adversarial robustness of AT. As CEM has both supervised and unsupervised variants, we develop AT variants for each scenario respectively while following the same spirit.
A.1
SUPERVISED ADVERSARIAL TRAINING
A.1.1 PROPOSED METHOD
we have mentioned that for the consistency gradient, original AT (Madry al. 2018) simply ignores it. Denoting x and x as the natural and adversarial inputs, the state-of-the-a AT variant, TRADES (Zhang al. 2019), instead adopts the KL regularization (p(((-))) [| that explicitly regularizes the consistency of the predicted probabilities on all classes, whose optinum implies that p(-18) p(-1x) So xx y) fo 3y)
In Section 4.2
Inspired by this discussion, alternatively, we can directly regularize the consistency gradient to zero. We achieve this with the following Consistency Regularization (CR) with least square loss:
(c) pp.(x.))) @ (x)) (fe(x, 3) fo y))²
(25)
only minimizes the gap between the logits of the label class fo( y). ALP was shown ineffective for improving AT, while in the experiments below, we show that our AT+CR objective indeed achieves comparable (even slightly better) results TRADES.
We note the proposed AT+CR differs to ALP (Kannan et al. 2018)
A.1.2 EMPIRICAL EVALUATION
Experimental setup. Following the conventions, we compare different AT methods with Pre- activated ResNet18 (He et al., 2016) and WideResNet34 (Zagoruyko & Komodakis
(016))
CIFAR-10. The maximum perturbation is bounded by ε = 8/255 under to norm. The training attack is 1GD (Madry et al. 2018) with random start and step size e/4. The test attack is PGD with random start and step size e/4. We evaluate both the final epoch and the early stopped epoch with the best robust accuracy.
Table Robustness (accuracy (%) on adversarial attacks) of supervised adversarial training methods on 10. R18: ResNet18. W34: WideResNet34.
Model
Training
Natural Acc (%)
Adversarial Ace (%)
AT Maddy et al.
2018
83.7
52.2
ResNet18
TRADES Zhang et al.
2019
82.5
54.3
AT+CR (ours)
81.5
55.2
AT (Madry et al. 2018
86.8
53.6
WideResNet34
TRADES (hang et al. 2019)
83.4
57.0
AT+CR (ours)
86.6
57.0
Result analysis. From Table 3 we can see that the AT+CR objective derived from our framework indeed enjoy much better robustness than vanilla AT. Compared to the state-of-the-art AT vari- ant TRADES, When the two have similar robustness (for WideResNee34)), AT+CR obtains better natural accuracy than TRADES (86.6 v.s. 83.4). These results empirically justify that our interpretation of AT and TRADES from probabilistic perspective.
see that AT+CR is comparable, and sometimes slightly better at robustness.
A.2 UNSUPERVISED ADVERSARIAL. TRAINING
Similarly, we can develop the same regularization technique for unsupervised adversarial training through a unified perspective of supervised and unsupervised AT offered by our CEM.
12

---

Published as a conference paper at ICLR 2022
A.2.1 PROPOSED METHOD
In Section 5.2 we have developed principled unsupervised adversarial training routine by an anal- ogy with the supervised AT (Section 4.2). Besides, we can also consider an unsupervised version of the consistency regularization above, namely, the Unsupervised Consistency Regularization (UCR),
LUCR = px(xxx)) 2 (ee())lf x') - fx(
(26)
which encourages the consistency between the similarity between the natural pair (x, x') and the adversarial pair (x,))).
A.2.2 EMPIRICAL EVALUATION
Experimental setup. Among the many variants of contrastive learning, we adopt SimCLR as our baseline method and take the recently proposed ACL. (Jiang al. 2020 as our Unsupervised Ad- versarial Training (UAT) following the same default setup as in Section C.I The training attack configuration kept the same as the supervised case, while after training, we freeze the encoder and fine-tune a linear classification layer (standard training) on top with labeled data for evaluating the learned features. In particular, attack methods, FGSM (Goodfellow et al. 2015) (one-step attack) and PGD (multi-step attack), both with 8/255, and report their natural and adversarial accuracy, respectively.
evaluate the composed model on the test data with two different
Table 4: Robustness (accuracy (%) on adversarial attacks) of unsupervised contrastive learning methods on CIFAR-10 with ResNet-18 backbone and two different attack methods: FGSM (Good-
fellow al. 2013 and PGD (Madry et 2018)
Adversarial Acc (%)
Training
Natural Acc (%)
FGSM
PGD
Standard Training (Chen et al. 2020
91.5
25.6
0.8
UAT (Jiang et al. 2020)
66.6
26.2
21.4
UAT+UCR (ours)
72.0
30.7
24.6
Result analysis. As shown in Table features learned by UAT is indeed more robust than standard training, e.g., 0.8% to 21.4% under PGD attack. Moreover, with proposed UCR regularizer (6),, we not only effectively improve both natural accuracy (66.6% to 72.0%), and also improve adversarial robustness: 26.2% to 30.7% under FGSM attack and 21.4% to 24.6% under PGD attack. This also helps justify the connection between contrastive learning and our CEM.
A.3 CONCLUDING REMARK
With the above experiments on adversarial robustness, we empirically verify that our framework could serve valid probabilistic understanding of adversarial training and can be used to develop new effective adversarial training objectives.
B OMITTED TECHNICAL DETAILS
For concise presentation, have omitted several technical details in the main text. Here present a complete description of the derivation process.
B. LOG LIKELIHOOD GRADIENT OF EBM
In Section 3, we have introduced Energy-based Models (EBM) and the gradient of their log likeli- hood. We now show how it can be derived out.
For EBM in the following form,
(x(x) = exp (x)(()))
(27)
Z(
13

---

Published as a conference paper at ICLR 2022
the gradient of the log likelihood can be derived follows:
pali) log po (x)
(28)
pax) (x)o(x) Vo log Z(0)
Vo I exp( - (x)()
ppx VoEe(x)
2(0)
exp(- Eg (x))
Pxp( Vo Eo)
Vo (8)
pps(x) Vo
E (x) FoFo
positive phase
negative phase
B.2 CONNECTION BETWEEN STANDARD TRAINING AND JEM
In Section 4.4, we have claimed that if we replace the model distribution px( with the data distri- bution pa pa(x) x) in Eqn. 10] the log likelihood gradient of JEM is equivalent to the negative gradient of the CE loss. Here we give detailed proof as follows:
V (x log (x)
Epptxx.. Vofo(x, y) E (o( (01x) Vofo(X, 0)
~.pp.(x.)) Vxfe(x,)) 9) (px(x)pa(xx Vxfe(x,9)
-Epu(x..)
Vofo(x. y) Epo( Vofo(x.)) 0)
= -E (x
Velo(x,))
pa [o(Klx)
[) Voloxxx
-Fpa(x
Vofo(x,y)
Σラ= exp(ff(xx
k
V exp(ff((x
=Ppa(x.))
Volo(x,))
==1 exp (xxp(So(xx))
k= exp(fo(x k))
-Epa(x.9)
Vofo(x.))
E cxp(ff(x,.)
-Epa(x.))
Volo(x,1) Va log Σ exp(ff(xxk))
- Epp(x.) Volog exp(fo(x,)) v))
ΣK_ exp(foo (x, (x,k)
expeexp(or
(elpp( log pe (v/x).
(29)
B.3 EQUIVALENCE BETWEEN AT LOSS AND CONTRASTIVE GRADIENT IN SUPERVISED LEARNING
In Section 4.3, have claimed that the contrastive gradient equals the negative gradient of the robust CE loss (AT loss) following the same deduction in Eqn. 29
(ps(k.)) pa (x..)
/V.fo y) Vefe(*, 0)] VoJol &, 9) - Epo(g(x) VoJo(X 0)] exp(.fo(X (X, v))
-Epu(x 2 (8(
-Epu(x.)) Vo log
k=1 exp(fe(8, k:)
-Epo(x.y) 2 (*) Ve log (y)x))
(30)
which is exactly the negative gradient of the canonical AT loss (Madry al. 2018
14

---

Published as a conference paper at ICLR 2022
B.4 EQUIVALENCE BETWEEN INFONCE LOSS AND NON-PARAMETRIC CEM
In Section 6.1, have claimed that the the log likelihood gradient of NP-CEM equals to exactly the negative gradient of the InfoNCE loss when we approximate po with pa(X) The derivation is presented as follows:
p Vxlo(x,x') ( Vojo (xx,
=E pa(xxx)) Vofo(xxx') Poo 21/x Vo oofo (x,x²)
pa Pa (8,
-apx() Vofo(x, x') Exp(x)
Vo (x, xx)xx)
po xx))
exp(fo fo (x, x¹)
_xp((xx)) Vx x') - (x)
Vele(x,8')
exp(So (x, x))
pa( exp Se
Paxaxx Vofo(xxx') (x))
Volo(
(as pa(*) -p((x)
Pa(x) expl So x))
[22
exp( (8, x)))
= =x Vo,o(xxx') (20() (x) exp Jo x)) Vo
exp(fo x))
E Vofe(xxx') (xa())aa(*)) ( exp(fol55 x)) V
(31)
=Epa(x.x)) Vofo(x x)) (((R) Vo log Epe( exp fo (xx
=Ppa(xxx)) (x x') Vo log B exp(fo(X; x'))] (merge pa(x) with pa(x))
= -Enalssee) Vo log exp(fo(x,x))
exp(fo(xx,x)) x())
Vo log
( exp(fo(x, x())
N
Σ xx,,
where (xxx)) x()'s 'ss are positive samples from (x, x') and X/y are negative samples independently drawn from Pa(X').
B.5 EQUIVALENCE BETWEEN ADVERSARIAL INFONCE AND NP-CEM
In Section 6.2, we have developed the unsupervised analogy AT loss and regularization. In particular, we have claimed that contrastive gradient is equivalent to the gradient of the Adversarial InfoNCE loss (i.e. the InfoNCE loss of the adversarial example x) utilized in previous work (Jiang
et al. 2020 Ho & Vasconcelos 2020
Kim et al. 2020). It can be derived following Eqn. 3T
F @ (k.x))
Vofo(xx Vofolxxx -Voooxxx'>>oo(( x')]
(xp ② Po (8)
Vafa(xxx)) Bxo(k( Vo (8, x')]
exp x'))
= _xx(xx) pe
Volo(*x Epu( Epu(Ko exp(fo(x, Volo(x,
Vo log exp(fo(8xx)) x'))
=B Po
(p)())) exp(fo
N
exp(fe(xx;x)))
log
(32)
Σ exppfa(Xxxaxcc x(,))
(x,x'x x'), X; 's are adversarial samples drawn from (X), and & 's are negative samples independently drawn from pa(x').
where (x;, x.) are positive samples drawn from Pa (x(x,
C MORE DETAILS AND RESULTS ON ADVERSARIAL. SAMPLING
C.I DETAILED EXPERIMENTAL SETUP
Supervised Adversarial Sampling, For supervised robust models, we adopt the same pretrained ResNet50 checkpoint on CIFAR-10 as Santurkar et al. (2019) for a fair comparison. As described
We download the checkpoint from the repository https: / taut..ub.... com/MadryLab/
robustness_appl1 actappppppiiiinssoocoa.c
15

---

Published as a conference paper at ICLR 2022
2.0
6.04
2.0
1.
6.78
14
2.011
0.01
santerdtairpiggg.o staee
Maaimal momm
Figure 2: Algorithmic analysis of our proposed supervised adversarial sampling algorithm (RCS). Left: Inception score with increasing sampling steps N. Middle: Inception score with increasing diffusion noise scale. Right: Inception score with increasing m-nnrm bound B.
6.70
90
6.6
3.3
2.5
8.00
1.3
63
Marber sampting stac
Avote 1al
Vaaina Eg mmm
Figure 3: Algorithmic analysis of our proposed unsupervised adversarial sampling algorithm (Max- Ent). Left: Inception score with increasing sampling steps N. Middle: Inception score with increas- ing diffusion noise scale. Right: Inception score with increasing 2n-norm bound B.
in Santurkar et al. (2019). the ResNet-50 model is adversarially trained for 350 epochs with learning rate 0.01 and batch size 256. The learning rate is dropped by 10 epoch 150 and 200. The training attack is f2-norm PGD attack with random start, maximal perturbation norm 0.5, step size 0.1 and steps.
Unsupervised Adversarial Sampling. As far as we know, we are the first to consider sampling from unsupervised robust models. Therefore, to obtain an unsupervised robust model for sampling, we adopt ACL (Jiang et al. 2020) as the baseline method and follow their default hyper-parametery
① to train it. The official implementation of ACL is built upon the SimCLR (Chen et al. 2020) Tramework for contrastive learning, while they choose a specific hyper-parameter configuration for adversarial training. In particular, they choose 512 for batch size and train for 1000 epochs with ResNet- 18 backbone (He et al. 2016). The base learning rate is 1.0, where they up strategy for the first 10 epochs, and apply cosine annealing scheduler after that. The training attack is kept the same that of the supervised case for a fair comparison. For ResNet-18, we adopt the ACL(A2A) setting, which adopts the normal ResNet with only one Batch Normalization module. Instead, for ResNet-50, we notice that the ACL(A2S) setting yields slightly better results. The ResNet variants in the ACL(A2S) setting contains two BN modules, where we assign natural and adversarial examples to different modules. We refer more details to the original paper (Jiang al. 2020
linear
Evaluation of sample quality. Note that there are four syper-parameters in our sampling protocol: step size falball size B, noise scale 7), and iteration steps K, for which we list our choice in Table 5) We evaluate sample quality quantitatively w.r.f. the Inception Score (IS) (Salimans et al., 2016) and Fréchet Inception Distance (FID) (Heusel et al. 2017) with 50,000 samples, where the standard variation of IS is around 0.1.
C.2 ADDITIONAL ANALYSIS
Besides the results presented in the main text, we also conduct more experiments to analyze the behavior of our proposed adversarial sampling algorithms, both quantitatively and qualitatively. We conduct a detailed analysis of our proposed sampling algorithms and present the results of supervised
code: http:///a////gghho..ooo/////oooo
Cont l.earning
16

---

Published as a conference paper at ICLR 2022
Table 5: Sampling hyper-parameters in each scenario.
Scenario
Model
7
K
Supervised
ResNet50
1
6
0.01
20
ResNet18
00 0.0
10
Unsupervised
ResNet50
7 0o 0.0
50
adversarial sampling (with RCS) Figure 2 and the results of unsupervised adversarial sampling
(with MaxEnt) in Figure 3 Note that in both cases, we adopt the ResNet-50 backbone and use the default hyper-papree peremmeamssterss unless specified.
C.2.1 CHAIN LENGTH
From the left panels of Figure and Figure 3, we can see the two adversarial algorithms both have a sweet spot sampling steps N (length of the sampling Markov chains) at around 30 to 40 steps, before and after which the results will be slightly worse.
C.2.2 NOISE RATIO
In the proper Langevin dynamics, the scale of the noise is determined by the step size, = 22a. However, in practice, this would lead to a catastrophically degraded sample quality as the noise takes over the gradient information. Therefore, following [Song & Ermon (2019) and Grathwohl et al. (2019), we also anneal the noise ratio vj to a smaller value for better sample quality. As shown the middle panels of Figure 2 and Figure the optimal noise ratio is around 0.01 for both cases.
C.2.3 MAXIMAL NORM
An apparent difference of our adversarial sampling algorithms to the canonical Langevin dynamics that ours have a projection step that limits the distance between the samples and the initial seeds. In the right panels of Figure 2 and Figure 3 we show the impact of the scale of the 4o-norm ball for the sample quality. We
that generally speaking, the ball grows larger, the samples get refined. the supervised case, the sample quality gets slightly worse with a large norm, which does not happen in the unsupervised case.
C.2.4 SAMPLING TRAJECTORY
Aside from the qualitative inspection of the proposed sampling algorithms, we also demonstrate the sampling trajectory of our supervised (RCS) and unsupervised (MaxEnt) adversarial sampling methods Figure
5 We can see that the samples get gradually refined in terms both low-level textures and leve- semantics.
17

---

Published as a conference paper at ICLR 2022
Figure 4: Sampling trajectory (the first 20 steps) of our proposed supervised adversarial sampling
algorithm (RCS). Each row represents the refinement progress of single sample.
Figure 5: Sampling trajectory (the first 20 steps) of our proposed unsupervised adversarial sampling
algorithm (MaxEnt). Each row represents the refinement progress of a single sample.
18


Reviews:
[{"review_id": "iwaX698QeVH", "reviewer": "Reviewer_MiyJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has 4 major contributions.\n1. The proposed Contrastive Energy-based Models (CEM) gives a probabilistic interpretation for adversarial training, which explains the generative ability of adversarial trained models.\n2. CEM's parametric and non-parametric form give probabilistic understanding of previous ST and AT in both supervised and unsupervised settings.\n3. Under CEM, the equivalence between the IS and the InfoNCE loss of contrastive learning is established, which enables us to design principled adversarial sampling for unsupervised learning.\n4. Experiments show that adversarial sampling methods derived under CEM outperforms or achieve comparable performance to SOTA.\n\nHowever, more experiments using different models, different dataset and different attacks are needed.", "summary_of_the_paper": "This paper proposed a unified probabilistic framework, dubbed as Contrastive Energy-based Models, to understand the robustness and generative capability. The proposed CEM is a special case of EBM that models the joint distribution over two variables with a similarity function defined in a contrastive form. CEM could be instantiated into parametric form and non-parametric form, which work for supervised learning and unsupervised learning respectively. CEM could demystify adversarial training's generative capability in both supervised and unsupervised setting. Moreover, with CEM, adversarial training could be extended to unsupervised scenario.", "summary_of_the_review": "The proposed unified probabilistic framework CEM is fundamentally novel and interesting. It gives explanation on the generative capability of adversarial trained model. It also gives probabilistic understanding of ST and AT in both supervised and unsupervised learnings. The derived adversarial sampling method show a better sample quality. However, more experiments are needed to further justify the correctness of the proposed CEM, given that there are a lot of approximation in the derivations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UwXWbIEOgKM", "reviewer": "Reviewer_9aEk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I like the direction in this paper. By formulating it as an energy-based model, we can look more into the normalization term of the softmax which is quite often overlooked. The connection drawn between Langevin dynamics and adversarial training is nice and new at least to me. I think the insight in contrastive learning is quite interesting. Although the so-called \"hard negative mining\" is an old idea in the computer vision community, it's not generally been casted in a theoretical framework. I enjoyed reading the paper which is not quite often (anymore) for my review batches in the ML conferences.\n\nThere is this small confusion in the notation that p_theta(x,y) defined in eq. (2) seems to be used equivalently with p_theta(y|x), it's better to clarify which part of f_theta(x,y) was attributed to the prior p(x), or my understanding would be eq.(2) should rather be a definition of p_theta(y|x) (where it doesn't take into account the prior).\n\nI'm also a little bit confused initially about the sign between eq. (12) and eq. (7). In eq.(7), the goal for the minimization step is MLE, which means to minimize - logp (or equivalently, maximize log p). In eq. (12), we are talking about the AT loss being the same as negative gradient of the loss in eq.(7), which is correct, but in order to match the sign w.r.t. the MLE in eq. (7), seems that we should be running gradient ascent. This is clear in the end from the appendix, but it might worth mentioning a single word in the main paper as well.\n\nThe results are also very competitive, I think the unsupervised contrastive results are especially quite strong. It would be nice to have some additional technical explanations why the methods of moments approaches would work better than this current approach, rather than only a descriptive explanation in the current manuscript (although that is nice to have as well).\n\nIn terms of the results, it is interesting to see unsupervised Langevin performing better than supervised Langevin method. Any insights in that?\n\nAlthough the paper focused on generative models, I also would be wondering whether the importance sampling approach might have consequences for discriminative models as well (the more common application in computer vision). It might be nice to show that as well since it shouldn't be a lot of work.\n\nNvasconcelos should be Vasconcelos.", "summary_of_the_paper": "By combining generative models and discriminative models in the framework of an energy-based model, this paper provides several theoretical insights about adversarial training, including that the softmax normalization term already provides a hint on adversarial training, and the importance sampling (with a justification) approach to unsupervised contrastive learning. The experiments show that their strategy with Langevin sampling improves over simpler adversarial training strategies.\n", "summary_of_the_review": "Overall, I like the insights provided in the paper and their experiments also proved their points well. Solid accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QSdxOkEo8bo", "reviewer": "Reviewer_Pwz6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is technically sound and brings a novel perspective to adversarial training. The paper is the first to propose sampling from an unsupervised adversarially trained model. The technical aspects of the paper are overall very impressive. \n\nThe experiment results support their claims well. The reported FID and IS of their method is also pretty good. The qualitative sample quality in Figure 1 and appendix seems worse than the JEM paper although they're getting a higher FID. \n\nThere are a few minor typos:\nIn related works, the second line: while standard classifiers cannot?\nIn section 3, it is the first time the abbreviation EBM is appearing.\nIn page 8 there is a Table ?? in the paragraph before Comparison with other generative models\n\n", "summary_of_the_paper": "This paper justifies why models trained with adversarial training are good generative models by proposing a probabilistic framework. The paper is heavily influenced by the JEM paper and proposes a generalized form of analysis in that paper to include the unsupervised scenario. The generalization defines a joint distribution $p(x, z)$ over data $x$ and representations $z$. For the supervised case, each class is represented by a \"center\" $w_y$ and for the unsupervised one, $p(x, z) = p(x) p(z|x)$ and $p(z|x)$ is reparametrized with augmentation.\n\nThen the authors use this probabilistic framework to analyze the adversarial training. Specifically, they show PGD is a biased form of the Langevin dynamics in their formulation. Then they show how the maximum likelihood training of their probabilistic framework is similar to adversarial training. As a result, they conclude adversarial training leads to a model with high generative power. Finally, they propose refined sampling strategies from adversarially trained models.\n\nIn the last part of the paper, they use their framework to propose an unsupervised adversarial training method followed by sampling algorithms from the produced models.", "summary_of_the_review": "Besides the few minor typos, I think the paper meets the quality bar for ICLR. The technical analysis has some valuable novel aspects that justify well why adversarially trained models are good generative models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ODppKui3mKu", "reviewer": "Reviewer_rJdV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution is a unified probabilistic framework to explain adversarial learning, however, it is not clear whether this probabilistic framework can be beneficial for improving the training of adversarial models. \n\nThe sampling strategy is not claimed as a contribution of the proposed work, however, the experiment part spends a lot space to demonstrate that the sampling in the proposed framework is better than existing works. The authors need to more clearly elaborate the contributions. \n\nThe proposed probabilistic framework is generic, and thus should be able to work with different large-scale and large-resolution datasets. The authors only show results on a relatively small-scale and low-resolution dataset. The results presented are not ideal to explain the effectiveness of the proposed framework. \n\n\n\n", "summary_of_the_paper": "The submission proposes a unified probabilistic framework to illustrate the generative capability of adversarial training. It also offers a unified perspective of adversarial training and sampling from both a supervised learning setting and an unsupervised learning setting. The proposed adversarial sampling strategy from the method is extensively demonstrated on different benchmarks, showing better quality compared with existing related works. ", "summary_of_the_review": "The problem studied is interesting, however, more investigation should be conducted to show the effectiveness and importance of the proposed probabilistic framework. Please also check the weakness part for more explanation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a probabilistic framework that explains why models trained adversarially are robust generators. It received fairly high initial scores. The reviewers thought the work was novel and interesting. They liked that the analysis provided a way to derive a novel training method and sampling algorithms. Reviewers confirmed their support of acceptance and I think this paper is clearly above the bar. Respectfully, I’d prefer that the authors don’t ask the reviewers to “raise your score”. It is up to the reviewers to make that decision.
Decision: Accept (Poster)