Year: 2021

Abstract:
Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently researchers notice that robust models with AT have good generative ability and can synthesize realistic images, while the reason behind it is yet under-explored. In this paper, we demystify this phenomenon by developing a unified probabilistic framework, called Contrastive Energy-based Models (CEM). On the one hand, we provide the first probabilistic characterization of AT through a unified understanding of robustness and generative ability. On the other hand, our unified framework can be extended to the unsupervised scenario, which interprets unsupervised contrastive learning as an important sampling of CEM. Based on these, we propose a principled method to develop adversarial learning and sampling methods. Experiments show that the sampling methods derived from our framework improve the sample quality in both supervised and unsupervised learning. Notably, our unsupervised adversarial sampling method achieves an Inception score of 9.61 on CIFAR-10, which is superior to previous energy-based models and comparable to state-of-the-art generative models.

heading: 1 INTRODUCTION
text: Adversarial Training (AT) is one of the most effective approaches developed so far to improve the robustness of deep neural networks (DNNs) (Madry et al., 2018). AT solves a minimax optimization problem, with the inner maximization generating adversarial examples by maximizing the classification loss, and the outer minimization finding model parameters by minimizing the loss on adversarial examples generated from the inner maximization. Recently, researchers have noticed that such robust classifiers obtained by AT are able to extract features that are perceptually aligned with humans (Engstrom et al., 2019). Furthermore, they are able to synthesize realistic images on par with stateof-the-art generative models (Santurkar et al., 2019). Nevertheless, it is still a mystery why AT is able to learn more semantically meaningful features and turn classifiers into generators. Besides, AT needs the labeled data {(xi,yi)} for training while canonical deep generative models do not, e.g., VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2015) only require {xi}. Thus, it is worth exploring if it is possible to train a robust model without labeled data. Several recent works (Jiang et al., 2020; Kim et al., 2020; Ho & Vasconcelos, 2020) have proposed unsupervised AT by adversarially attacking the InfoNCE loss (Oord et al., 2018) (a widely used objective in unsupervised contrastive learning), which indeed improves the robustness of contrastive encoders. However, a depth investigation and understanding for unsupervised AT is still missing.
To address the above issues, in this work, we propose a unified probabilistic framework, Contrastive Energy-based Models (CEM), that provides a principled understanding on the robustness and the generative ability of different training paradigms. Specifically, we make the following contributions:
• Demystifying adversarial training and sampling. We firstly propose a probabilistic interpretation for AT, that is, it is inherently a (biased) maximum likelihood training of the corresponding energy-based model, which explains the generative ability of robust models learned by AT. Inspired by this, we propose some novel sampling algorithms with better sample quality than previous methods.
• A unified probabilistic framework. Based on the understanding above, we propose Contrastive Energy-based Model (CEM) that incorporates both supervised and unsupervised learning paradigms. Our CEM provides a unified probabilistic understanding of previous standard and adversarial training methods in both supervised and unsupervised learning.
• Principled unsupervised adversarial training and sampling. Specifically, under our proposed CEM framework, we establish the equivalence between the importance sampling of CEM and the InfoNCE loss of contrastive learning, which enables us to design principled adversarial sampling for unsupervised learning.
Notably, we show that the sampling methods derived from our framework achieve state-of-the-art sample quality (9.61 Inception score) with unsupervised robust models, which is comparable to both the supervised counterparts and other state-of-the-art generative models.

heading: 2 RELATED WORK
text: Robust generative models. Researchers recently notice that features extracted by robust classifiers are perceptually aligned with humans, while standard classifiers are not (Engstrom et al., 2019; Kaur et al., 2019). Furthermore, Santurkar et al. (2019) show that we can also generate images of high quality with robust classifiers by iteratively updating from a randomly sampled noise. They show the sample quality of robust classifiers is comparable to state-of-the-art generative models like BigGAN (Brock et al., 2018).
Contrastive learning. Oord et al. (2018) firstly propose unsupervised contrastive learning by maximizing a tractable lower bound on mutual information (MI), i.e., the negative InfoNCE loss. However, later works find that the lower bounds degrade a lot with a large MI, and the success of these methods cannot be attributed to the properties of MI alone (Poole et al., 2019; Tschannen et al., 2020). Our work provides an alternative understanding of unsupervised contrastive learning as importance sampling of an energy-based model, which also enables us to characterize the limitations of existing methods from a new perspective. In fact, contrastive learning can also be seen as a general learning framework beyond the unsupervised scenarios. For example, SupContrast (Khosla et al., 2020) extends contrastive learning to supervised scenarios. Our work further bridges supervised, unsupervised and adversarial contrastive learning with a unified probabilistic framework.

heading: 3 CEM: A UNIFIED PROBABILISTIC FRAMEWORK
text: Inspired by previous work that bridges discriminative models with energy-based models (Grathwohl et al., 2019), in this work, we propose a unified framework, called Contrastive Energy-based Model (CEM), that incorporates both supervised and unsupervised scenarios.
Our proposed CEM is a special kind of Energy-based Models (EBMs) that models the joint distribution pθ(u,v) over two variables (u,v) with a similarity function fθ(u,v) defined in a contrastive form,
pθ(u,v) = exp(fθ(u,v))
Z(θ) , (1) where Z(θ) = ∫
exp (fθ(u,v)) dudv is the corresponding partition function. In other words, in CEM, a pair of samples (u,v) has higher probability if they are more alike. In particular, it can be instantiated into the two following variants under different learning scenarios.
Parametric CEM. In the supervised scenario, we specify the Parametric CEM (P-CEM) that models the joint distribution pθ(x, y) of data x and label y in the following form,
pθ(x, y) = exp(fθ(x, y))
Z(θ) =
exp(gθ(x) >wy)
Z(θ) , (2)
where gθ : Rn → Rm denotes the encoder, g(x) ∈ Rm is the representation of x, and wk ∈ Rm refers to the parametric cluster center of the k-th class. Denote the linear classification weight as W = [w1, · · · ,wK ] and the logit vector as h(x) = g(x)>W, we can see the equivalence between P-CEM and JEM (Grathwohl et al., 2019) as
fθ(x, y) = gθ(x) >wy = hθ(x)[y]. (3)
Non-Parametric CEM. In the unsupervised scenario, we do not have access to labels, thus we instead model the joint distribution between data x and representation z as
pθ(x, z) = exp(fθ(x, z))
Z(θ) =
exp(gθ(x) >z)
Z(θ) , (4)
and the corresponding likelihood gradient is
∇θEpd(x,z) log pθ(x, z) = Epd(x,z)∇θfθ(x, z)−Epθ(x̂,ẑ)∇θfθ(x̂, ẑ). (5)
In contrastive to P-CEM that incorporates parametric cluster centers, the joint distribution of NPCEM (Non-Parametric CEM) is directly defined based on the similarity between instances. We define the joint data distribution pd(x, z) = pd(x)pd(z|x) through re-parameterization,
z = fθ(t(x)), t u.a.r.∼ T , x ∼ pd(x), (6)
where u.a.r. denotes sampling uniformly at random and T refers to the user-define pretext, e.g., a set of data augmentation operators T = {t : Rn → Rn} 1.

heading: 4 SUPERVISED SCENARIO: REDISCOVERING ADVERSARIAL TRAINING AS MAXIMUM LIKELIHOOD TRAINING
text: In this section, we investigate why robust models have a good generative ability. The objective of AT is to solve the following minimax optimization problem:
min θ
Epd(x,y) [
max ‖x̂−x‖p≤ε `CE(x̂, y;θ)
] , where `CE(x̂, y;θ) = − log pθ(y|x̂). (7)
The inner maximization problem is to find an adversarial example x̂ within the `p-norm ε-ball around the natural example x that maximizes the CE loss. While the outer minimization problem is to find model parameters that minimize the loss on the adversarial examples x̂.

heading: 4.1 MAXIMIZATION PROCESS
text: For the inner maximization problem, Projected Gradient Descent (PGD) (Madry et al., 2018) is the commonly used method, which generates the adversarial example x̂ by maximizing the CE loss2 (i.e., minimizing the log conditional probability) starting from x̂0 = x:
x̂n+1 = x̂n + α∇x̂n`(x̂n, y;θ) = x̂n − α∇x̂n log pθ(y|x̂n)
= x̂n + α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, k)) ] − α∇x̂nfθ(x̂n, y), (8)
while the Langevin dynamics for sampling P-CEM starts from random noise x̂0 = δ and updates with
x̂n+1 = x̂n + α∇x̂ log pθ(x̂n) + √ 2α · ε (9)
= x̂n + α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, k))
] + √ 2α · ε.
Eqns. 8 and 9 both have a positive logsumexp gradient (the second term) to push up the marginal probability pθ(x̂). As for the third term, PGD starts from a data point (x, y) such that it requires the repulsive gradient to be away from the original data point and do the exploration in a local region. Langevin dynamics instead starts from a random noise and an additive noise ε is injected for exploration.
Comparing PGD and Langevin. Following the above analysis, the maximization process in AT can be seen as a (biased) sampling method that draws samples from the corresponding probabilistic
1For the data pair, we detach the gradient of z w.r.t. θ and assume that x and t(x) have the same marginal distribution pd(x).
2Note that we omit the projection operation and the gradient re-normalization steps.
model pθ(x̂). Compared to Langevin dynamics, PGD imposes specific inductive bias for sampling. With the additional repulsive gradient and ε-ball constraint, it explicitly encourages the samples to be misclassified around the original data points. In practice, adversarial training with such adversarial examples is generally more stable than training JEM with Langevin samples, which indicates that PGD attack is a competitive alternative for the negative sampling method for JEM training.

heading: 4.2 MINIMIZATION PROCESS
text: To begin with, the gradient of the joint log likelihood for P-CEM can be written as follows:
∇θEpd(x,y) log pθ(x, y) =Epd(x,y)∇θfθ(x, y)−Epθ(x̂,ŷ)∇θfθ(x̂, ŷ) =Epd(x,y)∇θfθ(x, y)−Epθ(x̂)pθ(ŷ|x̂)∇θfθ(x̂, ŷ),
(10)
where (x, y) ∼ pd(x, y) denotes the positive data pair, and (x̂, ŷ) ∼ pθ(x̂, ŷ) denotes the negative sample pair. As discussed above, the adversarial examples x̂ generated by the maximization process can be regarded as negative samples, and ŷ ∼ pθ(ŷ|x̂) denotes the predicted label of x̂. To see how the maximum likelihood training of P-CEM is related to the minimization process of AT, we add an interpolated adversarial pair (x̂, y) into Eq. 10 and decompose it as the consistency gradient and the contrastive gradient:
∇θEpd(x,y) log pθ(x, y) = Epd(x,y)⊗ pθ(x̂,ŷ) [∇θfθ(x, y)−∇θfθ(x̂, ŷ)] =Epd(x,y)⊗ pθ(x̂,ŷ)[∇θfθ(x, y)−∇θfθ(x̂, y)︸ ︷︷ ︸
consistency gradient
+∇θfθ(x̂, y)−∇θfθ(x̂, ŷ)︸ ︷︷ ︸ contrastive gradient
] . (11)
Next, we show that the two parts correspond to two effective mechanisms developed in the adversarial training literature.
AT loss. As the two sample pairs in the contrastive gradient share the same input x̂, we can see that the contrastive gradient can be written equivalently as
Epd(x,y)⊗ pθ(x̂,ŷ) [∇θfθ(x̂, y)−∇θfθ(x̂, ŷ)] =Epd(x,y)⊗ pθ(x̂) [∇θfθ(x̂, y)− Epθ(ŷ|x̂)∇θfθ(x̂, ŷ)] =Epd(x,y)⊗ pθ(x̂)∇θ log pθ(y|x̂), (12)
which is exactly the negative gradient of the robust CE loss (AT loss) in Eq. 7, in other words, gradient ascent with the contrastive gradient is equivalent to gradient descent w.r.t. the AT loss.
Regularization. As for the consistency gradient, original AT (Madry et al., 2018) simply ignores it. Its variant TRADES (Zhang et al., 2019) instead proposes the KL regularization KL(p(·|x̂)‖p(·|x)) that regularizes the consistency of the predicted probabilities on all classes, whose optimum implies that p(·|x̂) = p(·|x)→ fθ(x, y) = fθ(x̂, y). Comparing AT and JEM training paradigms. The above analysis indicates that the minimization objective of AT is closely related to the maximum likelihood training of JEM (Grathwohl et al., 2019). Compared to JEM that decomposes the joint likelihood into an unconditional model pθ(x) and a discriminative model pθ(y|x), the decomposition of AT in Eq. 10 instead stabilizes training by introducing an intermediate adversarial pair (x̂, y) that bridges the positive pair (x, y) and the negative pair (x̂, ŷ). Besides, it can inject the adversarial robustness bias by regularizing the consistency gradient. Together with our analysis on the maximization process, we show that AT is a competitive alternative for training JEM (a generative model) with more stable training behaviors. That explains why robust models with AT are also generative.

heading: 4.3 PROPOSED SUPERVISED ADVERSARIAL SAMPLING ALGORITHMS
text: Our interpretation of AT also inspires us to design principled sampling algorithms from robust classifiers.
Targeted Attack (TA). Previously, to draw samples from a robust classifier, Santurkar et al. (2019) utilize a targeted attack that optimizes the input initialized by a random noise x̂0 towards a specified
class ŷ ∼ pd(ŷ):
x̂n+1 = x̂n + α∇xn log pθ(ŷ|x̂n) = x̂n + α∇xf(x̂n, ŷ)− α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, k))
] .
(13) Compared to PGD attack in Eq. 8, while pushing x̂ towards ŷ, TA has a negative logsumexp gradient that decreases the marginal probability pθ(x̂). This could explain why TA is less powerful for adversarial attack and is rarely used for adversarial training.
Conditional Sampling (CS). To overcome the drawback of targeted attack, a natural idea would be dropping the negative logsumexp gradient. In fact, we can show that this is equivalent to sampling from the conditional distribution:
pθ(x|ŷ) = exp(fθ(x, ŷ))
Zx|ŷ(θ) , Zx|ŷ(θ) = ∫ x exp(fθ(x, ŷ))dx,
and its Langevin dynamics takes the form:
x̂n+1 = xn + α∇x̂n log pθ(x̂n|ŷ) + √ 2α · ε = x̂n + α∇x̂nfθ(x̂n, ŷ) + √
2α · ε. (14) Samples drawn this way essentially follow an approximated model distribution, pθ(x̂, ŷ) ≈ pd(ŷ)pθ(x̂|ŷ). Thus, CS can be seen as a debiased targeted attack algorithm. Reinforced Conditional Sampling (RCS). Inspired by the above analysis, we can design a biased sampling method that deliberately injects a positive logsumexp gradient:
x̂n+1 =x̂n + α∇x̂nfθ(x̂n, ŷ) + α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, k))
] + √ 2α · ε. (15)
With our designed bias, RCS will sample towards the target class ŷ by maximizing pθ(x̂|ŷ) (with the fθ(x̂n, ŷ) term), and at the same time improve the marginal probability pθ(x̂) (with the logsumexp term). As shown in our experiment, RCS indeed obtains improved sample quality.

heading: 4.4 DISCUSSION ON STANDARD TRAINING
text: In the above discussion, we have explained why adversarial training is generative from the perspective of CEM. In fact, it can also help characterize why classifiers with Standard Training (ST) are not generative (i.e., poor sample quality). A key insight is that if we replace the model distribution pθ(x̂) with the data distribution pd(x) in Eq. 10, we have
∇θEpd(x,y) log pθ(x, y) = Epd(x,y)∇θfθ(x, y)−Epθ(x̂)pθ(ŷ|x̂)∇θfθ(x̂, ŷ) ≈Epd(x,y)∇θfθ(x, y)− Epd(x)pθ(ŷ|x)∇θfθ(x, ŷ) = ∇θEpd(x,y) log pθ(y|x), (16)
which is the negative gradient of the CE loss in Eq. 7. Thus, ST is equivalent to training CEM by simply replacing model-based negative samples x̂ ∼ pθ(x) with data samples x ∼ pd(x). This approximation makes ST computationally efficient with good accuracy on natural data, but significantly limits its robustness on adversarial examples (as model-based negative samples). Similarly, because ST ignores exploring negative samples while training, standard classifiers also fail to generate realistic samples.

heading: 5 PRINCIPLED EXTENSION OF ADVERSARIAL TRAINING TO UNSUPERVISED SCENARIO
text: In this section, we show that with our unified framework, we can naturally extend the interpretation developed for supervised adversarial training to the unsupervised scenario.

heading: 5.1 UNDERSTANDING UNSUPERVISED STANDARD TRAINING
text: Recently, the following InfoNCE loss is widely adopted for unsupervised contrastive learning of data representations (Oord et al., 2018; Chen et al., 2020; He et al., 2020),
`NCE(x, x̄, X̂ ;θ) = − log exp(fθ(x, x̄))∑K i=j exp(fθ(x, x̂j)) , (17)
where fθ(x, x̂) = gθ(x)>gθ(x̂) calculates the similarity between the representations of the two data samples, x, x̄ are generated by two random augmentations (drawn from T ) of the same data example, and X̂ = {x̂j}Kj=1 consists of K independently drawn negative samples. In practice, one of the K negative samples is chosen to be the positive sample x̄. Therefore, InfoNCE can be seen as an instance-wise K-class cross entropy loss for non-parametric classification.
Perhaps surprisingly, we show that the InfoNCE loss is equivalent to the importance sampling estimate of our NP-CEM (Eq. 4) by approximating the negative samples from pθ(x) with data samples from pd(x), as what we have done in standard supervised training (Section 4.4):
Epd(x,z)∇θfθ(x, z)− Epθ(x̂,ẑ)∇θfθ (x̂, ẑ)
=Epd(x,z)∇θfθ(x, z)− Epθ(x̂)pθ(ẑ) pθ(x̂|ẑ) pθ(x̂) ∇θfθ(x̂, ẑ)
≈Epd(x,z)∇θfθ(x, z)− Epd(x̂)pd(ẑ) pθ(x̂|ẑ) pθ(x̂) ∇θfθ(x̂, ẑ)
≈ 1 N N∑ i=1 ∇θ log exp(fθ(xi, zi))∑K k=1 exp(fθ(xi, z − ik)) , (18)
which is exactly the negative gradient of the InfoNCE loss. In the above analysis, for an empirical estimate, we draw N positive pairs (xi, zi) ∼ pd(x, z), and for each anchor xi, we further draw K negative samples {z−ik} from pd(ẑ) through reparameterization (Eq. 6). The negative phase of NP-CEM is supposed to sample from pθ(x̂|ẑ), where samples semantically close to the anchor sample x̂, a.k.a. hard negative samples, should have high probabilities. However, InfoNCE adopts a non-informative uniform proposal pd(ẑ) for importance sampling, which is very sample inefficient because most samples are useless (Kalantidis et al., 2020). This observation motivates us to design more efficient sampling scheme for contrastive learning by mining hard negatives. For example, Robinson et al. (2021) directly replace the plain proposal with p̃θ(x̂|ẑ) = exp(βfθ(x̂,ẑ))Zβ(θ) while keeping the reweighing term. From the perspective of NP-CEM, it will introduce bias into the importance sampling, which should be treated carefully. In all, NP-CEM provides a principled framework to characterize and develop contrastive learning algorithms.

heading: 5.2 PROPOSED UNSUPERVISED ADVERSARIAL TRAINING
text: AT is initially designed for supervised learning, where adversarial examples can be clearly defined by misclassification. However, it remain unclear what is the right way to do Unsupervised Adversarial Training (UAT) without access to any labels. Previous works (Jiang et al., 2020; Ho & Vasconcelos, 2020; Kim et al., 2020) have carried out UAT with the adversarial InfoNCE loss, which works well but lacks theoretical justification. Our unified CEM framework offers a principled way to generalize adversarial training from supervised to unsupervised scenarios.
Maximization Process. Sampling from pθ(x) can be more difficult than that in supervised scenarios because it does not admit a closed form for variable z. Thus, we perform Langevin dynamics with K negative samples {z−k },
x̂n+1 = x̂n + α∇x̂n log pθ(x̂n) + √ 2α · ε (19)
≈ x̂n + α∇x̂n
[ log 1
K K∑ k=1 pθ(x̂n, z − k )
] + √ 2α · ε
= x̂n + α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, z − k ))
] + √ 2α · ε.
While the PGD attack of the InfoNCE loss (Eq. 29),
x̂n+1 = x̂n + α∇x̂n log exp(fθ(x̂n, z))∑K
k=1 exp(fθ(x̂n, z − k ))
(20)
=x̂n + α∇x̂n
[ log
K∑ k=1 exp(fθ(x̂n, z − k ))
] − α∇θfθ(x̂n, z),
resembles the Langevin dynamics as they both share the positive logsumexp gradient that pushes up pθ(x̂), and differs by a repulse negative gradient −fθ(x̂, z) away from the anchor z, which is a direct analogy of the PGD attack in supervised learning (Section 4.1). Therefore, we believe that the PGD attack of InfoNCE is a proper way to craft adversarial examples by sampling from pθ(x).
Minimization Process. Following the same routine in Section 4.2, with the adversarial example x̂ ∼ pθ(x̂), we can insert an interpolated adversarial pair (x̂, z) and decompose the gradient of NP-CEM into the consistency gradient and the contrastive gradient,
∇θEpd(x,z) log pθ(x, z) = Epd(x,z)⊗ pθ(x̂,ẑ) [∇θfθ(x, z)−∇θfθ(x̂, z)] =Epd(x,z)⊗ pθ(x̂,ẑ)[∇θfθ(x, z)−∇θfθ(x̂, z)︸ ︷︷ ︸
consistency gradient
+∇θfθ(x̂, z)−∇θfθ(x̂, ẑ)︸ ︷︷ ︸ contrastive gradient
] . (21)
In this way, we can directly develop the unsupervised analogy of AT loss and regularization (Sec. 4.2). Following Eq. 28, it is easy to see that the contrastive gradient is equivalent to the gradient of the Adversarial InfoNCE loss utilized in previous work (Jiang et al., 2020; Ho & Vasconcelos, 2020; Kim et al., 2020) with adversarial example x̂,
Epd(x,z)⊗ pθ(x̂,ẑ) [∇θfθ(x̂, z)−∇θfθ(x̂, ẑ)] =Epd(x,z)⊗ pθ(x̂) [ ∇θfθ(x̂, z)− Epθ(ẑ)pθ(x̂)
pθ(x̂|ẑ) pθ(x̂)
∇θfθ(x̂, ẑ) ]
=Epd(x,z)⊗ pθ(x̂) [ ∇θfθ(x̂, z)− Epθ(ẑ)
pθ(x̂|ẑ) pθ(x̂)
∇θfθ(x̂, ẑ) ]
≈ 1 N N∑ i=1 ∇θ log exp(fθ(x̂i, zi))∑K k=1 exp(fθ(x̂i, ẑ − ik)) , (22)
where {ẑ−ik} denotes the adversarial negative samples from pθ(ẑ).

heading: 5.3 PROPOSED UNSUPERVISED ADVERSARIAL SAMPLING
text: In supervised learning, a natural method to draw a sample x̂ from a robust classifier is to maximize its conditional probability w.r.t. a given class ŷ ∼ pd(ŷ), i.e., maxx̂ p(ŷ|x̂), by targeted attack (Santurkar et al., 2019). However, in the unsupervised scenarios, we do not have access to labels, and this approach is not applicable anymore.
PGD. A direct analogy would be using the PGD attack (Eq. 20) for sampling. However, different from the training process, at the sampling stage, we do not have access to real data points as our anchor points for adversarial attack. Therefore, PGD attack may not be effective at sampling.
Langevin. Instead, we believe that Langevin dynamics (Eq. 19) is more suitable for sampling from unsupervised robust models without data and labels. In particular, we initialize a batch of samples X̂ = {x̂i} independently, and update each sample x̂i by maximizing its marginal probability pθ(x̂i) following Eq. 19, where we take the rest of samples X̂ \{x̂i} as its negative samples. This joint evolution will gradually refine the quality of all samples in the batch.

heading: 6 EXPERIMENTS
text: In this section, we evaluate the adversarial sampling methods derived from our CEM framework, showing that they can bring significant improvement to the sample quality of previous work. Besides, in Appendix A, we also conduct a range of experiments on adversarial robustness to verify our probabilistic understandings of AT. We show adversarial training objectives derived from our CEM can indeed significantly improve the performance of AT in both supervised and unsupervised scenarios, which helps justify our interpretations and our framework.

heading: 6.1 EXPERIMENTAL SETUP
text: Models. For supervised robust models, we adopt the same pretrained ResNet50 checkpoint on CIFAR-10 as Santurkar et al. (2019) 3 for a fair comparison. The model is adversarially trained
3We download the checkpoint from the repository https://github.com/MadryLab/ robustness_applications.
with `2-norm PGD attack with random start, maximal perturbation norm 0.5, step size 0.1 and 7 steps. As for the unsupervised case, we are the first to consider sampling from unsupervised robust models. We train ResNet18 and ResNet50 (He et al., 2016) encoders following the setup of an existing unsupervised adversarial training method ACL (Jiang et al., 2020). The training attack is kept the same as that of the supervised case for a fair comparison. More details are provided in Appendix.
Sampling protocol. In practice, our adversarial sampling methods take the following general form as a mixture of the PGD and Langevin dynamics,
xn+1 = Π‖xn−x0‖2≤β [xn + αgk + ηεk] ,
x0 = δ, εk ∼ N (0,1), k = 0, 1, . . . ,K,
where gk is the update gradient, εk is the diffusion noise, ΠS is the projector operator, and δ is the (conditional) initial seeds drawn from the multivariate normal distribution whose mean and covariance are calculated from the CIFAR-10 test set following Santurkar et al. (2019). We evaluate the sample quality quantitatively with Inception Score (IS) (Salimans et al., 2016) and Fréchet Inception Distance (FID) (Heusel et al., 2017). More details can be found in in Appendix D.1.

heading: 6.2 RESULTS
text: Comparison with other generative models. In Table 1, we compare the sample quality of adversarial sampling methods with different kinds of generative models. We analyze the results in terms of the following aspects:
• Our adversarial sampling methods outperform many deep generative models like PixelCNN++, WGAN-GP and PresGAN, and obtain state-of-the-art Inception scores on par with StyleGAN2 (Karras et al., 2020).
• Comparing our AT-based methods with previous methods for training EBMs (Grathwohl et al., 2019; Gao et al., 2021), we see that it obtains state-of-the-art Inception scores among
the EBM-based methods. Remarkably, our unsupervised CEM with ResNet18 obtains both better IS and FID scores than the original JEM, which adopts WideResNet-28-10 (Zagoruyko & Komodakis, 2016) with even more parameters.
• Compared with previous AT-based method (Santurkar et al., 2019), our CEM-based methods also improve IS by a large margin (even with the unsupervised ResNet18). Remarkably, the supervised and unsupervised methods obtain similar sample quality, and the supervised methods are better (higher) at IS while the unsupervised methods are better (lower) at FID.
• We obtain similar Inception scores as state-of-the-art score-based models like NCSN++, while fail to match their FID scores. Nevertheless, a significant drawback of these methods is that they typically require more than 1,000 steps to draw a sample, while ours only require less than 50 steps.
Training Sampling Method IS (↑) FID (↓)
Supervised Cond
TA 9.26 56.72 Langevin 9.65 63.34 CS 9.77 56.26 RCS 9.80 55.91
Unsupervised (w/ ResNet18)
Uncond PGD 5.35 74.27Langevin 8.24 41.80
Cond PGD 5.85 68.54Langevin 8.68 36.44
Unsupervised (w/ ResNet50)
Uncond PGD 5.24 141.54Langevin 9.57 44.86
Cond PGD 5.37 137.68Langevin 9.61 40.25
Table 2: Inception Scores (IS) and Fréchet Inception Distance (FID) of different sampling methods for adversarially robust models. Cond: conditional. Uncond: unconditional.
Comparison among adversarial sampling methods. In Table 2, we further compare the sample quality of different adversarial sampling methods discussed in Sections 4.3 & 5.3. For supervised models, we can see that indeed TA obtains the lowest IS, while CS can significantly refine the sample quality, and RCS can further improve the sample quality by the injected bias. For unsupervised models, we can see that Langevin dynamics outperforms PGD consistently by a large margin. In particular, conditional sampling initialized with class-wise noise can improve a little on the sample quality compared to unconditional sampling. The average visual sample quality in Figure 1 is roughly consistent with the quantitative results.
The mismatch between IS and FID. A notable issue of adversarial sampling methods is the mismatch between the IS and FID scores. For example, in Table 1, DDPM and our unsupervised CEM (w/ ResNet50) have similar Inception scores, but the FID of DDPM (2.20) is significantly smaller than ours (40.25), a phenomenon also widely observed in previous methods (Santurkar et al., 2019; Grathwohl et al., 2019; Song & Ermon, 2019). Through a visual inspection of the samples in Figure 1, we can see that the samples have realistic global structure, but as for the local structure, we can find some common artifacts, which could be the reason why it has a relatively large distance (FID) to the real data.

heading: 7 CONCLUSION
text: In this paper, we proposed a unified probabilistic framework, named Contrastive Energy-based Model (CEM), which not only explains the generative ability of adversarial training, but also provides a unified perspective of adversarial training and sampling in both supervised and unsupervised paradigms. Extensive experiments show that adversarial sampling methods derived from our framework indeed demonstrate better sample quality than state-of-the-art methods.

heading: A EVALUATING ADVERSARIAL ROBUSTNESS
text: In Section 6, we have shown that the new adversarial sampling algorithms derived from our CEM framework indeed obtain improved sample quality, which helps justifies our interpretation of AT from a generative aspect. In this section, we further take a complementary way to verify our interpretation by studying its discriminative part. In particular, we develop new variants of AT regularization and verify their effectiveness on improving the adversarial robustness of AT. As CEM has both supervised and unsupervised variants, we develop AT variants for each scenario respectively while following the same spirit.

heading: A.1 SUPERVISED ADVERSARIAL TRAINING

heading: A.1.1 PROPOSED METHOD
text: In Section 4.2, we have mentioned that for the consistency gradient, original AT (Madry et al., 2018) simply ignores it. Denoting x and x̂ as the natural and adversarial inputs, the state-of-the-art AT variant, TRADES (Zhang et al., 2019), instead adopts the KL regularization KL(p(·|x̂)‖p(·|x)) that explicitly regularizes the consistency of the predicted probabilities on all classes, whose optimum implies that p(·|x̂) = p(·|x)→ fθ(x, y) = fθ(x̂, y). Inspired by this discussion, alternatively, we can directly regularize the consistency gradient to zero. We achieve this with the following Consistency Regularization (CR) with least square loss:
LCR(θ) = Epd(x,y)⊗ pθ(x) (fθ(x, y)− fθ(x̂, y))2 . (23) We note the our proposed AT+CR differs to ALP (Kannan et al., 2018) as we only minimizes the gap between the logits of the label class fθ(x, y). ALP was shown to be ineffective for improving AT, while in the experiments below, we show that our AT+CR objective indeed achieves comparable (even slightly better) results as TRADES.

heading: A.1.2 EMPIRICAL EVALUATION
text: Experimental setup. Following the conventions, we compare different AT methods with Preactivated ResNet18 (He et al., 2016) and WideResNet34 (Zagoruyko & Komodakis, 2016) on CIFAR-10. The maximum perturbation is bounded by ε = 8/255 under `∞ norm. The training attack is PGD10 (Madry et al., 2018) with random start and step size ε/4. The test attack is PGD20 with random start and step size ε/4. We evaluate both the final epoch and the early stopped epoch with the best robust accuracy.
Result analysis. From Table 3, we can see that the AT+CR objective derived from our framework indeed enjoy much better robustness than vanilla AT. Compared to the state-of-the-art AT variant TRADES, we can see that AT+CR is comparable, and sometimes slightly better at robustness. When the two have similar robustness (for WideResNet34), AT+CR obtains better natural accuracy than TRADES (86.6 v.s. 83.4). These results empirically justify that our interpretation of AT and TRADES from a probabilistic perspective.

heading: A.2 UNSUPERVISED ADVERSARIAL TRAINING
text: Similarly, we can develop the same regularization technique for unsupervised adversarial training through a unified perspective of supervised and unsupervised AT offered by our CEM.

heading: A.2.1 PROPOSED METHOD
text: In Section 5.2, we have developed a principled unsupervised adversarial training routine by an analogy with the supervised AT (Section 4.2). Besides, we can also consider an unsupervised version of the consistency regularization above, namely, the Unsupervised Consistency Regularization (UCR), LUCR(θ) = Epd(x,z)⊗ pθ(x̂)‖fθ(x, z)− fθ(x̂, z)‖2, (24) which encourages the consistency between the similarity between the natural pair (x, z) and the adversarial pair (x̂, z).

heading: A.2.2 EMPIRICAL EVALUATION
text: Experimental setup. Among the many variants of contrastive learning, we adopt SimCLR as our baseline method and take the recently proposed ACL (Jiang et al., 2020) as our Unsupervised Adversarial Training (UAT) following the same default setup as in Section D.1. The training attack configuration is kept the same as the supervised case, while after training, we freeze the encoder and fine-tune a linear classification layer (standard training) on top with labeled data for evaluating the learned features. In particular, we evaluate the composed model on the test data with two different attack methods, FGSM (Goodfellow et al., 2015) (one-step attack) and PGD20 (multi-step attack), both with ε = 8/255, and report their natural and adversarial accuracy, respectively.
Result analysis. As shown in Table 4, features learned by UAT is indeed more robust than standard training, e.g., 0.8% to 21.4% under PGD attack. Moreover, with our proposed UCR regularizer (Eq. 24), we not only effectively improve both natural accuracy (66.6% to 72.0%), and also improve adversarial robustness: 26.2% to 30.7% under FGSM attack and 21.4% to 24.6% under PGD attack. This also helps justify the connection between contrastive learning and our CEM.

heading: A.3 CONCLUDING REMARK
text: With the above experiments on adversarial robustness, we empirically verify that our framework could serve as a valid probabilistic understanding of adversarial training and can be used to develop new effective adversarial training objectives.

heading: B OMITTED TECHNICAL DETAILS
text: For a concise presentation, we have omitted several technical details in the main text. Here we present a complete description of the derivation process.

heading: B.1 LOG LIKELIHOOD GRADIENT OF EBM
text: In Section 3, we have introduced Energy-based Models (EBM) and the gradient of their log likelihood. We now show how it can be derived out.
For a EBM in the following form,
pθ(x) = exp (−Eθ(x))
Z(θ) , (25)
the gradient of the log likelihood can be derived as follows:
∇θEpd(x) log pθ(x) (26) =− Epd(x)∇θEθ(x)−∇θ logZ(θ)
=Epd(x)∇θEθ(x) + ∇θ ∫ x exp(−Eθ(x)) Z(θ)
=− Epd(x)∇θEθ(x) + ∫ x̂ exp(−Eθ(x̂)) Z(θ) ∇θEθ(x̂) =− Epd(x)∇θEθ(x)︸ ︷︷ ︸ positive phase + Epθ(x̂)∇θEθ(x̂)︸ ︷︷ ︸ negative phase .

heading: B.2 CONNECTION BETWEEN STANDARD TRAINING AND JEM
text: In Section 4.4, we have claimed that if we replace the model distribution pθ(x̂) with the data distribution pd(x) in Eqn. 10, the log likelihood gradient of JEM is equivalent to the negative gradient of the CE loss. Here we give a detailed proof as follows:
∇θEpd(x,y) log pθ(x, y) =Epd(x,y)∇θfθ(x, y)−Epθ(x̂)pθ(ŷ|x̂)∇θfθ(x̂, ŷ) ≈Epd(x,y)∇θfθ(x, y)− Epd(x)pθ(ŷ|x)∇θfθ(x, ŷ) =Epd(x,y) [ ∇θfθ(x, y)− Epθ(ŷ|x)∇θfθ(x, ŷ)
] =Epd(x,y) [ ∇θfθ(x, y)−
K∑ k=1 pθ(k|x)∇θfθ(x, k)
]
=Epd(x,y) [ ∇θfθ(x, y)− K∑ k=1 exp(fθ(x, k))∇θfθ(x, k)∑K j=1 exp(fθ(x, j)) ]
=Epd(x,y) [ ∇θfθ(x, y)− K∑ k=1 ∇θ exp(fθ(x, k))∑K j=1 exp(fθ(x, j)) ]
=Epd(x,y) [ ∇θfθ(x, y)− ∇θ ∑K k=1 exp(fθ(x, k))∑K
j=1 exp(fθ(x, j))
]
=Epd(x,y) [ ∇θfθ(x, y)−∇θ log K∑ k=1 exp(fθ(x, k)) ] =Epd(x,y)∇θ log exp(fθ(x, y))∑K k=1 exp(fθ(x, k) =∇θEpd(x,y) log pθ(y|x). (27)

heading: B.3 EQUIVALENCE BETWEEN AT LOSS AND CONTRASTIVE GRADIENT IN SUPERVISED LEARNING
text: In Section 4.3, we have claimed that the contrastive gradient equals to the negative gradient of the robust CE loss (AT loss) following the same deduction in Eqn. 27,
Epd(x,y)⊗ pθ(x̂,ŷ) [∇θfθ(x̂, y)−∇θfθ(x̂, ŷ)] =Epd(x,y)⊗ pθ(x̂) [∇θfθ(x̂, y)− Epθ(ŷ|x̂)∇θfθ(x̂, ŷ)] =Epd(x,y)∇θ log
exp(fθ(x̂, y))∑K k=1 exp(fθ(x̂, k)
=Epd(x,y)⊗ pθ(x̂)∇θ log pθ(y|x̂), (28) which is exactly the negative gradient of the canonical AT loss (Madry et al., 2018).

heading: B.4 EQUIVALENCE BETWEEN INFONCE LOSS AND NON-PARAMETRIC CEM
text: In Section 6.1, we have claimed that the the log likelihood gradient of NP-CEM equals to exactly the negative gradient of the InfoNCE loss when we approximate pθ(x) with pd(x). The derivation is presented as follows:
Epd(x,z)∇θfθ(x, z)− Epθ(x̂,ẑ)∇θfθ (x̂, ẑ) =Epd(x,z)∇θfθ(x, z)− Epθ(z)pθ(x̂|ẑ)∇θfθ (x̂, ẑ) =Epd(x,z)∇θfθ(x, z)− Epθ(ẑ)pθ(x̂) pθ(x̂|ẑ) pθ(x̂) ∇θfθ(x̂, ẑ)
=Epd(x,z)∇θfθ(x, z)− Epθ(x̂)pθ(ẑ) pθ(x̂|ẑ)
Epθ(ẑ)pθ(x̂|ẑ) ∇θfθ(x̂, ẑ)
≈Epd(x,z)∇θfθ(x, z)− Epd(x̂)pd(ẑ) pθ(x̂|ẑ)
Epd(ẑ)pθ(x̂|ẑ) ∇θfθ(x̂, ẑ)
≈ 1 N N∑ i=1
[ ∇θfθ(xi, zi)−
K∑ k=1 exp(fθ(x, z − ik))∇θfθ
( x, z−ik )∑K j=1 exp(fθ(x, z − ik)) ]
= 1
N N∑ i=1
[ ∇θfθ(xi, zi)−∇θ log
K∑ k=1 exp(fθ(x, z − ik))
]
≈ 1 N N∑ i=1 ∇θ log exp(fθ(xi, zi))∑K k=1 exp(fθ(xi, z − ik)) . (29)

heading: B.5 EQUIVALENCE BETWEEN ADVERSARIAL INFONCE AND NP-CEM
text: In Section 6.2, we have developed the unsupervised analogy of AT loss and regularization. In particular, we have claimed that contrastive gradient is equivalent to the gradient of the Adversarial InfoNCE loss (i.e., the InfoNCE loss of the adversarial example x̂) utilized in previous work (Jiang et al., 2020; Ho & Vasconcelos, 2020; Kim et al., 2020). It can be derived following Eqn. 29:
Epd(x,z)⊗ pθ(x̂,ẑ) [∇θfθ(x̂, z)−∇θfθ(x̂, ẑ)] =Epd(x,z)⊗ pθ(x̂) [∇θfθ(x̂, z)− Epθ(ẑ)pθ(x̂)∇θfθ(x̂, ẑ)] =Epd(x,z)⊗ pθ(x̂) [ ∇θfθ(x̂, z)− Epθ(ẑ)pθ(x̂)
pθ(x̂|ẑ) pθ(x̂)
∇θfθ(x̂, ẑ) ]
=Epd(x,z)⊗ pθ(x̂) [ ∇θfθ(x̂, z)− Epθ(ẑ)
pθ(x̂|ẑ) pθ(x̂)
∇θfθ(x̂, ẑ) ]
≈ 1 N N∑ i=1 ∇θ log exp(fθ(x̂i, zi))∑K k=1 exp(fθ(x̂i, ẑ − ik)) . (30)

heading: C DETAILS OF THE PROPOSED ALGORITHMS
text: In this section, we provide a detailed description of several algorithms we propose.

heading: C.1 SUPERVISED ADVERSARIAL SAMPLING ALGORITHMS
text: We describe the general procedure of supervised adversarial sampling in Algorithm 1.
Algorithm 1 General Supervised Adversarial Sampling Input: trained robust classifier fθ(x), test dataset Dtest; step size α, `2-norm maximal range β, noise scale η, number of sampling steps N , and specific update function g(x); Output: drawn sample x̂N ;
Initialization. Drawn seed x̂1 = δ as a random noise n ∼ N (µ,Σ), where µ,Σ are estimated from Dtest; for each step n ∈ [1, N − 1] do
Proposal step. x′ = x̂n + α g(x̂n)‖g(x̂n)‖2 + ηε, ε ∼ N (0,1); Projection step. Project x′ to the feasible set by: if ‖x′ − x̂n‖2 ≤ β then
Let x̂n+1 = x′; else
Let x̂n+1 = x̂n + β x ′−x̂n
‖x′−x̂n‖2 ;
end if
The four sampling algorithms discussed in our paper have the following update functions g(x):
• Targeted Attack (TA) (Santurkar et al., 2019): given a drawn class y ∼ pd(y) (fixed along the updating), the update rule of TA follows
gTA(x) = ∇x log pθ(y|x)
=∇xf(x, y)−∇x [ log
K∑ k=1 exp(fθ(x, k))
] .
(31)
• Our Conditional Sampling (CS): given a randomly drawn class y ∼ pd(y) (fixed along the updating), the update rule of CS follows
gCS(x) = ∇x log pθ(x|y) = ∇xf(x, y). (32)
• Our Reinforced Conditional Sampling (RCS): given a randomly drawn class y ∼ pd(y) (fixed along the updating), the update rule of RCS follows
gRCS(x) = ∇xf(x, y) +∇x [ log
K∑ k=1 exp(fθ(x, k))
] . (33)

heading: C.2 UNSUPERVISED ADVERSARIAL SAMPLING ALGORITHMS
text: We describe the general procedure of supervised adversarial sampling in Algorithm 2.
The two sampling algorithms discussed in our paper have the following update functions g(x):
• PGD. Given a batch of examples {x̂k}Kk=1, the PGD attack has the following update function g(x̂k) for x̂k,
gPGD(x̂ k) = − log exp(fθ(x̂ k, x̄k))∑K j=1 exp(fθ(x̂ k, x̂j)) ,
where its positive pair x̄k is a randomly drawn sample.
Algorithm 2 General Unsupervised Adversarial Sampling Input: trained unsupervised robust model fθ(x), test dataset Dtest; step size α, `2-norm maximal range β, noise scale η, number of steps N , batch size K, and specific update function g(x); Output: drawn samples {x̂kN}Kk=1;
Initialization. DrawnK seeds {x̂k1}Kk=1, where x̂k1 = δk refers to a random noise δk ∼ N (µ,Σ), where µ,Σ are estimated from Dtest; for each step n ∈ [1, N − 1] do
Proposal step. For each sample xkn at step n, we propose (xk)′ = x̂kn+α g(x̂kn) ‖g(x̂kn)‖2
+ηεk, εk ∼ N (0,1);
Projection step. For each sample xkn at step n, we project (xk)′ to the feasible set by: if ‖(xk)′ − x̂kn‖2 ≤ β then
Let x̂kn+1 = (x k)′;
else Let x̂kn+1 = x̂ k n + β (xk)′−x̂kn ‖(xk)′−x̂kn‖2 ;
• Langevin. Given a batch of examples {x̂k}Kk=1, the Langevin dynamics has the following update function g(x̂k) for x̂k,
gLangevin(x̂ k) = − log K∑ j=1 exp(fθ(x̂ k, x̂j)).

heading: D MORE DETAILS AND RESULTS ON ADVERSARIAL SAMPLING

heading: D.1 DETAILED EXPERIMENTAL SETUP
text: Supervised Adversarial Sampling. For supervised robust models, we adopt the same pretrained ResNet50 checkpoint on CIFAR-10 as Santurkar et al. (2019) 4 for a fair comparison. As described in Santurkar et al. (2019), the ResNet-50 model is adversarially trained for 350 epochs with learning rate 0.01 and batch size 256. The learning rate is dropped by 10 at epoch 150 and 200. The training attack is `2-norm PGD attack with random start, maximal perturbation norm 0.5, step size 0.1 and 7 steps.
Unsupervised Adversarial Sampling. As far as we know, we are the first to consider sampling from unsupervised robust models. Therefore, to obtain an unsupervised robust model for sampling, we adopt ACL (Jiang et al., 2020) as the baseline method and follow their default hyper-parameters 5 to train it. The official implementation of ACL is built upon the SimCLR (Chen et al., 2020) framework for contrastive learning, while they choose a specific hyper-parameter configuration for
4We download the checkpoint from the repository https://github.com/MadryLab/ robustness_applications.
5Official code: https://github.com/VITA-Group/Adversarial-Contrastive-Learning.
adversarial training. In particular, they choose 512 for batch size and train for 1000 epochs with ResNet-18 backbone (He et al., 2016). The base learning rate is 1.0, where they use a linear warm up strategy for the first 10 epochs, and apply cosine annealing scheduler after that. The training attack is kept the same as that of the supervised case for a fair comparison. For ResNet-18, we adopt the ACL(A2A) setting, which adopts the normal ResNet with only one Batch Normalization module. Instead, for ResNet-50, we notice that the ACL(A2S) setting yields slightly better results. The ResNet variants in the ACL(A2S) setting contains two BN modules, where we assign natural and adversarial examples to different modules. We refer more details to the original paper (Jiang et al., 2020).
Evaluation of sample quality. Note that there are four hyper-parameters in our sampling protocol: step size α, `2-ball size β, noise scale η, and iteration steps K, for which we list our choice in Table 5. We evaluate sample quality quantitatively w.r.t. the Inception Score (IS) (Salimans et al., 2016) and Fréchet Inception Distance (FID) (Heusel et al., 2017) with 50,000 samples, where the standard variation of IS is around 0.1.

heading: D.2 ADDITIONAL ANALYSIS
text: Besides the results presented in the main text, we also conduct more experiments to analyze the behavior of our proposed adversarial sampling algorithms, both quantitatively and qualitatively. We conduct a detailed analysis of our proposed sampling algorithms and present the results of supervised adversarial sampling (with RCS) in Figure 2 and the results of unsupervised adversarial sampling (with Langevin) in Figure 3. Note that in both cases, we adopt the ResNet-50 backbone and use the default hyper-parameters unless specified.

heading: D.2.1 CHAIN LENGTH
text: From the left panels of Figure 2 and Figure 3, we can see the two adversarial algorithms both have a sweet spot of sampling steps N (length of the sampling Markov chains) at around 30 to 40 steps, before and after which the results will be slightly worse.

heading: D.2.2 NOISE RATIO
text: In the proper Langevin dynamics, the scale of the noise is determined by the step size, η = √
2α. However, in practice, this would lead to a catastrophically degraded sample quality as the noise takes over the gradient information. Therefore, following Song & Ermon (2019) and Grathwohl
et al. (2019), we also anneal the noise ratio η to a smaller value for better sample quality. As shown in the middle panels of Figure 2 and Figure 3, the optimal noise ratio is around 0.01 for both cases.

heading: D.2.3 MAXIMAL NORM
text: An apparent difference of our adversarial sampling algorithms to the canonical Langevin dynamics is that ours have a projection step that limits the distance between the samples and the initial seeds. In the right panels of Figure 2 and Figure 3, we show the impact of the scale of the `2-norm ball for the sample quality. We can see that generally speaking, as the ball grows larger, the samples get refined. In the supervised case, the sample quality gets slightly worse with a large norm, which does not happen in the unsupervised case.

heading: D.2.4 SAMPLING TRAJECTORY
text: Aside from the qualitative inspection of the proposed sampling algorithms, we also demonstrate the sampling trajectory of our supervised (RCS) and unsupervised (Langevin) adversarial sampling methods in Figure 4 & 5. We can see that the samples get gradually refined in terms both low-level textures and high-level semantics.

Review summaries:
This paper justifies why models trained with adversarial training are good generative models by proposing a probabilistic framework. The paper is heavily influenced by the JEM paper and proposes a generalized form of analysis in that paper to include the unsupervised scenario. The generalization defines a joint distribution $p(x, z)$ over data $x$ and representations $z$. For the supervised case, each class is represented by a "center" $w_y$ and for the unsupervised one, $p(x, z) = p(x) p(z|x)$ and $p(z|x)$ is reparametrized with augmentation.  Then the authors use this probabilistic framework to analyze the adversarial training. Specifically, they show PGD is a biased form of the Langevin dynamics in their formulation. Then they show how the maximum likelihood training of their probabilistic framework is similar to adversarial training. As a result, they conclude adversarial training leads to a model with high generative power. Finally, they propose refined sampling strategies from adversarially trained models.  In the last part of the paper, they use their framework to propose an unsupervised adversarial training method followed by sampling algorithms from the produced models.

This paper proposed a unified probabilistic framework, dubbed as Contrastive Energy-based Models, to understand the robustness and generative capability. The proposed CEM is a special case of EBM that models the joint distribution over two variables with a similarity function defined in a contrastive form. CEM could be instantiated into parametric form and non-parametric form, which work for supervised learning and unsupervised learning respectively. CEM could demystify adversarial training's generative capability in both supervised and unsupervised setting. Moreover, with CEM, adversarial training could be extended to unsupervised scenario.

The submission proposes a unified probabilistic framework to illustrate the generative capability of adversarial training. It also offers a unified perspective of adversarial training and sampling from both a supervised learning setting and an unsupervised learning setting. The proposed adversarial sampling strategy from the method is extensively demonstrated on different benchmarks, showing better quality compared with existing related works.