Your classifier is secretly an energy based model and you should treat it like one
Authors: Will Grathwohl
Kuan-Chieh Wang
Joern-Henrik Jacobsen
David Duvenaud
Mohammad Norouzi
Kevin Swersky
Venue:  (2020)
Keywords: energy based models
adversarial robustness
generative models
out of distribution detection
outlier detection
hybrid models
robustness
calibration

Abstract: We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x, y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may be used and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, and out-of-distribution detection while also enabling our models to generate samples rivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and present an approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-art in both generative and discriminative learning within one hybrid model. 

Published as a conference paper at ICLR 2020
YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED MODEL AND YOU SHOULD TREAT IT LIKE ONE
Will Grathwohl
Kuan-Chich Wang Jörn-Henrik Jacobsen'
University of Toronto & Vector Institute Google Research
University Toronto & Vector Institute
wangkuallcs. toronto. edu
wgrathwohl @cs. toronto.oooou
jejeccdssee@eectorinssiiii at
David Duvenaud
Kevin Swersky & Mohammad Norouzi Google Research fkswersky, mmorouzi]@google.
University of Toronto & Vector Institute duvenaud8 cs. toronto. edu
ABSTRACT
We propose to reinterpret a standard discriminative classifier of p(y x) as an en- ergy based model for the joint distribution p(x, y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(xly). Within this framework, standard discriminative architectures may be used and the model can also be trained on unlabeled data. We demonstrate that en- ergy based training of the joint distribution improves calibration, robustness, and ditto--distribution detection while also enabling our models to generate samples rivaling the quality of recent GAN approaches. We improve upon recently pro- posed techniques for scaling up the training of energy based models and present an approach which adds little overhead compared to standard classification train- ing. Our approach is able to achieve performance rivaling the state-of-the-ard in both generative and discriminative learning within one hybrid model.
INTRODUCTION
For decades, research on generative models has been motivated by the promise that generative mod- els can benefit downstream problems such as semi-supervised learning, imputation of missing data, and calibration of uncertainty (e.g.. [Chapelle et al. (2006); Dempster et al. (197)).. Yet, most re- cent research on deep generative models ignores these problems, and instead focuses on qualitative sample quality and log- likeeioodoood on heldout validation sets.
Currently, there is large performance gap between the strongest generative modeling approach to downstream tasks of interest and hand-tailored solutions for each specific problem. One po- tential explanation is that most downstream tasks are discriminative in nature and state-of-the-art generative models have diverged quite heavily from stte-of-thheaar discriminative architectures. Thus, even when trained solely as classifiers, the performance of generative models far below the performance of the best discriminative models. Henee, the potential benefit from the generative component of the model is far outweighed by the decrease in discriminative performance. Recent work (Behrmann et al. 2018: Chen al. 2019) attempts to improve the discriminative performance of generative models by leveraging invertible architectures, but these methods still underperform their purely discriminative counterparts jointly trained generative models.
This paper advocates the use of energy based models (EBMs) to help realize the potential of gen- erative models on downstream discriminative problems. While EBMs currently challenging to work with, they fit more naturally within a discriminative framework than other generative models and facilitate the use of modern classifier architectures. Figure [ illustrates an overview of the ar- chitecture, where the logits of classifier are re-interpreted to define the joint density of data points and labels and the density of data points alone.
"Equal Contribtuion

---

Published as a conference paper at ICLR 2020
The contributions of this paper be summarized as: 1) We present a novel and intuitive framework for joint modeling of labels and data. 2) Our models consider- ably outperform previous state-of-the-art hybrid models both generative and discriminative modeling. 3) We show that the incorporation of generative modeling gives our models improved calibration, out-of-distribution de- tection, and adversarial robustness, performing on par with or better than hand-tailored methods for multiple tasks.
Classifier
JEM
log ((y) x)
Identity
log v)
Softmax
2 ENERGY BASED MODELS
Energy based models (LeCun et al. 2006) hinge on the observation that any probability density p(x) for RRD can be expressed as
LxgSumExp
Neural Net.
po(x)= (sp(-Eg(x))
log ((x)
(1)
((0)
where Ex( : RD + R. known as the energy function, exp(-Ex
maps each point to scalar, and Z(0) the normalizing constant (with respect to x) also known the partition function. Thus,
parameterize EBM using any function that takes x as the input and re- turns scalar.
Figure 1: Visualization of our method, JEM, which defines a joint EBM from classifier architectures.
For most choices Eg, one cannot compute or even reli- ably estimate (0), which means estimating normalized densities is intractable and standard maxi- mum likelihood estimation of the parameters, 0, is not straightffrward. Thus, we must rely on other methods train EBMs. We note that the derivative of the log-likeldhood for single example x with respect to can be expressed as
i log (x))
Exa)'
E
R.p (xx)
00
00
00
where the expectation is over the model distribution. Unfortunately. we cannot easily draw samples from Po (x), so we must resort MCMC to use this gradient estimator. This approach was used to train some of the earliest EBMs. For example, Restricted Boltzmann Machines (Hinton) 2002) were trained using a block Gibbs sampler to approximate the expectation Eq. (
Despite a long period of little development, there has been recent work using this method to train large-scale EBMs on high-dimensional data, parameterized by deep neural networks (Nijkamp) al. 2019b at Du & Mordatch, 20199; Xie et al., 2016). These recent successes have approxi- mated the expectation in Eq. (2) using a sampler based on Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh. 201 which draws samples following
a (x(xx) 0x;
Xg
(x().
X;+1 x;
+c, c~~(00.)
(3)
where (x) is typically a Uniform distribution over the input domain and the step-size should be decayed following a polynomial schedule. practice the step-size, a, and the standard deviation of is often chosen separately leading to biased sampler which allows for faster training. See Appendix H.I for further discussion of samplers for EBM training.
WHAT YOUR CLASSIFIER IS HIDING
In modern machine learning, a classification problem with K classes is typically addressed using parametric function, fo R K which maps each data point E R to K real-valued numbers known as logits. These logits are used to parameterize categorical distribution using the so-called Softmax transfer function:
exp (x)(
Po x) =
(4)
En exp (fo(x)(b))
2

---

Published as a conference paper at ICLR 2020
where fo (x)]ll indicates the nth index of fo(x), i.e., the logit corresponding the the yth class label.
Our key observation in this work is that one can slightly re-interpret the logits obtained from fe to define p(x.3) 3) and p(x) as well. Without changing fa. one can re-use the logits to define energy based model of the joint distribution of data point x and labels y via:
exp (ff(()))))
Pe (x,
(5)
Z(0)
where 2(0) is the unknown normalizing constant and (x y) = fo (x)
By marginalizing out y, we obtain an unnormalized density model for x as well,
(x Σ po 3) Ey exp (x)())
2(0)
Notice now that the LogSumExp(-)) of the logits of any classifier can be re-used to define the energy function at a data point x as
Ex(x) LogSumEE log Σ, exp(fo(x)(v))
Unlike typical classifiers, where shifting the logits fxx) by an arbitrary scalar does not affect the model at all, our framework, shifting the logits for a data point x will affect log po().. Thus, we are making use of the extra degree of freedom hidden within the logits to define the density function over input examples as well as the joint density among examples and labels. Finally, when compute po(y x) via (x(x, (x) by dividing Eq. (5) to Eq. (6) the normalizing constant cancels out, yielding the standard Softmax parameterization generative model hidden within every standard discriminative model! Since approach proposes to reinterpret a classifier as Joint Energy-based Model we refer to it throughout this work as JEM.
Eq. 4 Thus, we have found a
4 OPTIMIZATION
We now wish to take advantage of our new interpretation of classifier architectures to gain the ben- efits of generative models while retaining strong discriminative performance. Since our model's parameterization of p(y(x) is normalized over it is simple to maximize its likelihood as in stan- dard classifier training. Since our models for p(x) and p(x, y) are unnormalized, maximizing their likelihood is not easy. There are many ways we could train So to maximize the likelihood of the data under this model. We could apply the gradient estimator of Equation to the likelihood under the joint distribution of Equation 5 Using Equations 6 and 4
we can also factor the likelihood as
log (x. y) log pe (x) log po(3)xx).
The estimator of Equation 2 is biased when using MCMC sampler with finite number of steps. Given that the goal of our work to incorporate EBM training into the standard classification setting, the distribution of interest is p(y|x). For this reason propose train using the factorization of Equation
[ to ensure this distribution is being optimized with an unbiased objective. We optimize p(y]x) using standard cross-entropy and optimize log p(x) using Equation 2 with SGLD where gradients are taken with respect to LogSumExp,, ( (v)).. We find alternative factorings of the likelihood lead considerably worse performance as can be seen in Section 5..1
Following Du & Mordatch (019)) we use persistent contrastive divergence (Tieleman 2008 to esti- mate the expectation the right-hand-side of Equation since gives an order of magnitude savings in computation compared to seeding new chains at each iteration as in Nijkamp et al. (201%)) This comes the cost of decreased training stability. These trade-offs are discussed in Appendix H.2
APPLICATIONS
We completed a thorough empirical investigation to demonstrate the benefits JEM over stan- dard classifiers. First, we achieved performance rivaling the state of the art in both discriminative and generative modeling. Even more interesting, we observed a number of benefits related to the practical application of discriminative models including improved uncertainty quantification, out- of-distribution detection, and robustness to adversarial examples. Generative models have been long-expected to provide these benefits but have never been demonstrated to do so at this scale.
3

---

Published as a conference paper at ICLR 2020
Class
Model
Accuracy% f
1ST
FID_
Residual Flow
70.3
3.6
46.4
Glow
67.6
3.92
48.9
Hybrid
IGEBM
49.1
8.3
37.9
JEM p(xly) factored
30.1
6.36
61.8
JEM (Ours)
92.9
8.76
38.4
Disc.
Wide-Resnet
95.8
N/A
N/A
SNGAN
N/A
8.59
25.5
Gen.
NCSN
N/A
8.91
25.32
Table CIFARIO Hybrid modeling Results.
Residual
Figure 2: CIFAR10 class- conditional samples.
Flow (Chen et al. 2019), Glow (Kingma & Dhariwal (018). |
IGEBM (Du Mordatch 2019), SNGAN (Miyato et al.)
2018 NCSN (Song & Ermon 2019)
All architectures used are based on Wide Residual Networks Zagoruyko & Komodakis 2016) where we have removed soncc-oommaiiztion to ensure that our models' outputs are determinis- tic functions of the input. This slightly increases classification error of a WRN-28-11 10 from 4.2% to 6.4% on CIFARIO and from 2.3 to 3.4% on SVHN.
All models were trained in the same way with the same papree-arameters which were tuned on CIFARIO. 10. Intriguingly, the SGLD sampler parameters found here generalized well across datasets and model architectures. All models are trained on single GPU in approximately 36 hours. Full experimental details can be found in Appendix
SVHN
5.1 HYBRID MODELING
2
First, we show that a given classifier architecture can be trained as an EBM achieve competitive performance as both classifier and gen- erative model. We train JEM on CIFARIO, SVHN, and CIFAR100 100 and compare against other hybrid models well standalone generative and discriminative models. We find JEM performs near the state of the art in both tasks simultaneously. outperforming other hybrid models (Table I
3
5 $55 5
CIFARI00
Given that we cannot compute normalized likelihoods, we present incep- tion scores (IS) (Salimans al. 2016) and Frechet Inception Distance (FID) (Heusel et al. (2017) as a proxy for this quantity. We find that JEM competitive with SOTA generative models these metrics. These met- rics are not commonly reported on CIFAR100 and SVHN so we present accuracy and qualitative samples on these datasets. Our models achieve 96.7% and 72.2% accuracy on SVHN and CIFAR 100, respectively. Sam- ples from JEM can be seen in Figures
and in Appendix C
JEM is trained to maximize the likelihood factorization shown in Eq. 8 This was to ensure that no bias added into our estimate of pog((y) X
Figure 3:
Class-
conditional samples.
which can be computed exactly in setup. Prior work (Du & Mordatch, 2019 Xie et al. 2016)
proposes to factorize the objective as log p(xly) ( + log p(y). In these works, each p(x(z) ) is separate EBM with a distinct, unknown normalizing constant, meaning that their model cannot be used to compute p(yl x) or p(x). This explains why the model of Du & Mordatch (2019) (we will refer to this model as IGEBM) is not competitive classifier. As ablation, trained JEM maximize this objective and found a considerable decrease in discriminative performance (see Table | row 4).
5.2 CALIBRATION
A classifier is considered calibrated if its predictive confidence, max, p(y[x), aligns with its mis- classification rate. Thus, when calibrated classifier predicts label y with confidence .9 it should have a 90% chance of being correct. This is an important feature for model to have when deployed real-world scenarios where outputting an incorrect decision can have catastrophic consequences. The classifier's confidence can be used to decide when to output prediction or deffer to human,
'This was done to remove sources of stochasticity in early experiments. Since then we have been able to successfully train Joint-EBMs with Batch Normalization and other forms of stochastic regularization (such as dropout) without issue. We leave the incorporation of these methods further work.
4

---

Published as a conference paper at ICLR 2020
for example. Here, well-calibrated, but less accurate classifier can be considerably more useful than more accurate, but calibbaad model.
While classifiers have grown more accurate in recent years, they have also grown consider- ably less calibrated (Guo et al.] 2017). Contrary to this behav- ior, we find that JEM notably im- proves classification while retain- ing high accuracy.
Baseline
JEM
1.0
1.0
ECE: 22.32%
BCB: 4.87%
Accuracy
We focus on CIFARI00 since SOTA classifiers achieve approx- imately 80% accuracy. We train JEM on this dataset and com- pare to baseline of the same architecure without EBM train- ing. Our baseline model achieves 74.2% accuracy and JEM achieves 72.2% (for reference, ResNet- 110 achieves 74.8% accu- racy (Zagoruyko & Komodakis) (016)). We find the baseline model very poorly calibrated out- putting highly over-confident predictions. Conversely, ibrated classifier when measured with Expected Calibration Error (see Appendix [E.I]. Compared other calibration methods such as Platt scaling (Guo al. 2017). JEM requires no additional training data. Results can be seen in Figure and additional results can be found in Appendix E.2
0.8.0
0.8.0
0.5
1.0
0.5 Confidence
1.0
Confidence
Figure 4:
CIFAR 100 calbration results. ECE Expected Cali-
bration Error (Guo al. 2017). see Appendix E.
find JEM produces nearly perfectly cal-
5.3 OUT-OI-DISTRIBUTION DETECTION
In general, -id-tii-iitiittoonn (OOD) detection a binary classification problem, where the model is required to produce a score
sp (x) E R,
where x is the query, and f is the set of learnable parameters. We desire that the scores for in-distribution examples are higher than that out-of-distribution examples. Typically for evalu- ation, threshold-free metrics are used, such as the area under the receiver-operating curve (AU- ROC) (Hendrycks & Gimpel, 2016). There exist number of distinct OOD detection approaches to which JEM can be applied. We expand on them below. Further results and experimental details can be found in Appendix F2
5.3.1 INPUT DENSITY
natural approach to OOD detection is fit a density model on the data and consider examples with low likelihood to be OOD. While intuitive, this approach is currently not competitive on high- al. (2018) showed that tractable deep generative models such as Kingma & Dhariwal (2018) and Salimans al.) (201)) can assign higher densities to OOD exam- ples than in-distribution examples. Further work (Nalisnick et al. 2019) shows examples where the densities of an OOD dataset are completely indistinguishable from the in-distribution set, e,g. Table column 1. Conversely, Du & Mordatch| (2019) have shown that the likelihoods from EBMs be reliably used predictor for OOD inputs. As be seen in Table Saole@]oole column 2, JEM con- sistently assigns higher likelihoods to in-distribution data than OOD data. One possible explanation for JEM's further improvement training while also being able to derive a principled model of p(x). Intriguingly, Glow does not appear benefit the same way from this supervision is demonstrated by the little difference between our unconditional and class-conditional Glow results. Quantitative results can be found in Table (top).
dimensional data. Nalisnick
IGEBM is its ability incorporate labeled information during
5.3.2 PREDICTIVE DISTRIBUTION
Many successful approaches have utilized classifier's predictive distribution for OOD detec- tion (Gal Ghahramani 2016 Wang et al., 20188] Liang et al.) (017). useful OOD score that be derived from this distribution the maximum prediction probability: so(x) -mxx, Po x)
5

---

Published as a conference paper at ICLR 2020
(Hendrycks & Gimpel 2016). It has been demonstrated that OOD performance using this score is highly correlated with a model's classification accuracy. Since JEM is a competitive classifier, we find it performs on par (or beyond) the performance of strong baseline classifier and considerably outperforms other generative models. Results can be seen in Table 3 (middle).
5.3.3 A NEW SCORE: APPROXIMATE MASS
has been recently proposed that likelihood may not be enough for OOD detection high dimen- sions (Nalisnick al. 2019). It is possible for a point have high likelihood under a distribution yet be nearly impossible to be sampled. Real samples from a distribution lie in what is known as the "typical" set. This is the area of high probability mass. single point may have high density but if the surrounding areas have very low density, then that point is likely not in the typical set and therefore likely not a sample from the data distribution. For a high-likelihood datapoint outside of the typical set, we expect the density to change rapidly around it, thus the norm of the gradient of the log- degdeesity will be large compared to examples in the typical set (otherwise it would be in an area of high mass). We propose alternative OOD score based on this quantity:
Ô log po (x) 8x
sx
and other gen- erative model's likelihoods see Table columm 3. For tractable likelihood methods we find this predictor is anti-correlated with the model's likelihood and neither is reliable for OOD detection. Results can be seen Table 3 (bottom).
For EBMs (JEM and IGEBM), find this predictor greatly outperforms
5.4 ROBUSTNESS
Recent work (Athalye et al. 2017) has demonstrated that classifiers trained to be adversarially robust be re-purposed to generate convincing images, do in-painting, and translate examples from one class to another. This is done through an iterative refinement procedure, quite similar to the SGLD used sample from EBMs. We also note that adversarial training (Goodfellow et al. 2014) bears many similarities to SGLD training of EBMs. In both settings, we use gradient-basee optimiza- tion procedure generate examples which activate a specific high-level network activation, then optimize the weights of the network to minimize the generated example's 's effect on that activation. Further connections have been drawn between adversarial training and regularizing the gradients of the network's activations around the data (Simon-Gabriel et al.) 2018). This is similar to the objec- tive of Score Matching (Hyvärinen) 2005) which can also be used train EBMs (Kingma Lecun 2010 Song & Ermon 2019).
Given these connections one may wonder if a classifier derived from an EBM would be more robust to adversarial examples than a standard model. This behavior has been demonstrated in prior work on EBMs (Du & Mordatch 2019) but their work did not produce a competitive discriminative model
Glow log logp(z)
JEM (())
Approx. Mass JEM
SVHN
CIFARI00
CelebA
Table Histograms for OOD detection. All models trained on CIFARIO, Green corresponds to the score (in-distribution) CIFAR10, 10, and red corresponds the score the OOD dataset.

---

Published as a conference paper at ICLR 2020
CIFARIO
so (x)
Model
SVHN
Interp
CIFAR100
CelebA
Unconditional Glow
.05
.51
.55
.57
Class- Conditioonl Glow
.07
.45
.51
.53
logp(x)
IGEBM
.63
.70
.50
.70
JEM (Ours)
.67
.65
.67
.75
Wide-ResNet
.93
.77
.85
.62
ClassConditional Glow
.64
.61
.65
.54
maxy p(y)x)
IGEBM
.43
.69
.54
.69
JEM (Ours)
.89
.75
.87
.79
Unconditional Glow
.95
.27
.46
.29
a Blog p(x)
Cons--onniiional Glow
.47
.01
.52
.59
8x
IGEBM
.84
.65
.55
.66
JEM (Ours)
.83
.78
.82
.79
Table OOD Detection Results. Models trained on CIFARIO. Values are AUROC.
and is therefore of limited practical application this purporse. Similarly, we find JEM achieves considerable robustness without sacrificing discriminative performance.
5.4.1 IMPROVED ROBUSTNESS THROUGH EBM TRAINING
common threat model for adversarial robustness that of perturbation-baseed adversarial examples with Lo-norm constraint (Goodfellow et al., (014). They are defined as perturbed inputs = 0, which change model's pre-
Baseline
JEM-0
JEM-1
Accuracy
to-
JEM 10
diction subject to | x < e. These examples exploit se- mantically meaningless perturbations which the model overly sensitive. However, closeness to real inputs in terms of a given metric does not imply that adversarial examples reside within areas of high density according to the model distribution, hence it is not surprising that the model makes mistakes when asked to classify inputs it has rarely never encountered during training.
Training
RandkdySmooth
20-
(a) Loe Robustness
This insight has been used to detect and robustly classify adversarial examples with generative models (Song et al.
2017 Li al., [0188 Fetaya et al.) 2019).. The state-of-the- art method for adversarial robustness on MNIST classifies by comparing an input samples generated from class- conditional generative model (Schott et al. 2018). This can be thought of as classifying an example similar to the input but from an area of higher density under the model's learned distribution. This refined input resides areas where the model has already "seen" sufficient data and is thus able to accurately classify. Albeit promising, this family of meth- ods has not been able to scale beyond MNIST due to lack of sufficiently powerful conditional generative models. We believe JEM can help close this gap. We propose run few iterations of our model's sampling procedure seeded at given input. This should be able to transform low-probability inputs nearby point of high probability, "undoing" any adversarial attack and enabling the model to classify robustly.
Accurary
10
600
(b) L2 Robustness
Figure 5: Adversarial Robustness Re- sults with PGD attacks. JEM adds considerable robustness.
We number of powerful adversarial attacks els. We run white-box PGD attack, giving the attacker access to the gradients through our sam- pling procedure Because our sampling procedure is stochastie, compute the "expectation over transformations" Athalye et al. (2018), the expected gradient over multiple runs of the sampling
Perturbation Robustness
CIFAR10 mod-
In Du & Mordatch 2019) the attacker not given access the gradients of the refinement procedure.
We these stronger attacks their model well and provide comparison in Appendix

---

Published as a conference paper at ICLR 2020
procedure. We also run gradient-free black-box attacks; the boundary attack (Brendel et al. 2017) and the brute-force pointwise attack (Rauber et al. 2017). All attacks are run with respect to the L2 and Lo norms and we test JEM with 0, 1, and 10 steps of sampling seeded at the input.
Results from the PGD experiments can be seen in Figure S Experimental details and remaining results, including gradient-free attacks, can be found in Appendix G Our model is considerably more robust than a baseline with standard classifier training. With respect to both norms, JEM delivers considerably improved robustness when compared the baseline but for many epsilons falls below state-of-the-art adversarial training (Madry et al. 2017 Santurkar et al., 2019) and the sfate-of-the-art certified robutness method Salman et al. (2019) ( "RandAdySmooth in Figure (5) We note that each of these baseline methods is trained to be robust to the norm through which it being attacked and it has been shown that attacking an L adversarially trained model with an T2 adversary decreases robustness considerably (Madry et al. (017). However, we attack the same JEM model with both norms and observe competitive robustness in both cases.
JEM with 0 steps refinement is noticeably more robust than the baseline model trained as a standard classifier, thus simply adding EBM training can produce more robust models. We also find that increasing the number of refinement steps further increases robustness to levels at robustness-specile approaches. We expect that increasing the number of refinement steps will lead to more robust models but due to computational constraints we could not run attacks in this setting.
Distal Adversarials Another common failure mode of non-robust models is their tendency classify non- sensical inputs with high confidence. To analyze this property, we follow (Scoott et al. (2018). Starting from noise we generate images to maximize ply = ''car"(x..
CNN
ADV
Results are shown in figure [ The baseline confidently classifies unstructured noise images. The Le adversari- ally trained ResNet with 0.5 (Santurkar al.) 2019) confidently classifies somewhat structured, but unreal- istic images. JEM does not confidently classify nonsen- sical images, so instead, car attributes and natural image properties visibly emerge.
JEM
Figure 6: Distal Adversarials. Con- fidently classified images generated from noise, such that: ply = "car"(x) > .9.
LIMITATIONS
Energy based models can be very challenging to work with. Since normalized likelihoods cannot be computed, can be hard to verify that learning is taking place at all. When working in domains such as images, samples can be drawn and checked to assess learning, but this is far from a generalizable strategy. Even so, these samples are only samples from an approximation to the model so they can only be so useful. Furthermore, the gradient estimators we use to train JEM are quite unstable and are prone to diverging if the sampling and optimization parameters are not tuned correctly. Regularizers may be added (Du & Mordatch] 2019) to increase stability but it is not clear what effect they have on the final model. The models used to generate the results in this work regularly diverged throughout training. requiring them to be restarted with lower learning rates with increased regularization. See Appendix H.3 for a detailed description of how these difficulties were handled.
While this may seem prohibitive, believe the results presented in this work sufficient to motivate the community to find solutions to these issues as any improvement in the training of energy based models will further improve the results we have presented in this work.
7 RELATED WORK
Prior work (Xie et al. 2016) made a similar observation to ours about classifiers and EBMs but define the model differently. They reinterpret the logits to define class- caas-conniiional EBM p(xly), similar to Du & Mordatch| (2019). This setting requires additional parameters to be learned to derive classifier and an unconditional model. We believe this subtle distinction is responsible for our model's 's success. The model of (Song& & Ou 2018) is similar as well but is trained using a GAN-IiNe generator and is applied to different applications. Also related are Introspective Networks (Jin et al.) 2017 Lee et al.) 2018) which have drawn a similar connection between discriminative classifiers and generative models. They derive generative model from classifier which learns to distinguish
8

---

Published as a conference paper at ICLR 2020
between data and negative examples generative via MCMC-like procedure. Training in this way has also been shown to improve adversarial robustness.
Our work builds heavily on Nijkamp al. 2019bba) Du & Mordatch (2019) which scales the training of EBMs to high-dimensional data using Contrastive Divergence and SGLD. While these works have pushed the boundaries of the types of data which can apply EBMs, many issues still exist. These methods require many steps of SGLD to take place at each training iteration. Each step requires approximately the same amount of computation as one iteration of standard discriminitive model training, therefore training EBMs at this scale is orders of magnitude slower than training a classifier limiting the size of problems we can attack with these methods. There exist orthogonal approaches to training EBMs which we believe have promise to scale more gracefully.
Score matching (Hyvärinen 2005) attempts to match the derivative of the model's ' density with the derivative of the data density. This approach saw some development towards high-dimensional data (Kingma & Lecun 2010) and recently has been successfully applied to large natural im- ages (Song & Ermon 2019). This approach required model to output the derivatives of the density function, not the density function itself, so is unclear what utility this model can provide to the applications we have discussed in this work. Regardless, we believe this ís promising avenue for further research. Noise Contrastive Estimation (Gutmann & Hyvärinen 2010) rephrases the density estimation problem distribution. If the classifier is properly structured, then once the classification problem is solved, an unnormalized density estimator be derived from the classifier and noise distribution. While this method has been recently extended (Ceylan & Gutmann 2018), these methods are challenging extend high-dimensional data.
classification problem, attempting to distinguish data from known noise
8 CONCLUSION AND FURTHER WORK
In this work we have presented JEM, a novel reinterpretation of standard classifier architectures which retains the strong performance of SOTA discriminative models while adding the benefits of generative modeling approaches. Our work is enabled by recent work scaling techniques for training EBMs to high dimensional data. We have demonstrated the utility incorporating this type of training into discriminative models. While there exist many issues in training EBMs we hope the results presented here will encourage the community to improve upon current approaches.
ACKNOWLEDGEMENTS
We would like to thank Ying Nian Wu and Mitch Hill for providing some EBM training tips and tricks which were crucial in getting this project off the ground. We would also like to thank Jeremy Cohen for his useful feedback which greatly strengthened our adversarial robustness results. We would like to thank Lukas Schott for feedback Francesco Croce for spotting some typos and suggesting the transfer attack. We would also like to thank Zhuowen Tu and Kwonjoon Lee for bringing related work our attention.
the robustness evaluation, Alexander Meinke and
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint 17Xiv:1707.07397, 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.004422 2018.
Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint | 1801.0197, 2018.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. arXiv preprint arXiv:1811.00995 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv: 1712,0477 2017.

---

Published as a conference paper at ICLR 2020
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. arXiv preprint arXiv: 1 88880066666 2018.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press, 2006.
Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. arXiv preprint arXiv:/906. 1906.0275,, 2019.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statisfical Society: Series B (Methodological), 39(1): 1-22, 1977.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903. 1et...99...88,,, 2019.
Ethan Fetaya, Jörn-Henrik Jacobsen, and Richard Zemel. Conditional generative models are not robust. arXiv preprint arXiv:/906.0/177. 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty deep learning. In International Conference on Machine Learning (ICML), pp- 1050-1059, 2016.
lan J Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: (4I2.66112.672. 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1321-1330. JMLR. org, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610v21610.021 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709 2005.
Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In Advances in Neural Information Processing Systems, pp. 823-833, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma and Yann Lecun. Regularized estimation of image statistics by score matching. In Advances in neural information processing systems, pp. 114,1.34, 2010.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based
learning. Predicting structured data, 1(0), 2006.
Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu. Wasserstein introspective neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3702- 3711, 2018.
10

---

Published as a conference paper at ICLR 2020
Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers robust adversarial attacks? arXiv preprint arXiv:1802.06530 2018.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of tion in neural networks. arXiv preprint ToXiv:1706.066680 2017.
image detec-
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv: 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint /:Xiv:/906602299,, 2019.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy meme- based maximum likelihood learning of energy-based models. arXiv preprint arXiv: /903. 12370, 2019a.
Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. On learning non-convergent short-run meme toward energy-basee model. arXiv preprint arXiv:1904.0977 2019b.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv preprint arXiv:1707.04131, 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234 2242, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 2017.
Hadi Salman, Greg Yang, Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint arXiv::906, 045804 2019.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Computer vision with a single (robust) classifier. CoRR, abss/190..om 906.0945,, 2019. URL
org/ abs /10//.906.... 09453
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. arXiv preprint arXiv: 1805.09909 2018.
Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou, Bernhard Schölkopf, and David Lopez- Paz. Adversarial vulnerability of neural networks increases with input dimension. arXiv preprint arXiv:1802.01422,, 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 1907::990..06600 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv: 17 1010010000000611 2017.
Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators.
arXiv preprint arXiv: 1806.0027, 2018
11

---

Published as a conference paper at ICLR 2020
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064 1071. ACM, 2008.
Kuan-Chieh Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adversar- ial distillation of bayesian neural network posteriors. In International Conference on Machine Learning (ICML), 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-II), pp. 681-688, 2011.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644 2644, 2016.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
12

---

Published as a conference paper at ICLR 2020
A TRAINING DETAILS
We train all models with the Adam optimizer (Kingma & Ba 2014) for 150 epochs through the dataset using staircase decay schedule. All network architecutres based on WideResNNet-28-10 10 with no batch normalization. We generate samples using PCD with hyperparameters in Table[4 We evolve the chains with 20-steps of SGLD per iteration and with probability .05 we reiniatilize the chains with uniform random noise. For preprocessing, we scale images to the range [-1, 1] and add
Gaussian noise of stddev .03. Pseudo-code for our training procedure is in Algorithm I
When training via contrastive divergence there are few different ways one could potentially draw samples from px(x). We could:
1. Sample ply) then sample x ~ (x y) via SGLD with energy E(xly) - fo (x) then throw away y-
2. Sample (x) via SGLD with energy E(x) fo (x)
We experimented with both methods during training and found that while method produced more visually appealing samples (from human's perspective), method 2 produced slightly stronger dis- cirminative performance - 92.9% vs. 91.2% accuracy on CIFARIO. For this reason we use method 2 in all results presented.
Algorithm JEM training: Given network Se. SGLD step-size o, SGLD noise a, replay buffer B, SGLD steps 17, reinitialization frequency p
while not converged do
2: Sample x and from dataset
3: (fr(()) (fot([f(xi]]
4: Sample 8g B with probability else Rn ~ ~U(-11,1
& Initialize SGLD D SSLL
5:
for E [1, .., 17] do
X, x a. oggSan (ol8, 1)(e') to NN(, I) end for
6:
0k:
7:
8:
Tagen (0) LogSumExp, (f(x)[[)) LogSumExp,
LogSuuffn (J (J(2.)ly)))
Surrogate for Eg
9: ((0) Lar (0) + Lgen
10: Obtain gradients at(()) for training
20
Add & B
12:
end while
Variable
Values
initial learning rate
.0001
learning epochs
150
learning rate decay
.3
learning rate decay epochs
50, 100
SGLD steps n
20
Buffer-sizz
10000
reinitialization frequency p
.05
SGLDD step-size a
SGLD noise a
.01
Table 4: Hyperparameters
B SAMPLE QUALITY EVALUTION
In this section describe the details for reproducing the Inception Score (IS) and FID results reported in the paper. First we note that both IS and FID are scores computed based on a pretrained classifier network, and thus can be very dependent more detailed discussion on the variability of IS, please refer to Barratt Sharmal (2018). To gauge model against the other papers, we document attempt to fairly compare the scores across
the exact model/code repository used. For
13

---

Published as a conference paper at ICLR 2020
papers in Table As direct comparison of IS, got 8.76 using the code provided by Du & Mordatch| (209)) and is better than their best reported score of 8.3. For FID, we used the official implementation from Heusel et al. (2017). Note that FID computed from this repository assigned much worse FID than reported in Chen et al. (019).
Conditional vs unconditional samples. Since we are interested in training Hybrid model, our model. by definition, is conditional generative model as it has access to label information. In Table unconditional samples mean samples directly obtained from running SGLD using p(x). Conditional samples are obtained by taking the max of our p(y(z) model. The reported scores are obtained by keeping the top 10 percentile samples with the highest p(ylz) values. Scores obtained on "single" model are computed directly the training replay buffer of the last checkpoint. "Ensemble" here are obtained by lumping together 5 buffers over the last few epochs of training. As initialize SGLD with uniform noise, using the training buffer is exactly the same re-sampling from the model.
Conditional single
Unconditional
Method
ensemble
single
ensemble
JEM (Ours)
8,76
7.82
7.79
EBM (D&M)
8.3
X
6.02
6.78
Table 5: Conditional vs. unconditional Inception Scores.
Inception Score
Method
from paper
B&S
D&M
from paper
H
D&M
Residual Flow
X
3.6
46.4
Glow
X
3.9
48.9*
107
JEM (Ours)
X
7.13
38.4
JEM p(xly) factored
X
6.36
X
61.8
EBM (D&M)
8.3
8.3
37.9
37.9
SNGAN NCSN
8.59
25.5
8.91
25.3
Table 6: The headings: B&S, D&M, and H denotes scores computed using code provided by Barratt & Sharmal (2018), Du & Mordatch (2019) Heusel al. (2017). ddenotes numbers copied from Chen et al. (2019), but not the original papers. As unfortunate as the case is with Inception Score and FID (i.e., taking different code repository yields vastly different results), from this table we can still see that our model performs well. Using D&M Inception Score we beat their own model, and using the official repository for FID we beat the Glow F model by a big margin.
'Code taken from httppe//gtthbe.ccoo///arr//ttppp
14

---

Published as a conference paper at ICLR 2020
C FURTHER HYBRID MODEL SAMPLES
Additional samples from CIFAR10 and SVHN can be seen Figure and samples from CIFAR100 can be seen in Figure g
600
18
21
2
2
22
20
3
3
3
3
30
3
11
4
5
55
5
55
15
95
6
16
6
6
6
$6
7
28
18
28
8
50
10
00
9
41
Figure 7: Class-conditional Samples. Left to right: CIFAR1O, SVHN.
Figure 8: CIFAR100 Class-conditional Samples.
15

---

Published as a conference paper at ICLR 2020
D QUALITATIVE ANALYSIS OF SAMPLES
Visual quality is difficult to quantify. Of the known metrics like and FID, using samples that have higher p(y)x) values results in higher scores, but necessary if use samples with higher logp(x p()). However, this is likely because of the downfalls of the evaluation metrics themselves rather than reflecting true sample quality.
Based on our analysis (below), we find
Our log p(x) model assigns values that cluster around different means for different classes. The class automobiles has the highest log p(x). Of all generated samples, all top 100 samples are of this class.
2. Given the class, the samples that have higher log p(x) values all have white background and centered object, and lower log p(x) samples have colorful (e.g., forest-like) background.
3. Of all samples, higher p(y/xx values means clearly centered objects, and lower p(y x) otherwise.
Figure 9: Each row corresponds to I class, subfigures corresponds to different values log (xx). left: highest, mid: random, right: lowest.
Casa at clane
OaseTrug
Class "bedd
Casy
Clea deer
dog'
Oan wetombild
hore'
Cas: cat
Figure 10: Histograms (oriented horizontally for easier visual alignment) of log arranged by class.
16

---

Published as a conference paper at ICLR 2020
Figure 11: left: samples with highest log p(x), right: left: samples with lowest log p(x)
Figure 12: left: samples with highest p(y|z), right: left: samples with lowest p(y|æ:)
17

---

Published as a conference paper at ICLR 2020
E CALIBRATION
E. EXPECTED CALIBRATION ERROR
Expected Calibration Error (ECE) is metric measure the calibration of classifier. It works by first computing the confidence, maxy p(y|x;), for each x, in some dataset. We then group the items into equally spaced buckets {Bm J MM based on the classifier's output confidence. For example, M 20, then Bo would represent all examples for which the classifier's confidence between 0.0 and 0.05.
We then define:
M
Bm
ECE
acc acc( Bm) -oonfco conf(Bm))
(10)
where n is the number of examples in the dataset, acc Bm) is the averaged accuracy of the classifier of examples in Bn and conf(B.,, the averaged confidence over all examples Bm
For perfectly calibrated classifier, this value will be O for any choice of M. In our analysis, we choose M 20 throughout.
E.2 FURTHER RESULTS
We find that JEM also improves calibration on CIFARIO can be seen in Table 13 There we improvement calibration, but both classifiers are well calibrated because their accuracy is so high. In interesting experiment, this setting the accuracy drops to 78.0% and 74.9% in the baseline and JEM, respectively. Given the JEM be trained unlabeled data, treat the remainder of the training set unlabeled and train in a semi-supervised manner. We find this gives noticeable boost in the classifier's calibration seen in Figure 13 Surprisingly this did not improve generalization. We leave exploring this phenomenon for future work.
limit the size of the training to 4,000 labeled examples. In
DCB. 2.70%
PCF: 2.23%
SCUnC3
0.6
04
0.0
0.7
0-4
0.6
2.8
3.0
0.0
0.7
0.4
3.6
3.8
3.0
crffieece
confidence
(a) CIFAR10 Baseline
(b) CIFARIO JEM
ECE: 10.16%
RCP: 1.85
KCUNCY
87
0.0
0.4
0.0
13.5
3.0
0.8
12.4
116
1..5
2.0
confdence
confidence
(c) CIFAR100 Baseline (4k labels)
(d) CIFAR100 JEM (4k labels)
Figure 13: CIFAR10 Calibration results
18

---

Published as a conference paper at ICLR 2020
F OUF-OF-DISTRIBUTION DETECTION
F.1 EXPERIMENTAL DETAILS
To obtain OOD results for unconditional Glow, we used the pre-trained model and implementation wwonn//oasst/loo-yTTorr
of https ://gi thub wwt.commmmaoaa as well using this codebase which was used generate the class-conditional OOD results.
We trained Class-Conditionas model
We obtained the IGEBM of Du & Mordatch (2019) from their open-source implementation at
com wwwpooi/eeeeeeeee //emmmm//mmmmm.mmmm wwwecooeeeeeo re reverreeeeeee ease, For likelihood and likelihood- gradient OOD scores used their pre-trained ci farll large model uncond model. We were able to replicate the likelihood based OOD results presented their work. We implemented likelihood-gradient approximate- mass score on top of their codebase, For predictive distribution based OOD scores we used their ci fandcond model which was the model used in their work to generate their robustness results.
F.2 FURTHER RESULTS
Figure contains results on two datasets, Constant and Uniform, which were omitted for space. Most models perform very well at the Uniform dataset. On the Constant dataset (all examples 0) generative models mainly fail with JEM being the only one whose likelihoods can used to derive a predictive score function for OOD detection. Intrestinly, we could not obtain approximate mass scores on this dataset from the Glow models due to numerical stability issues.
CIFARIO
Score
Model
SVHN
Uniform Constant
Interp
CIFAR100 CelebA
Unconditional Glow
.05
1.0
0.0
.51
.55
.57
logp(x) p(x)
Glow Supervised
.07
1.0
0.0
.45
.51
.53
IGEBM
.63
1.0
.30
.70
.50
.70
JEM (Ours)
.67
1.0
.51
.65
.67
.75
WRN-baseline
.93
.97
.99
.77
.85
.62
maxy p(y[x)
Class-Conditional Glow
.64
0.0
.82
.61
.65
.54
IGEBM
.43
.05
.60
.69
.54
.69
JEM (Ours)
.89
.41
.84
.75
.87
.79
Unconditional Glow
.95
.99
NaN
.27
.46
.29
a p(x)
Class-Conditions Glow
.47
.99
NaN
.01
.52
.59
8x
IGEBM
.84
.99
0.0
.65
.55
.66
JEM (Ours)
.83
1.0
.75
.78
.82
.79
Table 7: OOD Detection Results. Values are AUROC.
G ATTACK DETAILS AND FURTHER ROBUSTNESS RESULTS
We use foolbox (Rauber et al. 2017) for our experiments. PGD uses binary search to determine minimal epsilons for every input and plot the resulting robustness-distortonn curves. PGD runs with 20 random restarts and 40 iterations. For the boundary attack, we run default foolbox settings with one important difference. The random initialization often fails for JEM and thus we initialize the attack with a correclty classified input of another class. This other class is chosen based on the top-2 prediction for the image to attacked. As all our attacks are expensive to run, we only attacked 300 randomly chosen inputs. The same randomly chosen inputs were used to attack each model.
we see the results of the boundary attack and pointwise attack on JEM and baseline. The main point to running these attacks was to demonstrate that our model was not able to "cheat" by having vanishing gradients through successful than these gradient- free methods, this is clearly not the case and the attacker was able use the gradients of the sampling procedure attack model. Further, observe the same
In Figure 14
gradient-basel based sampling procedure. Since PGD more
19

---

Published as a conference paper at ICLR 2020
behavior across all attacks; the EBM with 0 steps sampling is more robust than the baseline and the robustness increases as we add more steps of sampling.
We also compare JEM to the IGEBM of Du & Mordatch (2019) with 10 steps of sampling refine- ment, see Figure D3 We run the same gradient- attacks on their model and find that despite not having competitive clean accuracy, is quite robust to large attacks especially with respect to the so norm. After c = 12 their model is more robust than ours and after c = 18 it is more robust than the adversarial training baseline. With respect the L2 norm their model more robust than the adversarial training baseline above 280 but remains less robust than JEM until 525.
We believe these results demonstrate that EBMs compelling class of models explore for further work on building robust models.
Accuracy
Accuracy
6.-
Baseline
JEM-0
JEM-1
40-
60-
Baseline
JEM-10
JEM-O
20-
20
JEM-1
JEM-10
(a) Boundary Loo
(b) Boundary L2
93.0
92.5
Baseline
Accuracy
92.0
Accuracy
JEM-0
JEM-1
50
91.0
JEM-10
Baseline
JEM-O
90.5
JEM-1
20
JEM-10
90.0
100
200
300
500
(c) Pointwise Loo
(d) Pointwise L-2
Figure 14: Gradient-free adversarial attacks.
G.1 EXPECTATION OVER TRANSFORMATIONS
Our SGLD-based refinement procedure is stochastic in nature and it has been shown that stochastic defenses to adversarial attacks can provide false sense of security (Athalye et al. 2018). To deal with this, when we attack our stochastically refined classifiers, we average the classifier's ' predictions over multiple samples of this refinement procedure. This makes the defense more deterministic and easier to attack. We redefine the logits of our classifier as:
log (x(x))
log p(y(xx))))
SGLD(L, k))
(11)
where we have defined SGLD(x, k) as an SGLD chain run for & steps seeded at x. Intuitively, we draw different samples {x,) from model seeded input x, then compute log (())x,) for each of these samples, then average the results. We then attack these averaged logits with PGD to generate the results in Figure 5 We experimented with different numbers of samples and found that
20

---

Published as a conference paper at ICLR 2020
Baseline
JEM-10
IGEBM-10
Accuracy
60-
Adv Training
Accuracy
60-
RandAdvSmooth
60
20
25
30
0
100
200
300
400
500
600
(a) PGD Loo
(b) PGD L2
Figure 15: PGD attacks comparing JEM to the IG EBM of Du & Mordatch 2019)
10 samples yields very similar results to 5 samples on JEM with one refinement step (see Figure 16] Because 10 samples took very long to run on the JEM model with ten refinement steps, we settled on using 5 samples in the results reported in the main body the paper.
Baseline
JEM-0
JEM-1-5
Accuracy
60
JEM-1-10
Accuracy
60
Adv Training
40
RandAdvSmooth
80
300
400
(a) PGD Too
(b) PGD L2
Figure 16: Comparing the effect of the number of samples in the EOT attack. We find negligible
difference between and 10 for JEM-1 (red and green curves).
G.2 TRANSFER ATTACKS
would like to see JEM's refinement procedure can correct adversarial perturbed inputs - inputs which cause the model to fail. To do this, generate series of adversarial examples for JEM-0, with respect to the Lox norm, and test the accuracy JEM-(1 (MMM[1,10] on these examples. Ideally, with further refinement the accuracy will increase. The results of this experiment can be seen in Figure We see here that JEM's refinement procedure can correct for adversarial perturbations.
17
A DISCUSSION ON SAMPLERS
H.
IMPROPER SGLD
Recall the transition kereel of SGLD:
Xg px())
(x.i)x)))
X 2 a0
+e,
e~ a)),aa))
21

---

Published as a conference paper at ICLR 2020
80
60-
JEM-O
JEM-1
JEM-10
5
10
15
20
25
30
(a) PGD Loc
Figure 17: PGD transfer attack Loo. We attack JEM-O and evaluate success of the same adversarial examples under JEM-1 and JEM-10. Whenever an adversarial example is refined back its correct class, we set the distance to infinity. Note that the adversarial examples do not transfer well from JEM-0 JEM-1/-10.
In the proper formulation of this sampler (Welling Teh 2011), the step-size and the variance of the Gaussian noise are related Var(<) = a. If the stepsize is decayed with a polynomial schedule, then samples from SGLD converge samples from our unnomralized density as the number of steps goes to
In practice, we approximate these samples with sampler that runs for a finite number of steps. When using the proper step-size to noise ratio, the signal from the gradient is overtaken by the noise when step-sizes are large enough to be informative. In practice the sampler is typically "relaxed" in that different values used for the step-size and the amount of Guassian noise added typically the amount of noise is significantly reduced.
longer working with valid MCMC sampler, this approximation has been success- fully applied in practice in most recent work scaling EBM training to high dimensional data (Ni- jkamp al., 2019b Du & Mordatch, 2019) with the exception of Song & Ermon (2019) (which develops a clever work-around). The model they train is actually an ensemble of models trained on data with different amounts of noise added. They as they sample, moving from their high-noise models to their low-noise models. This provides one possible explanation for the compelling results of their model.
While
proper SGLD sampler decaying the step size
In our work we have set the step-size a = 2 and draw ~ (O00.003)) We have found these parameters to work well across a variety of datasets, domains, architectures, and sampling proce- dures (persistent vs. short-run). We believe they are a decent "starting place" for energy-functions parameterized by deep neural networks.
H.2 PERSISTENT SHORT-RUN CHAINS?
Both persistent and short-run markov chains have been able to succesfully train EBMs. Nijkamp al. (209a) presents careful study various samplers which can be used and the tradeoff's one makes when choosing one sampler over another. In our work we have found that if computation allows, short-run MCMC chains are preferable terms of training stability. Given that each step of SGLD requires approximately the computation of training iteration of standard classifier incentivized to find sampler which can stably train EBMs requiring as few steps as possible per training iteration.
In our experiments we found the smallest number of SGLD steps we could take to stably train an EBM the scale of this work was 80 steps. Even so, these models eventually would diverge late into training. At 80 steps, we found the cost of training to be prohibitively high compared to a standard classifier.
22

---

Published as a conference paper at ICLR 2020
We found that by using persistent markov chains, could further reduce the number of steps per iteration to 20 and still allow for relatively stable training. This gave a 4x speedup over our fastest short-run MCMC sampler. Still, this PCD sampler was noticebly less stable than the fastest short- run sampler we could use but we found the multiple factor increase in speed to be a worth-while trade-off.
If time allows, we recommend using a short-run MCMC sampler with a large enough number of steps to be stable. Given that is not always possible on problems of scale, PCD can be made to work more efficiently, but at the cost of a greater number of stability-related These additional parameters include the buffer size and the re-initialization frequency of the Markov chains. We found both to be important for training stability and found no general recipe for which to set them. We ran most of our experiments with re-initialization frequency 5%.
A particuairly interesting observation we discovered while using PCD is that the model would use the length of the markov chains to encode semantic information. We found that when training models CIFARIO 10, when chains were young they almost always could be identified frogs. When chains were old they could almost always identified as cars. This behavior likely some degeneracy of PCD which would not be possible with a short-run MCMC since all chains have the same length.
H.3 DEALING WITH INSTABILITY
Training model with the gradient estimator of Eq. 2 can be quite unstable especially when combined with other objective as was the case with all models presented in this work. There exists a "stable region" of sorts when training these models where the energy values of the true data are in the same range as the energy values of the generated samples. Intuitively, if the generated samples create energies that are not trivially separated from the training data, then real learning has to take place. Nijkamp al. (2019b provide a careful analysis of this and we refer the reader there for more in-depth analysis.
We find that when using PCD occasionally throughout training sample will be drawn from the replay buffer that has a considerably higher-than average energy (higher than the energy a random initialization). This causes the gradients w.r.t this example to be orders of magnitude larger than gradients w.r.t the rest of the examples and causes the model to diverge. We tried a number of heuristic approaches such gradient clipping, energy clipping, ignoring examples with atypical energy values, and many others but could not find an approach that stabilized training and did not hurt generative and discriminative performance.
The only two approaches we found to consistently work to increase stability of a model which has diverged is 1) decrease the learning rate and 2) increase the number of SGLD steps in each PCD iteration. Unfortunately, both of these approaches slow down learning. We also had some success simply restarting models from a saved checkpoint with a different random seed. This was the main approach taken unless the model was late into training. In this case, random restarts were less effective and we increased the number of SGLD steps from 20 to 40 which stabilized training.
While we are very optimistic about the future of large-scale scale EBMs we believe these are the most important issues that must be addressed order for these models be succeful.
23


Reviews:
[{"review_id": "Skglf_h_9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the idea of energy based model to the traditional classifier, and proposes a new framework to improve the performances of the model in multiple aspects. The idea of reinterpreting the traditional classifier is very interesting, and the experiments show some good results of the proposed method.  \n\nHere are my main concerns of the current paper:\n1. The training procedure seems to be very sensitive, and the SGLD may take a long time at each iteration to converge. This may be a big limitation of the proposed method.\n2. According to equation (8), the proposed method is having a trade-off between classification and generation, and this seems to be the key to improve the performance of the model in generation by sacrificing some classification accuracy. I think author should emphasize this instead of energy based model.\n3. The presentation is not very clear in section 5. What is the task of calibration, and what is the definition of ECE?\n4. The robustness guarantee seems too good to be true. Although the authors claim that they allow the attacker to have access to the gradient  of SGLD, the SGLD will add noise during the forward process, this will obfuscate the gradient. In this sense, I don’t think the proposed method will have the strong robustness as they claimed.\n\n----------------\nPost-Rebuttal Comments:\nThanks for addressing my concerns. Although I think the proposed method is not comprehensive to check obfuscated gradients, I do think the current version is a good fit for ICLR, and I decide to increase my score. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hklnbfn6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper uses energy-based model interpretation for the logits of standard discriminative neural network models to define a generative model inside a classifier that proves useful in many downstream tasks such as uncertainty quantification, out-of-distribution detection, etc.\nAlthough there has been previous work attempting to bridge discriminative classifiers with generative modeling, this work proves to be competitive with both specialized models on discriminative/generative tasks as well as in many downstream tasks such as out-of-distribution detection, calibration, and adversarial robustness. The paper provides a clear exposition of the method, succeeds to discuss related work it bases on, conducts a thorough experimental study providing convincing explanations for results and does not hide the limitations of the work (high computational requirements, optimization difficulties connected with training energy-based model and the method used, limited approximation of the true energy). Overall, the paper provides a substantial contribution and paves the way for further work improving this joint discriminative - generative setting. However, there are points I would like the paper to address for better exposition.\n1. It would benefit the paper showing that samples with higher unnormalized likelihood are visually more compelling than those with lower likelihood.\n2. On CIFAR100 the accuracy drop from the reference value is larger than for datasets with 10 classes, could it be due the logits dimension is higher and challenges optimization?\n3. It would also be helpful to clarify whether application of the proposed method is primarily restricted by the computational complexity or is there any property inherent to energy-based models that makes treating high-dimensional data challenging?\n\nMinor remark\n- Although the paper doesn't state on which dataset results shown in Table 1 were obtained, I suspect its CIFAR10, please specify this.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlmF6qoOr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is an attempt to bridge the gap between discriminative models, which currently obtain the state of the art on most classification problems, and generative models, which (through a model of the marginal p(x)) have the potential to shine on many tasks beyond generalization to a hold-out set with minimal shift in distributions: out of distribution detection, better generalization out of distribution, unsupervised learning etc.\n\nWhile much of the current work is related to normalizing flows / invertible neural networks, the authors here propose a quite simple but appealing method: A standard neural classifier is taken and the softmax is layer chopped off and replaced by an energy based model, which models the joint probability p(x,y) instead of the posterior p(y|x). The advantage is an additional degree of freedom in the scale of the logit vector, which is would have been otherwise normalized by the softmax layer and now can now model the data distribution. The downside is the loss in ease of training. Whereas (discriminative) deep networks can be easily trained by gradient descent on a cross-entropy objective, the partition function in the energy model makes this un tractable. This is addressed through sampling, similar to (Welling & Teh, 2011).\n\nOne of the biggest achievements reported by the authors is that the performance on discriminative tasks is not hurt (much) by adding the generative model. There is only a 3 point gap between Wide-ResNet and the proposed model (92.9% vs. 95.8%) … but on what dataset? 3 datasets are mentioned in the experimental section, but table 1 does not mention on which datasets the accuracy is reported. My guess is that this is a mean or mixture, since GEM performances of 96.7% and 72.2% are reported for SVHN and CIFAR10, respectively, but this should be made clearer. \t\n\nOn out of distribution detection, could the authors comment on the histograms in table 2, in particular the difference between the new measure (AM JEM) compared to JEM log p(x) on CelebA? The proposed measure does not seem to fare well here.\n\nAlthough the method does not outperform the gold standard of adversarial training, I found the models robustness to adversarial examples quite appealing, given that it was not trained for this objective (which also means that it does not require an adaptation to a norm). \n\nI was very impressed by Figure 6 showing distal adversarial initialized from random images, showing pretty clear images of the modelled class. The modelled variations require more investigation to verify whether we have a collapse for each class, but the results look very promising.\n\nThe paper is well written and easy to understand. A couple of details on the training procedure are missing in the experimental part. It is stated that, both, p(y|x) and the generative part p(x), are optimized, but how are these exactly integrated? Given the difficult in training this model reported in the paper, this seems to be particularly important.\n\nI also appreciated the description of the limitations of the algorithm, and the details in the appendix (ICLR should go back to unlimited paper lengths, btw.).\n\nMore information on complexity (training times etc.) should also be helpful. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses energy based model to interpret standard discriminative classifier and demonstrates that energy based model training of the joint distribution improves calibration, robustness, and out-of-distribution detection while generating samples with better quality than GAN-based approaches. The reviewers are very excited about this work, and the energy-based perspective of generative and discriminative learning. There is a unanimous agreement to strongly accept this paper after author response.
Decision: Accept (Talk)