Hybrid Discriminative-Generative Training via Contrastive Learning
Authors: Hao Liu
Pieter Abbeel
Venue:  (2021)
Keywords: Hybrid Models
Contrastive Learning
Energy-Based Models
Discriminative-Generative Models

Abstract: Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss significantly improves energy-based models and contrastive learning based methods in confidence-calibration, out-of-distribution detection, adversarial robustness, generative modeling, and image classification tasks. In addition to significantly improved performance, our method also gets rid of SGLD training and does not suffer from training instability. Our evaluations also demonstrate that our method performs better than or on par with state-of-the-art hand-tailored methods in each task. 

Under review as a conference paper at ICLR 2021
HYBRID DISCRIMINAAIVV-GEENEAAVVV TRAINING VIA CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive learning and supervised learning have both significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discrininative-geneneie energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-basee -bsed loss significantly improves energy- based models and contrastive learning based methods in confidence-calibration duttr--distribution detection, adversarial robustness, generative modeling, and image classification tasks. In addition to significantly improved performance, our method also gets rid of SGLD training and does not suffer from training instability. Our evaluations also demonstrate that method performs better than on par with state-of-ther -het--rr- hand-tailored methods in each task.
training of
INTRODUCTION
In the past few years, the field of deep learning has seen significant progress. Example successes include large-scale image classification (He al. 2016; Simonyan Zisserman 2014 Srivastava et al., 2015: Szegedy et al.] (016) on the challenging ImageNet benchmark (Deng et al. 2009). The common objective for solving supervised machine learning problems is to minimize the cross- entropy loss, which is defined as the cross entropy between target distribution and categorical distribution called Softmax which parameterized by the model's real-valued outputs known as logits. The target distribution usually consists of one-hot labels. There has been continuing effort on improving upon the cross-entropy loss, various methods have been proposed, motivated by different considerations (Hinton et al. 2015 Müller al. 2019 Szegedy et al. 2016).
Recently, contrastive learning has achieved remarkable success in representation learning. Contrastive learning allows learning good representations and enables efficient training on downstream tasks, an incomplete list includes image classification (Chen et al.] 22020a b Grill et al. 2020 He et al.) 2019 Tian al. 2019 Oord al., 2018), video understanding (Han al. 2019). and knowledge distillation (Tian et al. 2019). Many different training approaches have been proposed to learn such representations, usually relying on visual pretext tasks. Among them, state-of-the-ard contrastive methods (He et al., 2019 Chen et al. 2020a c) are trained by reducing the distance between represen- tations of different augmented views of the same image ('positive pairs') ), and increasing the distance between representations of augment views from different images ('negative pairs').
Despite the success of the two objectives, they have been treated as two separate objectives, brought together only by having shared neural network.
In this paper, to show direct connection between contrastive learning and supervised learning, we consider the energy-based interpretation of models trained with cross-entropy loss, building on Grathwohl et al. (201)). We propose novel objective that consists a term for the conditional of the label given the input (the classifier) and term for the conditional of the input given the label. We optimize the classifier term the normal way. Different from Grathwohl et al. (019). we approximately optimize the second conditional over the input with contrastive learning objective instead of a Monte-Carlo sampling-bas based approximation. In doing so, we provide a unified view on existing practice.

---

Under review as a conference paper at ICLR 2021
Our work takes inspiration from the work by Ng & Jordan 2002). In their 2002 paper, Ng & Jordan (2002) showed that classifiers trained with a generative loss (i.c., optimizing p(aly), with x the imput and y the classification label) can outperform classifiers with the same expressiveness trained with discriminative loss (i.e., optimizing p(y(ar). Later it was shown that hybrid discriminative generative model training can get the best of both worlds (Raina et al. 2004). The work by Ngg & Jordan (2002) was done in the (simpler) context of Naive Bayes and Logistic Regression. Our work can be seen as lifting this work into today's context of training deep neural net classifiers.
Our empirical evaluation shows our method improves both the confidence-callibatiion and the classi- fication accuracy of the learned classifiers, beating state-of- of-tthe-rrt methods. Despite its simplicity, our method outperforms competitive baselines in out-of out-of-distrine distribbutiion (OOD) detection for all tested datasets. hybrid mnnerative-diserimnnee obtains superior performance without needing to run computational expensive SGLDD steps. Our method learns significantly more robust classifiers than supervised training and achieves highly competitive results with hand-tailored adversarial robustness algorithms.
modeling tasks (Grathwohl al. 2019), our method
The contributions of this paper can be summarized as: (i) To the best of our knowledge, we are the first to reveal the connection between contrastive learning and supervised learning. We connect the two objectives through energy-basee model. (ii) Built upon the insight, we present a novel framework for hybrid generative discriminative modeling via contrastive learning. (iii) Our method gets rid of SGLD therefore does not suffer from training instability of energy-based model. We empirically show that our method improves confidence-calibration robustness, generative modeling, and classification accuracy, performing on par with or better than fate-of-theeart energy-based models and contrastive learning algorithms for each task.
OOD detection, adversarial
2
RELATED WORK
Our work falls into the category of hybrid generative discriminative models. Ng & Jordan (2002); Raina et al.] (2004); Lasserre et af.] (2006); Larochelle & Bengio (2008); Tu y Lazarow et al. (2017) compare and study the connections and differences between discriminative model and genera- tive model, and shows hybrid generative discriminative models can outperform purely discriminative models and purely generative models. Our work differs in that we propose an effective training approach in the context of deep neural network. By using contrastive learning to optimize the generative models, our method achieves of-te-of performance on wide range of tasks.
Energy-based models (EBMs) have been shown can be derived from classifiers in supervised learning in the work of Xie et al. ()016) Du Mordatch (2019), they reinterpret the logits define a class- conditional EBM p(zlg) Our work builds heavily on JEM (Grathwohl et al. 2019) which reveals that one can re-interpret the logits obtained from classifiers to define EBM p(x) and p(z, v). and shows this leads to significant improvement in OOD detection, calibration, and robustness while retain compelling classification accuracy. Our method differs in that we optimize our generative term via contrastive learning, buying the performance of state-of-the-as canonical EBMs algorithms (Grath wohl et al. 2019) without suffering from running computational expensive and slow SGLD (Welling & Teh 2011 every iteration.
Concurrent to our work, Winkens et al. 2020) proposes to pretrain using contrastive loss and then finetune with a joint supervised and contrastive loss, and shows the SimCLR loss improves likelibood-based OOD detection. TTck et al. (2020) also demonstrate contrastive learning improves OOD detection and calibration. Our work differs in that instead of contrastive representation pre-train followed by supervised loss fine-tune, we use the contrastive loss to approximate hybrid -eaiinnntiii-eeeaiiveeeeee
discriminative generative model. We also empirically demonstrate our method enjoys broader usage by applying it to generative modeling, calibration, and adversarial robustness.
BACKGROUND
3.1
SUPERVISED LEARNING
In supervised learning, given a data distribution p(z) and a label distribution p(y|z) with C' categories, classification problem is typically addressed using a parametric function, TDD CRC which
2

---

Under review as a conference paper at ICLR 2021
maps each data point RR' C real- valued numbers termed as logits. These logits are used to parameterize a categorical distribution using the Softmax function:
exp(fo(x))u))
qo (y/a = Evv exp exp(fo(x)(v))
(1)
where [b] indicates the y" element of fo(x), i.e., the logit corresponding to the ydh class label. One of the most widely used loss functions for learning is minimizing the negative log likelihood:
min (p..aa(F.)) [log qo (ylz))]
(2)
This loss function is often referred to as the cross-entropy loss function, because corresponds to minimizing the KL-divergence with a target distribution p(y ar), which consists of one-hot vectors with the zer-zero element denoting the correct prediction.
3.2 ENERGY-BASED MODELS
Energy- based models. Energy based models (EBMs) (LeCun al. 2006) are based on the obser- vation that probability densities p(z) for x E R can be expressed as
(= exp( (ep(-
Z(0)
where Ex(x): DRD + R maps each data point to a scalar; and ((0) = =x exp(-EEo (or, for continuous x we'd have Z(0) = Jee, exp(-Ea (z))) is the normalizing constant, also known as the partition function. Here X' is the full domain of For example, in the case of (let's say) 16x16 RGB images, computing Z exactly would require summation over (256 256 x 256)(16x16) ~ 102500 terms.
We can parameterize an EBM using any function that takes the input and returns scalar. For most choices of Eg, one cannot compute or even reliably estimate (0),, which means estimating the normalized densities is intractable and standard maximum likelihood estimation of the parameters, 0, is not straightforward.
Training EBMs.
The liglliilldood objective for an EBM consists of sum of log (1) terms,
one term for each data point æ. The gradient of each term is given by:
i (x(r
(Ea(a')
eE((r)
(x')
(4)
00
a0
00
where the expectation is over the model distribution po(x'). This expectation is typically intractable (for much the same reasons computing Z(0) is typically intractable). However, it can be approximated through samples-assuming we can sample from ps. Generating exact samples from pe is typically expensive, but there are some well established approximate (sometimes exact in the limit) methods
based on MCMC (Grathwohl al.) 2019 Du & Mordatch, 2019: Hinton 2002).
Among such sampling methods, recent success in training (and sampling from) energy-based models often relies on the Stochastic Gradient Langevin Dynamics (SGLD) approach (Welling Teh 201T. which generates samples by following this stochastic process:
a (.
FD (x)),
R++1 Z;
+ +
<~~1(000
(5)
2 Ox:
where \O a)) is the normal distribution with mean of 0 and standard deviation of a, and po (2) typically Uniform distribution over the input domain and the step-size should be decayed following polynomial schedule. The SGLD sampling steps are tractable, assuming the gradient of the energy function can be computed with respect to z, which is often the case. It is worth noting this process does not require evaluation the partition function Z() (or any derivatives thereof).
Joint Energy Models. The joint energy based model (JEM) (rrthhwohl et al. 2019) shows that classifiers in supervised learning are secretly also energy-based based models p(x, y). The key
3

---

Under review as a conference paper at ICLR 2021
insight is that the logits (z)))yyl in the supervised cross-entropy loss be seen as defining energy-based model over (z; V). as follows:
p(,,y) exp(Jo(z)vv) Z(0)
(6)
where Z(0) is the unknown normalization constant. Le.. matching this with the typical EBM notation, we have [x) - (E 3). Subsequently. the density model of data points p(æ) can be obtained by marginalizing over y/:
p(x) Ey exp(fo(=)wv)
2(0)
Grathwohl al. 2019) adds the marginal log-likelihood p(ar) the training objective, where p(ar) expressed with the energy based model from Equation JEM uses SGLD sampling for training.
with the energy Ex(x) - E, exp(fo(x))1) JEM
3.3 CONTRASTIVE LEARNING
In contrastive learning (Hadsell et al.
2006 Gutmann & Hyvärinen 2010 2012 Mnih
Kavukcuoglu 2013 Mikolov et al. 2013), is common to optimize an objective of the following
form:
min E (x) log exp(ho(x) hh(x'))
Σ i==l exp(ho(z) [
where x and x' are two different augmented views of the same data point, K is the number of negative examples, hg RD + R maps each data point to normalized representation space with dimension H. This objective tries to maximally distinguish an input v; from alternative inputs ay. The intuition is that by doing so, the representation captures important information between similar data points, and therefore might improve performance on downstream tasks. This is usually called the contrastive learning loss or InfoNCE loss (Oord et al. 2018 and has been successful used for learning unsupervised representations (Sohn) 2016 Wu et al.) 2018 He et al.) 2019 Chen et al.) 2020a). the context of supervised learning, the Supervised Contrastive Loss (Khosla et al.) 20200 shows that selecting from different categories as negative examples can improve the standard cross-entropy training. Their objective for learning the representation ho(x) is given by:
min Σ
1
Σ
114ي log 1=1
cxp(ho (2;) ho())))
2N
2N
lizk exp(ho (2:4) ho(xx))
where Ng, is the total number of images in the minibatch that have the same label ÿJ the anchor i. We'll see that our approach outperforms Supervised Contrastive Learning, while also simplifying by removing the need for selecting negative examples or pre-training a representation. Through the simplification we might get closer hint at where the leverage is coming from.
4 HYBRID DISCRIMINATIVE GENERATIVE ENERGY-BASER MODEL (HDGE)
As in the typical classification setting, we assume we are given a dataset (I, y) ~ Pdata- The primary goal is train model that classify (z to y). In addition, would like the learned model to be capable of out-of-distribution detection, providing calibrated outputs, and serving as generative model.
To achieve these goals, we propose to train a hybrid model, which consists of a discriminative conditional and generative conditional by maximizing the sum of both conditional log-likelihoods:
min E (x.) [log qo (y/ar) + log ge (z/y))
(10)
where ge(y(z) .z) is standard Softmax neural net classifier, and where go(zly) = (x)(f((x)))) 7(0) = Σ. (x)(((())))))
with
Z(()
4

---

Under review as a conference paper at ICLR 2021
The rationale for this objective originates from (Ng & Jordan 2002: Raina al., 2004), where they discuss the connections between logistic regression and naive Bayes, and show that hybrid discriminative and generative models can out-perform purely generative or purely discriminative counterparts. The main challenge with the objective from Equation (T)) is the intractable partition function (0).
Our main contribution is to propose (crude, yet experimentally effective) approximation with a contrastive loss:
E (a
[log (o(x)))
(11)
log exp(fe(x)
E Eaasaa (x,y)
(12)
2
log exp(ff(>)))))
2 E Bp (z.a)
(13)
L==1 exp (x:(()(
where K denotes the number of normalization samples. This similar to existing contrastive learning objectives, although in formulation, we also use labels.
Intuitively, in order to have an accurate approximation in Equation [3 K has to be sufficiently becomnbg exact in the limit of summing over all X. We don't 't know of any formal guarantees for our proposed approximation, and ultimately the justification has to come from our experiments. Nevertheless, there two main intuitions we considered: (i) We try make K as large as is practical. Increasing K is not trivial as it requires a larger memory. To still push the limits, following He et al. (2019) we use a memory bank to store negative examples. More specifically, we resort to using a queue to store past logits, and sample normalization examples from this queue X, we could expect to achieve a good approximation by focusing on (zz y) that have low energy. Since the training examples v; are encouraged to have low energy, we draw from those for our approximation. It is worth noting that the training examples Z;. 3/ are getting incorporated in the denominator using the same label y as in the numerator. So effectively this objective is (largely) contrasting the logit value fo (z) [v] for with label from the logit values of other training examples x; that don't 't have the same label y.
during the training process. (ii) While in principle we would need to sum over all possible
To bring it all together, objective can be seen hybrid combination of supervised learning and contrastive learning given by:
min EE [a log go (v)z) + (1 (1-
a) log (x(l))
(14)
Rs min - -E (a. a log Ey exp(fo(x)(v') + (1 a1) a- log exp(fo(x)())
exp(fa(x)lv)
(15)
0
E (xp(ff((x))))
where a is weight between [0, 1]. When a = 1, the objective reduces to the standard cross-entropy loss, while = 0, it reduces to an end-to-end supervised version of contrastive learning. We evaluated these variants in experiments, and we found that a = 0.5 delivers the highest performance on classification accuracy well robustness, calibration, and out-of-distribution detection.
The resulting model, dubbed Hybrid Discriminative Generative Energy-based Model (HDGE), learns to jointly optimize supervised learning and contrastive learning. A PyTorrc (Paszke al. -ike pseudo code corresponding to this algorithm is included in Appendix Algorithm
EXPERIMENT
5.1
OUT-OF-DISTRIBUTION DETECTION
We conduct experiments to evaluate HDGE on out-of-distribution (OOD) detection tasks. In general, OOD detection a binary classification problem, where the model required to produce a score (x E I. where æ is the query, and @ is the model parameters. We desire that the scores for in-distribution examples are higher than that out-oof-dstrribun of Grathwohl al. (2019). we use the area under the receiver-operating curve (AUROC) (Hendrycks) & Gimpel, 2016) as the evaluation metric. In our evaluation, we will consider two different score functions, the input density q(ar) (Section 5.1.1b and the predictive distribution g(yl=:) (Section 5.1.2).
examples. Following the setting
5

---

Under review as a conference paper at ICLR 2021
5.1.1
INPUT DENSITY ((x)
Out-of-distribttion CIFARI00
88
Model
SVHN
Interp
CelebA
WideResNeet28-11
.46
.41
.47
.49
Unconditional Glow
.05
.51
.55
.57
ClassCoonittionll Glow
.07
.45
.51
.53
IGEBM
.63
.70
.50
.70
log ((x)
JEM
.67
.65
.67
.75
HDGE (ours)
.96
.82
.91
.80
.955
.299
HDGE (ResNet-50)
.997
.988
WideResNett28--10
.93
.77
.85
.62
Contrastive pretraining
.87
.80
.58
max, p(ylz)
Class-Conditional Glow
.64
.61
.65
.54
IGEBM JEM HDGE (ours)
.43
.69
.54
.69
.89
.75
.87
.79
.95
.76
.84
.81
Table 1: OOD Detection Results.
The model is WideResNet-28-1 (without BN) following the settings of JEM (Grathwohl 2019)) The comparison with JointLoss (Winkens et al. 2020) follows their setting
ResNet-50. The results JointLoss are obtained from its paper. The training dataset CIFAR-10. Values are AUROC. Standard deviations given in Table (Appendix))
Prior work show that fitting a density model on the data and consider examples with low likelihood be reliably used as a predictor for OOD inputs (Du & Mordatch 2019 Grathwohl et al.) 2019). We are interested in whether HDGE results in better likelihood function for OOD detection. All the methods are based the WideRes.Ne-228 10 (Zagoruyko & Komodakis 2016). We follow the same experiment settings of Grathwohl et al. (2019) to remove the batch normalization (BN) (Toffe & Szegedy 2013) in WideResNNet22-100, In addition to standard discriminative models and hybrid model JEM, we also compare HDGE with other canonical algorithms: 1) Glow (Kingma Dhariwal 2018) which is compelling flow-based generative model. 2) JointL (Winkens et (0220), recent state-of-the-art which proposes to pretrain using contrastive loss and then finetune with a joint supervised and contrastive loss, and shows the SimCLR loss improves likelihood- (top), HDGE consistently outperforms all the baselines. The corresponding distribution score visualized in Figure it shows that HDGE correctly assign lower scores to out-of-distribution samples and performs extremely well on detecting samples from SVHN, CIFAR- 100, and CelebA
to be OOD is effective, and the likelihoods from EBMs
OOD detection. The results are shown in Table
Interestingly, while Nalisnick al. (2019) demonstrates powerful neural generative models trained to estimate density p(x)) can perform poorly on OOD detection, often assigning higher scores to OOD data points (e.g. SVHN) than in-distribution data points (e.g. CIFAR10). HDGE successfully assign higher scores only to in-distribution data points as shown in the histograms in Figure] []
We believe that the improvement of HDGE over JEM is due compared with SGLD sampling based methods, HDGE holds the ability to incorporate large number and diverse samples and their corresponding labels information train the generative conditional log q( ly). Comparing with contrastive learning approach (Winkens et al. 2020). HDGE differs in that the contrastive loss inside the log q yyly utilizes label information to help contrast similar data points. The empirical advantage of HDGE over Jointl loss shows the benefit of incorporating label information.
5.1.2 PREDICTIVE DISTRIBUTION p(y|æ)
widely used OOD score function the maximum prediction probability (Hendrycks & Gimpel 2016) which is given by so (z:) max, Po (y (z). Intuitively, model with high classification accuracy tends to has a better OOD performance using this score function.
We compare with HDGE with standard discriminative models, generative models, and hybrid models. We also evaluate a contrastive pre- pre-trainin training baseline which consists of learning a representation via contrastive learning and training linear classifier on top of the representation.

---

Under review as a conference paper at ICLR 2021
The results of OOD detection are show in Table 5.1 (bottom). We find HDGE performs beyond the performance of a strong baseline classifier and considerably outperforms all other generative modeling and hybrid modeling methods. The OOD detection evaluation shows that it is helpful to jointly train the generative model q(<ly) together with the classifier p(yl x) to have a better classifier model. HDGE provides effective and simple approach improve duttt--distribution detection.
Freoy Say
HDGE
Score
Score
Froquetcr
The enty
Housera
JEM
cfar20 colosa
dlari
obrro
Score
Score
Score
Figure I: Histograms for OOD detection using density ((2) as score function. The model is WideResNet-25 28-10 (without BN). Green corresponds to the score (in distriibution) training dataset CIFAR- 10, and red corresponds the score on the testing dataset. The cifar cifar 10interp denotes dataset that consists of linear interpolation of the CIFAR-10 10 dataset.
5.2 CONFIDENCE CONFDDENCE-CALIIRATION CALIBRATION
Calibration plays an important role when deploy the model in real-world scenarios where outputting an incorrect decision can have catastrophic consequences (Guo et al. 2017). 'he goodness of calibration is usually evaluated in terms of the Expected Calibration Error (ECE), which is a metric to measure the calibration of classifier.
works by first computing the confidence, max, p(y(z;;, for each x; in some dataset and then grouping the items into equally spaced buckets [Bm] | based on the classifier's output confidence. For example, if M 20, then Bu would represent all examples for which the classifier's confidence was between 0.0 and 0.05. The ECE is defined as following:
ECE
Σ
M
Bmll
B
lacc( conf conf(Bnnll.
(16)
12
m==1
where n is the number of examples in the dataset, acc( is the averaged accuracy of the classifier of all examples in Bn, and conf(B,,) is the averaged confidence over all examples in Bm- For perfectly calibrated classifier, this value will (2019), we choose M = 20 throughout the experiments. A classifier is considered calibrated if its predictive confidence, max,, ky) aligns with its misclassification rate. Thus, when a calibrated classifier predicts label y with confidence score that is the same at the accuracy.
for any choice of M. Following Grathwohl et al.
AOO
ACO
Coulieene
Coulfoone
Coule-
05 Accommy
Accermayy
Figure 2: CIFAR- calibration results. The model is WideRessNet28-111 10 (without BN). Expected calibration (ECE) (Guo al. 2017) CIFAR- 100 dataset under various training losses.
We evaluate the methods CIFAR-100 where we train HDGE and baselines of the same architecture, and compute the ECE on hold-out datasets. The histograms of confidence and accuracy of each method are shown in Figure
While classifiers have grown more accurate in recent years, they have also grown considerably less calibrated (Guo et al. 2017), as shown in the left of Figure Grathwohl et al. (019)) significantly
7

---

Under review as a conference paper at ICLR 2021
improves the calibration of classifiers by optimizing g(z) EBMs training (Figure middle), however, their method is computational expensive due to the contrastive divergence and SGLD sampling process and their training also sacrifices the accuracy of the classifiers. In contrast, HDGE provides a computational feasible method to significantly improve both the accuracy and the calibration at the same time (Figure 22 right).
5.3 IMAGE CLASSIFICATION
We compare HDGE with (i) the supervised learning baseline uses the standard cross-entropy loss. We follow the settings of Zagoruyko & Komodakis (2016) for evaluation on CIFAR-I0 and CIFAR- 100, and we decay the learning rate by 0.2 at epoch 60. 120, 160. (ii) Supervised Contrastive Learning from (Khosla et al. 2020). which proposes use label information select negative examples the contrastive pre-training stage, and shows incorporating the label information helps the downstream supervised training of classifiers. We adapt the official implementation of the Supervised Contrastive Loss to use WideResNet. (iii) JEM from (Grathwohl et al. 2019), which proposes to incorporate energy-based modeling training with the standard cross cross-entropy loss.
As reported in Table 2 HDGE outperforms standard Supervised Learning (which uses only the qo (yla)) loss term), outperforms Supervised Contrastive Learning from Khosla et al. (2020) (which uses different approximation the go (1)z))) outperforms JEM (which uses the classification loss ga (y(x)) supplemented with a loss on the marginal (x)),, and outperforms HDGE with log ge ( (xlyy (which only trains the generative loss term). This shows the benefit of hybrid discriminative and generative model via jointly optimizing the discriminative (classifier) loss and the generative (con- trastive) loss. addition, when studying methods that only have the generative term qo (<r (r)y), we see that HDGE (log go (aly) only) achieves higher accuracy than Khosla et al. (2020), verifying our method provides an improved generative loss term.
Method
Supervised Learning
Supervised Contrastive
HDGE
Dataset
JEM
HDGE (ours)
(log go (<r (c)ly) only)
CIFAR10 10
95.8 .15
96.3 + 24
94.4 + .17
96.7
.10
96.4 + .12
CIFAR100
79.9 + .21
80.5 + .11
78.1 + .10
80.9
.09
80.6 + .10
Table 2: Comparison three standard image classification datasets: All models use the same batch size of 256 and step-wise learning rate decay, the number of training epochs is 200. The baselines Supervised Contrastive (Khosla et al. (2000), JEM (Grathwohl et (2019), and our method HDGE based WideResNet- 28-10 Zagoruyko Komodakis 2016).
5.4 HYBRID DISCRIMINATIVES CACRLMINAATVEEGGEEERRVVVVV MODELING TASKS
GENERATIVE
HDGE models can be sampled from with SGLD. However, during experiments we found that adding the marginal oogliiili over x (as done JEM) improved the generation. we hypothesis that this is due the approximation via contrastive learning focuses on discriminating between images of different categories rather than estimating density.
So we evaluated generative modeling through SGLD sampling from model trained with the following objective:
min (x. [log (z)()) + log qo (2ly) + log go(z))
(17)
where log qo (æ) is optimized by running SGLD sampling and contrastive divergence as in JEM and log ()()))) + log ge(xlg (2lg)) is optimized through HDGE.
We train this approach on CIFAR- 10 and compare against other hybrid models as well as standalone generative and discriminative models. We present inception scores (IS) (Salimans al. (0166) and Frechet Inception Distance (FID) (Heusel et al. 2017) given that we cannot compute normalized likelihoods. The results shown in Table and Figure
The results show that jointly optimizing log go (y (x) + log g0 (xly) + log (x) by HDGE (first two terms) and JEM (third term) together can outperform optimizing log go go (y)ar) + log go (ar) by JEM, and
8

---

Under review as a conference paper at ICLR 2021
it significantly improves the generative performance the state of the art in generative modeling methods and retains high classification accuracy simultaneously. We believe the superior performance of HDGE JEM is due to the fact that HDGE learns a better classifier and JEM can exploit it and maybe optimizing log (<rly) via HDGE is a good auxiliary objective.
Class
Model
Accuracy%
ISt
FID
Residual Flow
70.3
3.6
46.4
Glow
67.6
3.92
48.9
IGEBM
49.1
8.3
37.9
Hybrid
JEM
92.9
8.76
38.4
HDGE
94.6
N/A
N/A
HDGE+JEM
94.4
9.19
37. 6
WideResNet
95.8
N/A
N/A
Disc.
WedeResNeetw/o BN)
93.6
N/A
N/A
SNGAN
N/A
8.59
25.5
Gen.
NCSN
N/A
8.91
25.32
Table 3: Hybrid modeling results on CIFAR-10. All models are based on WideResNNe-28-10 (Zagoruyko & Komodakis ( without BN). Residual
Figure 3: Class-conditional
Flow (Chen et al. 2019). Glow (Kingma & Dhariwal 2018). IGEBM (Du & Mordatch 22099) SNGAN (Miyato et 2018), NCSN (Song & Ermon 2019).
samples generated by running HDGE+JEM CIFAR-10 10.
JEM (Grthwwohl 2019)
5.5 ADVERSARIAL ROBUSTNESS
The commonly considered adversarial attack is the -norm con- strained adversarial examples, which are defined as è B(x, E) that changes the model's prediction, where ((x, r) denotes ball cen- tered at x with radius under the norm metric. In this work, we run white-box box PGD (projected gradient descent) attack with respect to the L2 and Log norms, giving the attacker access to gradients, which PGD is used to find local maximal within given perturba- tion ball (Madry et al. 2017). We train HDGE and compare with the state-of -fee-----th-rrttt adversarial training methods. Adv Training (Madry et al.) 2017 Santurkar et al. 2019) which proposes to use robust optimization train classifier be robust to the norm through which it is being attacked. Results from the PGD experiments can be seen in Figure 4 We can see that HDGE can achieve compelling robustness to the state-of-the-arr adversarial training methods.
L Robustness
Accurnet
Joe Traens
We note that while JEM improves the robustness too by optimizing the likelihood of EBMs, it requires computationally expensive SGLD contrast, HDGE significantly improves the robustness of standard classifiers by computationally scalable contrastive learning.
sampling procedure.
(b) L2 Robustness
Figure 4: Adversarial robust- ness results with PGD attacks. HDGE adds considerable robust- ness to standard supervised train- ing and achieves comparable ro- bustness with JEM.
CONCLUSION
In this work, we develop HDGE, a new framework for supervised learning and contrastive learning through the perspective of hybrid diseriminative and generative model. We propose leverage contrastive learning to approximately optimize the model for discriminative and generative tasks. JEM (Grathwohl et al., (0199) shows energy-based models have improved confidence-callirattin.. out-of-distribution detection, and adver- sarial robustness. HDGE builds on top of JEM and contrastive learning beats JEM and contrastive loss all of the tasks and performs significantly better or on par with state-of-ther of-the-att arrt hand-tailore tailored methods in each task. HDGE gets rid of SGLD therefore does not suffer from training instability and also conceptual simple to implement. We hope HDGE will be useful for future research of hybrid seerriinative-eeeraati training.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jöm-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp. 9916-9926, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint UrXiv:2002.05709 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-superrvsed models are strong semi-supervise learners. arXiv preprint arXiv:2006.10029, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003., 04297, 2020c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3608-3618, 2019.
Will Grathwohl, Kuan-Chich Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: 02.02263, 2019.
-aantensteen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.07755 07733, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. arXiv preprint arXiv: 106,::/70 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-con estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13 (Fcb):307-361, 2012.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR '06), volume 2. pp. 1735-1742. IEEE, 2006.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 19Xiv:1911.05722. 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribation examples in neural networks. arXiv preprint 1610.022 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
10

---

Under review as a conference paper at ICLR 2021
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1711-180) 2002.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint wrxivv/502.03100 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprini orXiv:2004.11362 2020.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in neural information processing systems, pp. 10215-10222 2018.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In Proceedings of the 25th international conference on Machine learning. pp. 536-543, 2008.
Julia A Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In 2006 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'06) 066), volume 1, pp. 87-94. IEEE, 2006.
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference Computer Vision, pp. 2774-2783, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy based learning. 2006.
Weitang Liu, Xiaoyun Wang. John Owens, and Sharon Yixuan Li. Energy-based out- -iit---ditttitttion detection. Advances in Neural Information Processing Systems, 33, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv: 156i::70660608.. 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances neural information processing systems, pp. 31 3111-33 11-3119, 2013.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint wxX.:0800.00557. 2018.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with contrastie estimation. In Advances in neural information processing systems, pp. 2265-2273, 2013.
Rafael Müller, Simon Korbliihh, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019. URL
net/ wwtp://eee//ooom.eteeeee.. http:/orruuiiiiiiiiiiiiiiiiiiiii XwNhhhCYmmmmmmmm
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, 1.
Andrew Y Ng and Michael I Jordan. On discriminative generative classifiers: A comparison of logistic regression and naive bayes. In Advances in neural information processing systems, pp. 848-848, 2002.
11

---

Under review as a conference paper at ICLR 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8820-002333 2019.
Rajat Raina, Yirong Shen, Andrew Mccallum, and Andrew Y Ng. Classification with hybrid seneraaive/discciminatve models. In Advances in neural information processing systems, pp. 545-552, 2004.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Image synthesis with a single (robust) classifier. In Advances in Neural Information Processing Systems, pp. 1262-12777 1273, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems, pp. 1857-1865, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 19910-1933, 2019.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 28 188822 2016.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. arXiv preprint arXiv:2007.08177 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05844 05.599960549, 2019.
Zhuowen Tu. Learning generative models via discriminative approaches. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-oo-diistrbuution detection. arXiv preprint arXiv:2007.05566, 2020.
Zhirong WU, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. arXiv preprint 180iv: 009977 2018.
Jianwen Xie. Yang Lu, Song- Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 16X1v:1605.07146. 2016.
12

---

Under review as a conference paper at ICLR 2021
A
PYTORCH LIKE IMPLEMENTATION
Algorithm Hybrid Discrininative-Geneaatiee Training via Contrastive Learning
Ft encoder networks for images
negativv iive Tog'ta tas Rax
quellez queue of probabl1 1es [KXC]
I_neg naon naonithos thoo, oeeobbok Ty. guece. alo-lloneellll - dt
size; Kt queue sizej number
BNCNK
classes; mi momentum; t: temperature
I_neg Logsunexp (-_neg, din-1;
for in loadez:
Load minibarch with
A Logits: Bx [5-R]
images x and a opctonat rendon data
logits cat ([-_pos, [_neg], din=1)
x aug aug (x)
positives labels Sar0aiK]
0-th
augmentat transform to one-hot vector with
y (y)
dimansion
opt log
1oss
logits: Exc
cl_loss = croosbarroyyoosssuugtttt labels)
ce ve_logits
(or
hybrid training
standazd czore-entropy loss
1oss ce_loss ci_loss
ce ce_loss CroeeEnt ropyloss Iee lce_logits, y)
# octimize log glylx)
sap updates encoder netwerk
nermalize locits by, 12: BxC
update 11. (2.parans)
probe normal [98]08_ lggita)
update (queue, probs)
positive
the current
poc logaumexp iprobsey, din-1,
minibst probs
keapaim=Tsua)
daqueue (quaue) dequeue the earliest minibateh
vinsua Eissel sun cal. conatesatin, Jequusaagi LeghurdEap opeaaioo
Nut-of-distribution
98
Model
SVHN
Interp
CIFAR100
CelebA
WideResNet-28-
.46 + 21
.41
.19
.47 + .23
.49 21
Unconditional Glow
:05
.51 + 23
.55 + .22
.57 + .19
Caas-Connittioal Glow
.07
.45
.11
53
IGEBM
.63
.70
.50 .50
.70 114
log p(z)
JEM
.67
.65
.67 1.15
.75 + .12
HDGE (ours)
96
.11
91 .09
.80 + .12
JEM + HDGE (ours)
9.12
413
.90 4.12
.80 + .15
.995 + .10
.929 + .09
HDGE (ResNet-50)
.995 .08
.938 + .06
WideResNet-28.
.93 93 .13
.77
.85 + .21
.62 + .23
Contrastive pretraining
.11
155
.8 116
.58 117
Conditonni Glow
.64
.61
.65
.54 .2
max, p(y(z)
IGEBM
.13
.69 221
.4 166
.69 699
JEM
.89
.18
871
.79 .22
HDGE (ours)
.95 111
.12
.84 849
.81 + .07
JEM + HDGE (ours)
148
.77 113
.88 4: .13
.80 + .11
Table 4: OOD Detection Results. The model is WideResNet-255 28-10 (without BN) following the settings of
JEM (Grathwohl et al.) 2019), except ResNet-50 when comparing with JointLoss (Winkens al. (2020). The training dataset is CIFAR-IO. Values are AUROC, Results of the baselines are from Grathwohl et al. 2019 and Winkens et al. 2020
B EXPERIMENT DETAILS
B.1 EVALUATION DATA
To evaluate HDGE, we completed a thorough empirical investigation on several standard datasets: CIFAR-10 10 and CIFAR- 100 (Krizhevsky al. 2009), two labeled datasets composed of 32 x 32 images with 10 and 100 classes respectively (Sections 5.1 5.2 5.3 and 5.4h: SVHN (Netzer et al.
2011), labeled dataset composed of over 600, 000 digit images (Section 5.1); CelebA (Liu et al. 2013). a labeled dataset consisting of over 200, 000 face images and each with attribute annotation (Section 5.1
13

---

Under review as a conference paper at ICLR 2021
B.2 TRAINING DETAILS
Our training settings follow exactly that of JEM, except stated otherwise in some ablation study. Pseudo-code for training procedure is in Algorithm
The cross-entropy baseline is based on the code from the official PyTToch training code HDGE's im- plementation based on the official codes of MoCo and JEM Our source code in Py Torch (Paszke et al. 2019) is available online
In the OOD evaluation, the results of JointLoss are obtained from Winkens et 2020), the results of JEM and other baselines are obtained from Grathwohl et al. (2019).
Our method HDGE follows the experimental settings of JEM and have exactly the same hyperparam- eters in optimization and model choices as JEM. the temperature = 0.1 as in other experiments conducted in this work.
One baseline JointLoss (Winkens et al. 2020) uses a different model ResNet-50, to have a fair comparison, our HDGE (ResNet-50) also uses ResNet-50. JointLoss also incorporates multiple training techniques such as LARS optimizer and label smoothing to help training which we do not use in HDGE.
The likelihood score log ((r) is calculated by applying LogSumExp operation on the log ((yl=:) within HDGE. Specifically,
log (x) = log Σ (T. y) = log cxp(f(r)ll)
(18)
Z
where Z is the normalization constant. The score log ((r) we care about is then Ey exp(f(x)()) (f())l)) A similar scheme also proposed in recent OOD detection work (Liu et al. 2020
C SIMCLR STYLE IMPLEMENTATION OF log p(zl))
We conducted a comparison between HDGE with MoCo and SimCLR style approximations of the contrastive loss in log p(rly). One of the key differences between MoCo and SimCLR is that MoCo uses a gradient disablaed memory to save logits while SimCLR simply increase batch size. Chen et al. (2020a demonstrate that SimCLR can outperform MoCo significantly.
We use batch size 2048 in our SimCLR style HDGE and its pseudo code similar to Algorithm is shown in Algorithm 2
The results are shown in Table we can see that SimCLR style of HDGE performs comparably with the default MoCo style, indicating Hybrid DieerriinaiveeGGeeraaiiee Training insensible to detail implementation choices, However, our default implementation Algorithm has less requirements on computation memory size, which makes it widely applicable.
D GOODNESS OF APPROXIMATION
Since made the approximation energy-based model by contrastive learning in Equation 13) we are interested in evaluating the impact of the number negative examples K on the goodness of this approximation. We consider a classification task and density based OOD detection task as proxies of evaluating the approximation.
Classification. We compare the image classification of HDGE on CIFAR-100. The results are shown in Figure We found that increasing number of negative samples K improves the performance of HDGE, and with sufficient number negative examples HDGE significantly outperform the
com/ftcchookreeeacc//moooo
'https://gtthbb.. http::/gghuubcoomwggneee com//grratwohh/JJM aanonymuus during double-blind review
14

---

Under review as a conference paper at ICLR 2021
Algorithm 2 HDGE with SimCLR style approximation
F: encoder networks For Images temperature
pop logsunexp (probaxy, dim=1,
classes;
keepatn-Tróe)
negst logita: BxB
images x and Jabeis 3 randon data augmentat ion
y in loadez:
Load minibasch with
Leg einsun bb,
Togsunexp ("_neg, din-1?
IVr x1)
y (ty # trarsform to one-hot vector with
logits eat po-, i_neg), dim-1)
logits: Bx (1-B1
dimension C
positives are the 0-th
logits: BxC ce ce_logits - "f, (or
1 1sbels seros
standazd ccozs-entropy
contraatiive
optinize log
el d_loss - rrooeadntroyylosssloiiio
labels)
ce_loss troesEntropylors ice ice_logits,
hybrid treining
nermalize Locits by, 12; Bxc
1oss ce_loss ci_loss
probs normsl feleeeee__le logits)
f pasitive logits: Bxl
5.00 updates encoder network
update1f.parena. 1f parans)
einsur: Eiessi eat. comotecttion Zogoumargs LeghtnEEp opecation
Out-of-distribution
(0())
HDGE
SVHN
Interp
CIFAR100
CelebA
default
.96 + .08
.82 + .11
.91 = .09
.80 + .12
logp(x)
SimCLR style
.97 = .06
.82 + .13
.91 .111
.80 + .15
no logits normalization
.96 = .12
.11
.91 = .09
.80 + .12
default
.95 .11
.76 76..12
.84 .09
.81 .07
maxy p(y|x)
SimCLR style
.96 - .12
76.
.91 - .05
.80 110
no logits normalization
.95 .11
.76
.84 .11
.81 +
Table 5: OOD Detection Results. The model is WideResNet-28 28-11 (without BN) following the settings of JEM (Grathwohl et al. 2019). The training dataset is CIFAR-10. Values are AUROC.
cross-entropy loss. The reason may be training with many negative examples helps to discriminate between positive and negative samples.
OOD detection. We evaluate HDGE with different value of by rumning experiments the log p(x) based OOD tasks, use the same experiments setting as Section[5..
15
We vary the batch size of SGLD sampling process in Grathwohl al. (2019), effectively, we change the number of samples used to esti-
mate the derivative of the normalization constant Epa(ar) 0Ea(z') 88
in the JEM update rule Equation (4). Specifically, we increase the default batch size N from 64 128 and 256, due to running the SGLD process is memory intensive and the technique constraints of the limited CUDA memory, we were unable to further increase the batch size. We also decrease K in HDGE to {64, 128, 256} to study the effect of approximation.
(lqgwaal))
Figure 5: Accuracy compari- son with respect different K on CIFAR- 100. The baseline is standard cross-entropy loss. The model is WideResNet-28-100 Batch size is 256.
The results shown in Table the results show that HDGE with small K performs fairly well except on CelebA probably due to the simplicity of other datasets. We note HDGE(K 64) outperforms JEM and three out of four datasets, which shows the approximation HDGE is reasonable good. While increasing batch size of JEM improves the performance, we found increasing K HDGE can more significantly boost the performance on all of the four datasets. We note JEM with a large batch size significantly more computational expensive than HDGE, as a result JEM runs more slower than HDGE with the largest K.
15

---

Under review as a conference paper at ICLR 2021
Out-of-distribution
sx(x)
Model
SVHN
Interp
CIFARI00
CelebA
JEM (N 64) (default)
.67
.65
.67
.75
JEM (N 128)
.69
.67
.68
.75
JEM (N 256)
.70
.69
.68
.76
logp(x)
HDGE (K =44)
.89
.79
.84
.62
HDGE (K -128)
.91
800
.89
.73
HDGE (K 256)
.93
.81
.90
.76
HDGE (K 65536) (default)
.96
.82
.91
.80
Table 6: Ablation of approximation on detecting OOD samples. We use CIFARIO in-distribution. N the batch size of JEM and HDGE. HDGE uses learning.
64. K the number of negative samples in contrastive
16


Reviews:
[{"review_id": "OD5vdPJauN_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n- Paper proposes Hybrid Discriminative Generative training of Energy based models (HDGE) which combines supervised and generative modeling by using a contrastive approximation of the energy based loss\n- Approach shows this is better than baselines on various tasks like confidence calibration, OOD detection, robustness and classification accuracy\n\n\nClarity\n- Overall well written paper. Figures and tables are informative and supplement the flow.\n- Formatting error in figure 4 in appendix\n\n\nNovelty\n- Paper proposes a simple but unified view of contrastive training, generative and discriminative modeling - a nice, novel contribution with empirically strong results\n- Gets rid of computationally expensive SGLD by using contrastive approximation which was a key limitation of prior energy based modeling work like JEM\n\n\nSignificance\n- Results are compelling across a wide range of tasks over existing (EBM) baselines including calibration, robustness, OOD detection, generative modeling and classification accuracy\n\n\nQuestions/clarifications/comments\n- “, Grathwohl et al. (2019) show the alternative class-conditional EBM p(y|x) leads to significant improvement in generative modeling while retain compelling classification accuracy” -> Not sure the JEM model is class conditional\n\n- How alpha = 0.5 (the weighting chosen)? The details are not presented.\n\n- Error bars are missing for the classification accuracy experiments in Table 1 which makes it hard to verify improvements especially wrt supervised contrastive loss method\n\n- Detail on how classification accuracy is computed when using generative term in HDGE is missing? Is it a linear classifier on top of learned representations?\n\n- “Prior work show that fitting a density model on the data and consider examples with low likelihood to be OOD is effective” -> Not completely true see https://arxiv.org/abs/1810.09136\n\n- Please share exact details on how p(x) for OOD score is calculated\n- Error bars again missing in Table 2\n\n- “We find HDGE performs beyond the performance of a strong baseline classifier” - this is a strong statement as only for CIFAR10/Celeb A the gains of HDGE are clear\n\n- Why was the Winkens et al, 2020 contrastive baseline not used here to compare in Table 2 - https://arxiv.org/abs/2007.05566?\n- “HDGE is conceptual simple to implement, scalable, and powerful.” -> conceptually. Also scalability is a somewhat strong claim as the main datasets used here are CIFAR variants.\n\n- Was HDGE + JEM experiments also performed for OOD detection?\n\n- Legend in figure 4 should be “HDGE” not “HDSE”?\n\nOverall good effort with seemingly good improvements over prior efforts on hybrid EBMs over a number of tasks. Main concern is lack of error bars which makes it hard to validate claims in certain cases.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MUfUK2TS8rU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===============Update after rebuttal period================\nThe connection between the contrastive learning objective and discriminative learning is made via \"resemblance\". And the author claims the \"resemblance\" as a theoretical contribution, which the first reason I vote for a clear rejection. This issue has not been addressed by the authors. The second reason for my rejection of the paper is the paper requires an effort to make it self-contained, especially for the experimental section. I remain my score of clear rejection.\n\n=======================================================\nThis paper connected contrastive-learning and supervised learning from the perspective of the energy-based models. Then, the authors combine both objectives and evaluate the presented method on various datasets and tasks.\n\nStrengths: The paper attempts to connect supervised and contrastive learning. I like the attempt. But unfortunately, I don't think it is valid. See explanations as follows.\n\nWeakness:\n1. I feel the claim in the paper is too strong. The approximation from equation 12 to 13 is very crude. Specifically, the approximation states that the infinite integral (for the normalization constant) can be replaced by a finite sum, which is generally not true. \n2. Even if we assume the above approximation is fine, the connection with contrastive learning is very unclear. Precisely, the approximation is for modeling p(x|y), yet the contrastive learning is modeling p(x_1|x_2) with  x_1 and x_2 being the outcomes from correlated data. The authors do not discuss or compare between p(x|y) and p(x_1|x_2), and hence it makes the connection very vague. \n3. The resulting objective (eq. 15) is a combination of the discriminative and generative modeling, which has already been studied. \n4. On page 4, \"the representation captures important information between similar data points, and therefore might improve performance on downstream tasks.\" This sentence is super vague, and I can't understand what the \"important information\" is and why if we capture \"this important information\", we \"may improve performance on downstream tasks.\" The author should spend time polishing the presentation.\n5. The main complaint of the presentation is the overclaim for the experimental section. I understand the contents are too much, and hence the author must move some experimental sections into Appendix. The author claims that the proposed method is performed on adversarial modeling and generative modeling, while these two sections only appear in the Appendix. In the last few lines of page 5, the author seems to rush the remaining experimental sections into the Appendix and asks the reviewer/reader to read themselves. The author should spend time arranging the contents and make sure the paper is self-contained. \n\n==================================\nSummary of the reasons why I vote for rejection:\n1. The main contribution of the paper by connecting supervised learning and contrastive learning is overclaimed. The approximation of the intractable normalization term is not appropriate. The connection with contrastive learning is not solid.\n2. The paper doesn't seem to be ready for submission. The content is not organized well and some ambiguous wordings should be avoided.\n\n\n[1] Representation Learning with Contrastive Predictive Coding by Oord et al.\n[2] On Variational Bounds of Mutual Information by Poole et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5NwFaKQ5y-Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes HDGE - a simple method to improve over JEM. JEM is optimized using a combination of two terms:\n$\\log p(y|x) + \\log p(x)$\n\nThe first term is optimized using the standard cross-entropy loss, while the second term is optimized using SGLD. Running SGLD chains in each iteration can cause instability. In HDGE, instead of optimizing $\\log p(x)$, an approximation to the conditional density $\\log p(x|y)$ is optimized. The idea is to approximate the normalization constant $Z(\\theta)$ with an empirical averaging of energy functions over a large memory bank. This yields a simple objective to optimize. The benefit of using such an approximation is that this eliminates the need for running SGLD, thereby improving the stability of training.\n\nThe idea itself is simple and intuitive. Experiments show that HDGE consistently outperform / perform on-par with JEM on image classification, OOD detection and calibration.\n\nShould we call this a contrastive objective? Im not super convinced if the objective of $\\log p(x|y)$ can be called a contrastive objective. Because in contrastive losses, we always focus on pairs of samples, i.e., we contrast the representation of one sample to another, while the objective in this paper takes the form similar to cross-entropy loss instead. Should the loss be called something instead?\n\nI would like the authors to have a discussion on the training stability of HDGE compared to JEM. It looks like HDGE would be more stable since we don't need to run SGLD, but this message should be made more clear as this is the most important improvement over JEM.\n\nThe performance improvement in table 1 is marginal. So, it is important to perform multiple runs and report mean and standard deviations to understand the statistical significance of the results.\n\nCan HDGE be used for generative modeling? i.e., how do you sample from p(x)? Experiments in appendix show that HDGE can be used in combination with JEM for generative modeling, but this again requires running SGLD. Can HDGE be used in isolation for generative modeling tasks?\n\nHow does the performance compare with other SOTA metods for OOD detection and calibation? Some comparisons that could be done include (but not limited to): Ren et al., \"Likelihood Ratios for Out-of-Distribution Detection\", Padhy et al., \"Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks\", Morningstar et al. \"Density of States Estimation for Out-of-Distribution Detection\", etc. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5EPTqE3Y8u_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1, Summary of contribution:\nThis paper gives a unified view of contrastive learning and supervised learning through energy-based model (EBM) training. Specifically, it claims that the contrastive loss can approximate the log-likelihood of the energy-based model, and shows that the proposed objective works on par with or better than its counterparts. \n\n2, Strengths and Weaknesses:\nThe proposed method of training energy-based models achieves good performance on various tasks (e.g. OOD) without SGLD which usually requires a lot of tricks to work well on high-dimensional data.  This is the strength of this paper, and the community might be able to leverage the idea like this to extend the application of EBM to other domains in the future. \nMeanwhile,  as the authors admit themselves,  the approximations are not thoroughly justified. Whether the research is appropriately bridging the gap between EBM and contrastive learning is questionable. This is the weakness of the paper. \n\n3, Recommendation:\nMarginally below the acceptance threshold. I believe that their claimed connection to EBM is overstated.\n\n4, Reasons for Recommendation:\nThere are several factors that make me question whether the model learned by their proposed method can be legitimately interpreted as a variant of EBM. To name a few, \n(1) As the authors write themselves, the approximation in eq (13) is crude. While the summands in the denominator are sampled from probability distributions (data distribution), when in fact it is supposed to be integral with respect to Lebesgue measure. \n(2) The algorithm seems to be introducing some mysterious trick that makes the model ignore the gradient with respect to negative samples and normalization factor. For some unstated reason, the algorithm is also normalizing the logits.\n\nThe sheer results look promising, and it seems that the method is succeeding in capturing some aspects of the data distribution. However, they do not necessarily justify the claimed connection between HDGE and EBM. It could be that HDGE is just succeeding in learning the support of the dataset.  \nIn order for this paper to reach the standard of ICLR, I believe that the authors must revise the strength of their work and redesign their experiments as such.  \n\nI believe that the paper can be improved if, within the scope of revision, the author can provide more solid justifications for approximations and tricks used in the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes hybrid discriminative + generative training of energy-based models (HDGE) building on JEM. By connecting contrastive loss functions to generative loss, HDGE proposes an alternative loss function that reduces computational cost of training EBMs.

The reviewers agree that this is an interesting idea and that the empirical results look promising.
However, multiple reviewers raised concerns that the theoretical justification was incomplete and felt that some of the claims about the equivalence between the two, as well as some of the practical approximations introduced, need more justification. 

I encourage the authors to revise the paper and resubmit to a different venue.

Decision: Reject