Venue: ICLR 2020 Conference

Reviews:

final_score: rating: null
review_content: Congrats on the acceptance! Very inspiring work.

I am curious about what the authors think about a line of recent work using classifiers to "boost" the performance of a base generative model. 

1. Boosted Generative Models.  AAAI 2018.
2. Discriminator rejection sampling. ICLR 2019.
3. Metropolis-hastings generative adversarial networks. ICML 2019.
4. Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting. NeurIPS 2019.

Both the current work and the above line of work exploit the inductive bias of classifiers for specifying an energy function (see Section 4 of Ref. 4 above for a concise explanation). The training of the induced energy function here is multi-step (first a generative model and then a classifier) but the labels for the classifier training are artificially induced (real vs. fake) which makes the setup more broadly applicable to even unlabelled datasets. 

I'd be interested to hear if the authors have additional insights to share on similarities, differences, potential hybrids etc. of their work with the above line of work. Thanks!
review_title: Classifier based energy models in the absence of labels

final_score: rating: 6: Weak Accept
review_content: This paper introduces the idea of energy based model to the traditional classifier, and proposes a new framework to improve the performances of the model in multiple aspects. The idea of reinterpreting the traditional classifier is very interesting, and the experiments show some good results of the proposed method.  

Here are my main concerns of the current paper:
1. The training procedure seems to be very sensitive, and the SGLD may take a long time at each iteration to converge. This may be a big limitation of the proposed method.
2. According to equation (8), the proposed method is having a trade-off between classification and generation, and this seems to be the key to improve the performance of the model in generation by sacrificing some classification accuracy. I think author should emphasize this instead of energy based model.
3. The presentation is not very clear in section 5. What is the task of calibration, and what is the definition of ECE?
4. The robustness guarantee seems too good to be true. Although the authors claim that they allow the attacker to have access to the gradient  of SGLD, the SGLD will add noise during the forward process, this will obfuscate the gradient. In this sense, I don’t think the proposed method will have the strong robustness as they claimed.

----------------
Post-Rebuttal Comments:
Thanks for addressing my concerns. Although I think the proposed method is not comprehensive to check obfuscated gradients, I do think the current version is a good fit for ICLR, and I decide to increase my score. 
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper uses energy-based model interpretation for the logits of standard discriminative neural network models to define a generative model inside a classifier that proves useful in many downstream tasks such as uncertainty quantification, out-of-distribution detection, etc.
Although there has been previous work attempting to bridge discriminative classifiers with generative modeling, this work proves to be competitive with both specialized models on discriminative/generative tasks as well as in many downstream tasks such as out-of-distribution detection, calibration, and adversarial robustness. The paper provides a clear exposition of the method, succeeds to discuss related work it bases on, conducts a thorough experimental study providing convincing explanations for results and does not hide the limitations of the work (high computational requirements, optimization difficulties connected with training energy-based model and the method used, limited approximation of the true energy). Overall, the paper provides a substantial contribution and paves the way for further work improving this joint discriminative - generative setting. However, there are points I would like the paper to address for better exposition.
1. It would benefit the paper showing that samples with higher unnormalized likelihood are visually more compelling than those with lower likelihood.
2. On CIFAR100 the accuracy drop from the reference value is larger than for datasets with 10 classes, could it be due the logits dimension is higher and challenges optimization?
3. It would also be helpful to clarify whether application of the proposed method is primarily restricted by the computational complexity or is there any property inherent to energy-based models that makes treating high-dimensional data challenging?

Minor remark
- Although the paper doesn't state on which dataset results shown in Table 1 were obtained, I suspect its CIFAR10, please specify this.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This work is an attempt to bridge the gap between discriminative models, which currently obtain the state of the art on most classification problems, and generative models, which (through a model of the marginal p(x)) have the potential to shine on many tasks beyond generalization to a hold-out set with minimal shift in distributions: out of distribution detection, better generalization out of distribution, unsupervised learning etc.

While much of the current work is related to normalizing flows / invertible neural networks, the authors here propose a quite simple but appealing method: A standard neural classifier is taken and the softmax is layer chopped off and replaced by an energy based model, which models the joint probability p(x,y) instead of the posterior p(y|x). The advantage is an additional degree of freedom in the scale of the logit vector, which is would have been otherwise normalized by the softmax layer and now can now model the data distribution. The downside is the loss in ease of training. Whereas (discriminative) deep networks can be easily trained by gradient descent on a cross-entropy objective, the partition function in the energy model makes this un tractable. This is addressed through sampling, similar to (Welling & Teh, 2011).

One of the biggest achievements reported by the authors is that the performance on discriminative tasks is not hurt (much) by adding the generative model. There is only a 3 point gap between Wide-ResNet and the proposed model (92.9% vs. 95.8%) … but on what dataset? 3 datasets are mentioned in the experimental section, but table 1 does not mention on which datasets the accuracy is reported. My guess is that this is a mean or mixture, since GEM performances of 96.7% and 72.2% are reported for SVHN and CIFAR10, respectively, but this should be made clearer. 	

On out of distribution detection, could the authors comment on the histograms in table 2, in particular the difference between the new measure (AM JEM) compared to JEM log p(x) on CelebA? The proposed measure does not seem to fare well here.

Although the method does not outperform the gold standard of adversarial training, I found the models robustness to adversarial examples quite appealing, given that it was not trained for this objective (which also means that it does not require an adaptation to a norm). 

I was very impressed by Figure 6 showing distal adversarial initialized from random images, showing pretty clear images of the modelled class. The modelled variations require more investigation to verify whether we have a collapse for each class, but the results look very promising.

The paper is well written and easy to understand. A couple of details on the training procedure are missing in the experimental part. It is stated that, both, p(y|x) and the generative part p(x), are optimized, but how are these exactly integrated? Given the difficult in training this model reported in the paper, this seems to be particularly important.

I also appreciated the description of the limitations of the algorithm, and the details in the appendix (ICLR should go back to unlimited paper lengths, btw.).

More information on complexity (training times etc.) should also be helpful. 

review_title: Official Blind Review #2

Meta-review: This paper uses energy based model to interpret standard discriminative classifier and demonstrates that energy based model training of the joint distribution improves calibration, robustness, and out-of-distribution detection while generating samples with better quality than GAN-based approaches. The reviewers are very excited about this work, and the energy-based perspective of generative and discriminative learning. There is a unanimous agreement to strongly accept this paper after author response.
Decision: Accept (Talk)