Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
- Paper proposes Hybrid Discriminative Generative training of Energy based models (HDGE) which combines supervised and generative modeling by using a contrastive approximation of the energy based loss
- Approach shows this is better than baselines on various tasks like confidence calibration, OOD detection, robustness and classification accuracy


Clarity
- Overall well written paper. Figures and tables are informative and supplement the flow.
- Formatting error in figure 4 in appendix


Novelty
- Paper proposes a simple but unified view of contrastive training, generative and discriminative modeling - a nice, novel contribution with empirically strong results
- Gets rid of computationally expensive SGLD by using contrastive approximation which was a key limitation of prior energy based modeling work like JEM


Significance
- Results are compelling across a wide range of tasks over existing (EBM) baselines including calibration, robustness, OOD detection, generative modeling and classification accuracy


Questions/clarifications/comments
- “, Grathwohl et al. (2019) show the alternative class-conditional EBM p(y|x) leads to significant improvement in generative modeling while retain compelling classification accuracy” -> Not sure the JEM model is class conditional

- How alpha = 0.5 (the weighting chosen)? The details are not presented.

- Error bars are missing for the classification accuracy experiments in Table 1 which makes it hard to verify improvements especially wrt supervised contrastive loss method

- Detail on how classification accuracy is computed when using generative term in HDGE is missing? Is it a linear classifier on top of learned representations?

- “Prior work show that fitting a density model on the data and consider examples with low likelihood to be OOD is effective” -> Not completely true see https://arxiv.org/abs/1810.09136

- Please share exact details on how p(x) for OOD score is calculated
- Error bars again missing in Table 2

- “We find HDGE performs beyond the performance of a strong baseline classifier” - this is a strong statement as only for CIFAR10/Celeb A the gains of HDGE are clear

- Why was the Winkens et al, 2020 contrastive baseline not used here to compare in Table 2 - https://arxiv.org/abs/2007.05566?
- “HDGE is conceptual simple to implement, scalable, and powerful.” -> conceptually. Also scalability is a somewhat strong claim as the main datasets used here are CIFAR variants.

- Was HDGE + JEM experiments also performed for OOD detection?

- Legend in figure 4 should be “HDGE” not “HDSE”?

Overall good effort with seemingly good improvements over prior efforts on hybrid EBMs over a number of tasks. Main concern is lack of error bars which makes it hard to validate claims in certain cases.

review_title: a good submission empirically improving on top of prior hybrid EBM works

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes HDGE - a simple method to improve over JEM. JEM is optimized using a combination of two terms:
$\log p(y|x) + \log p(x)$

The first term is optimized using the standard cross-entropy loss, while the second term is optimized using SGLD. Running SGLD chains in each iteration can cause instability. In HDGE, instead of optimizing $\log p(x)$, an approximation to the conditional density $\log p(x|y)$ is optimized. The idea is to approximate the normalization constant $Z(\theta)$ with an empirical averaging of energy functions over a large memory bank. This yields a simple objective to optimize. The benefit of using such an approximation is that this eliminates the need for running SGLD, thereby improving the stability of training.

The idea itself is simple and intuitive. Experiments show that HDGE consistently outperform / perform on-par with JEM on image classification, OOD detection and calibration.

Should we call this a contrastive objective? Im not super convinced if the objective of $\log p(x|y)$ can be called a contrastive objective. Because in contrastive losses, we always focus on pairs of samples, i.e., we contrast the representation of one sample to another, while the objective in this paper takes the form similar to cross-entropy loss instead. Should the loss be called something instead?

I would like the authors to have a discussion on the training stability of HDGE compared to JEM. It looks like HDGE would be more stable since we don't need to run SGLD, but this message should be made more clear as this is the most important improvement over JEM.

The performance improvement in table 1 is marginal. So, it is important to perform multiple runs and report mean and standard deviations to understand the statistical significance of the results.

Can HDGE be used for generative modeling? i.e., how do you sample from p(x)? Experiments in appendix show that HDGE can be used in combination with JEM for generative modeling, but this again requires running SGLD. Can HDGE be used in isolation for generative modeling tasks?

How does the performance compare with other SOTA metods for OOD detection and calibation? Some comparisons that could be done include (but not limited to): Ren et al., "Likelihood Ratios for Out-of-Distribution Detection", Padhy et al., "Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks", Morningstar et al. "Density of States Estimation for Out-of-Distribution Detection", etc. 


review_title: Simple yet effective method for improvement over JEM

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1, Summary of contribution:
This paper gives a unified view of contrastive learning and supervised learning through energy-based model (EBM) training. Specifically, it claims that the contrastive loss can approximate the log-likelihood of the energy-based model, and shows that the proposed objective works on par with or better than its counterparts. 

2, Strengths and Weaknesses:
The proposed method of training energy-based models achieves good performance on various tasks (e.g. OOD) without SGLD which usually requires a lot of tricks to work well on high-dimensional data.  This is the strength of this paper, and the community might be able to leverage the idea like this to extend the application of EBM to other domains in the future. 
Meanwhile,  as the authors admit themselves,  the approximations are not thoroughly justified. Whether the research is appropriately bridging the gap between EBM and contrastive learning is questionable. This is the weakness of the paper. 

3, Recommendation:
Marginally below the acceptance threshold. I believe that their claimed connection to EBM is overstated.

4, Reasons for Recommendation:
There are several factors that make me question whether the model learned by their proposed method can be legitimately interpreted as a variant of EBM. To name a few, 
(1) As the authors write themselves, the approximation in eq (13) is crude. While the summands in the denominator are sampled from probability distributions (data distribution), when in fact it is supposed to be integral with respect to Lebesgue measure. 
(2) The algorithm seems to be introducing some mysterious trick that makes the model ignore the gradient with respect to negative samples and normalization factor. For some unstated reason, the algorithm is also normalizing the logits.

The sheer results look promising, and it seems that the method is succeeding in capturing some aspects of the data distribution. However, they do not necessarily justify the claimed connection between HDGE and EBM. It could be that HDGE is just succeeding in learning the support of the dataset.  
In order for this paper to reach the standard of ICLR, I believe that the authors must revise the strength of their work and redesign their experiments as such.  

I believe that the paper can be improved if, within the scope of revision, the author can provide more solid justifications for approximations and tricks used in the paper.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: ===============Update after rebuttal period================
The connection between the contrastive learning objective and discriminative learning is made via "resemblance". And the author claims the "resemblance" as a theoretical contribution, which the first reason I vote for a clear rejection. This issue has not been addressed by the authors. The second reason for my rejection of the paper is the paper requires an effort to make it self-contained, especially for the experimental section. I remain my score of clear rejection.

=======================================================
This paper connected contrastive-learning and supervised learning from the perspective of the energy-based models. Then, the authors combine both objectives and evaluate the presented method on various datasets and tasks.

Strengths: The paper attempts to connect supervised and contrastive learning. I like the attempt. But unfortunately, I don't think it is valid. See explanations as follows.

Weakness:
1. I feel the claim in the paper is too strong. The approximation from equation 12 to 13 is very crude. Specifically, the approximation states that the infinite integral (for the normalization constant) can be replaced by a finite sum, which is generally not true. 
2. Even if we assume the above approximation is fine, the connection with contrastive learning is very unclear. Precisely, the approximation is for modeling p(x|y), yet the contrastive learning is modeling p(x_1|x_2) with  x_1 and x_2 being the outcomes from correlated data. The authors do not discuss or compare between p(x|y) and p(x_1|x_2), and hence it makes the connection very vague. 
3. The resulting objective (eq. 15) is a combination of the discriminative and generative modeling, which has already been studied. 
4. On page 4, "the representation captures important information between similar data points, and therefore might improve performance on downstream tasks." This sentence is super vague, and I can't understand what the "important information" is and why if we capture "this important information", we "may improve performance on downstream tasks." The author should spend time polishing the presentation.
5. The main complaint of the presentation is the overclaim for the experimental section. I understand the contents are too much, and hence the author must move some experimental sections into Appendix. The author claims that the proposed method is performed on adversarial modeling and generative modeling, while these two sections only appear in the Appendix. In the last few lines of page 5, the author seems to rush the remaining experimental sections into the Appendix and asks the reviewer/reader to read themselves. The author should spend time arranging the contents and make sure the paper is self-contained. 

==================================
Summary of the reasons why I vote for rejection:
1. The main contribution of the paper by connecting supervised learning and contrastive learning is overclaimed. The approximation of the intractable normalization term is not appropriate. The connection with contrastive learning is not solid.
2. The paper doesn't seem to be ready for submission. The content is not organized well and some ambiguous wordings should be avoided.


[1] Representation Learning with Contrastive Predictive Coding by Oord et al.
[2] On Variational Bounds of Mutual Information by Poole et al.
review_title: Review

Meta-review: The paper proposes hybrid discriminative + generative training of energy-based models (HDGE) building on JEM. By connecting contrastive loss functions to generative loss, HDGE proposes an alternative loss function that reduces computational cost of training EBMs.

The reviewers agree that this is an interesting idea and that the empirical results look promising.
However, multiple reviewers raised concerns that the theoretical justification was incomplete and felt that some of the claims about the equivalence between the two, as well as some of the practical approximations introduced, need more justification. 

I encourage the authors to revise the paper and resubmit to a different venue.

Decision: Reject