Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a representations learning method, Energy-Based Contrastive Learning (EBCLR), that combines contrastive learning with energy-based models (EBMs). The authors not only provide theoretical formulation of the proposed framework but also back up all the design choices empirically by providing adequate ablations. 

The efficiency of the method is demonstrated on MNIST, CIFAR-10 and CIFAR-100 where it outperforms contrastive learning baselines in sample efficiency and linear evaluations.
strengths_and_weaknesses: **Strengths:**
- The paper proposes a simple extension of contrastive learning frameworks to energy based models for learning better representations efficiently.
- The proposed method outperforms contrastive baselines and reach convergence (4x to 20x) faster than the baselines.
- The authors provide theoretical formulation of their newly proposed framework.
- Detailed ablations are provided to justify all the design choices.

**Weaknesses:**
- Currently, the related work section is more about explaining the inner workings of contrastive and energy based methods instead of focusing on the prior or similar works. Although I am not highly familiar with literature on energy based models but some insights from contrastive divergence in energy based models [2] (or differences with the present work [1]) would go a long way in helping new readers.
- I would appreciate if the authors could position their work and contributions appropriately w.r.t to the prior work [1].
- Although it's not a weakness, but I would have appreciated if the authors demonstrated benefits of EBCLR on one of the large-scale datasets.

**Minor Comments:**
- In abstract, you used terms positive and negative pairs without introducing them. It’s okay for the readers who know about contrastive learning, but someone who is new to the domain wouldn’t get it.
- Missing appendix reference in line 132.
- Could you please add subscript $\textit(t)$ in Eq. 21 similar to update steps in Sec. 2.2, Eq. 5?
- Line 193, repetition of $\pi_{\theta}$.

[1] https://openreview.net/forum?id=XhF2VOMRHS
[2] http://yann.lecun.com/exdb/publis/pdf/ranzato-unsup-07.pdf
questions: - How do you initialise SGLD in the very first step when there are no previous iterations for generated samples (or replay buffer is empty)?
- I know the paper focuses on EBCLR’s representation learning capability, however, since there is a generative loss in the objective function, what’s the generative performance of the model? Does contrastive learning help it or deteriorate it?
- Given the scaling difficulties associated with EBMs, could the contrastive learning be paired with other generative loss to a similar effect?
limitations: The main limitations seem to be around experimental evaluation. I would be happy to increase my score if authors could demonstrate the benefits on one of the large scale dataset.

There doesn't seem to be any obvious negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new perspective to leverage contrastive self-supervised learning by combining energy-based models (EBMs). The code is to approximate the joint distribution of a pair of positive samples in contrastive learning by EBMs. Following the idea, the authors build a new contrastive learning method called EBCLR. To support the proposed method, the authors provide both theoretical and empirical evidence.
strengths_and_weaknesses: Strengths:
1. The authors provide both theoretical and empirical evidence to support their claim.
2. The experiment design is good, especially how it provides support for the efficiency of the proposed method given the limited number of negative pairs. Instead of the absolute performance advantage shown in Table 1, which is relatively not significant, the results in Table 3 are more impressive.
3. The paper is well written and easy to follow. It is well organized in most parts.
4. I would put it as a strength the limitation analysis by the authors which answers some of my concerns. 

Weaknesses:
1. I have some concerns about the idea of connecting EMBs and contrastive learning has been studied for long, even not necessarily under the framework created by recent deep-contrastive-self-supervised learning, such as MoCo and SimCLR. Some related works might be missing for the overview of this area, such as:
* A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training, ICLR 2022
* A Unified Energy-Based Framework for Unsupervised Learning, Artificial Intelligence and Statistics. PMLR, 2007.
2. Some parts of the paper is not well organized. For example, at the end of the Introduction section, the authors claim five Contributions, but not all of them are well organized in the experiment section and following discussions. Besides this, it would be not convincing to put some minor points about of the proposed EBCLR and independent contributions.
3. Some details should be polished, such as missing reference at L132 and the writing of Section 3, where some sentences are not fluent to follow. 
4. As the theoretical part in Section 3 mainly contains intuitive and very straightforward steps, I would recommend the authors to put some more efforts to backup the claim of contributions in following sections.

questions: I have put some of my concerns in the analysis of "Weaknesses" above. The clarification or improvements over them would be helpful for me to better justify the significance of this paper and to make my final rating more fairly and confident.
limitations: The authors have made the limitations analysis in the draft. I don't recognize the additional potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Summary:


The paper proposes an energy based contrastive learning framework that outperforms the existing contrastive learning methods such as SimCLR, MoCo, BYOL, SimSiam on four datasets, MNIST, FMNIST, CIFAR10, CIFAR100, using linear evaluations. In essence, the energy-based contrastive learning adds an energy-based generative term to the existing contrastive learning objective. Subsequently, the paper focusses on proposing relevant challenges to Stochastic Gradient Langevin Dynamics (SGLD) for improved representation learning. Overall, the paper performs some relevant experiments along with the ablation studies that indicate the usefulness of the EBCLR framework. However, the mathematical formulation has some obvious mistakes that the authors would need to clarify. Additionally, the paper is not well-written and requires proper narrative to better pitch the idea along with additional experiments.


Strengths:

- It was interesting to see that EBCLR boils down to standard contrastive learning when solved for the KL divergence between the joint distribution of the positive pairs modeled via EBM.
- The results of the proposed framework showcase that EBCLR outperforms all the existing methods on four datasets. Additionally, the authors show that EBCLR is more sample efficient as it achieves good performance with smaller batch sizes.
- The authors study the various aspects of the framework through ablation studies. 

Weaknesses and Suggestions:

- The abstract and introduction present this work with a motivation to reduce the batch size used for improved representation learning through contrastive methods. However, it seems that being able to perform well with small batch size is a byproduct of EBCLR framework. I don’t see any connection between the formulations discussed in the paper towards making EBCLR sample-efficient. Unless the authors present relevant works which have shown EBMs to be sample-efficient or prove it themselves, I don’t buy their motivation directly.
- My another major concern is with the core observation itself. Years of previous works has shown that more data should improve the performance more. A good representation learning method should be able to perform better when encountered with more data. Line 235 claims that EBCLR is invariant to batch size which to me is a double edged sword — good performance with low batch size is a good thing but not improving much with more batch size is not good.
- Equation 20 is wrong. It is calculated by solving Eq 12 and Eq 19. The authors cancel out the normalization terms from Eq 12 and Eq 19 but they are not actually equal! 
- Most of standard contrastive learning methods benchmark themselves on ImageNet dataset which paper does not do. The authors may use the latest FFCV library to fasten their ImageNet training pipeline.
- The paper uses linear probing accuracy to evaluate the model accuracies. To further understand the representation quality, the authors should perform K-NN classification in the representation space too.
- Further insights or discussions as to why EBCLR is more sample-efficient are definitely required.


Edits:
- Figure 1 left is not very clear. log 1/ p(v, v’) threw me off at the first place. You may need to explain it more.
- Eq.4 should have N(0, \sigma) 
- Line 193 has repetitions
strengths_and_weaknesses: Mentioned in the main review
questions: Mentioned in the main review
limitations: Mentioned in the main review
review_title: null

Meta-review: metareview: The paper connects contrastive learning and energy-based models and proposes a new variant of contrastive learning based on SGLD. All of the reviewers believe the paper is a good fit for NeurIPS, and I recommend acceptance. That said, as reviewers point out, results on ImageNet are also expected.
confidence: Certain
Decision: Accept