Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to improve the performance of energy-based models by incorporating latent variables to the model, assuming the positive samples of which come from an encoder optimized by contrastive representation learning. The latent variable EBM is parametrized in a way such that $p(x)$ and $p(z|x)$ can be easily separated. The performance can be further improved by updating the encoder with sampled $z$ from the EBM as auxiliary negative samples in CRL. Experimental results show it leads to state-of-the-art generative performance within EBMs. The learned model is also capable of conditional and compositional generation. 
strength_and_weaknesses: ### Strength
- The paper presents an interesting way to parametrize latent-variable EBMs, such that $p(x)$ and $p(z|x)$ are in closed-form. 
- Given the proposed model formulation, the paper proposes a wise way to combine it with contrastive representation learning. Results indicate that such combination can boost the performance of EBMs reasonably well.

### Weaknesses and questions
- The parametrization of $E_\theta(x) = \frac{1}{2} \|f_\theta(x)\|_2^2 + C$ seems to be a constrained set of functions. I'd like to see more discussions in terms of why it could serve as a general set for modeling energy functions. 
- I don't find information in the paper about whether the SimCLR encoder is pretrained or jointly estimated with the EBM starting from scratch. If it is pretrained, I'd like to see another ablation where the encoder is fixed as the pretrained one and only EBM is learned. 
- In the ablation study, all model have only been updated for 50k iterations. Do all the models really converge with such a small amount of iterations? 
- Given the proposed framework also provides a new objective for the encoder of contrastive representation learning, does it also lead to better learned features $z$ compared to the original CRL in terms of downstream tasks such as classification? 
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is clearly written and easy to follow. 

Quality: the formulation is technically sound to the best of my knowledge. Experimental results are convincing.

Novelty: the proposed jointly training method of EBM and CRL encoder is novel. The parametrization of latent-variable EBM is also novel.

Reproducibility: implementation details are adequately provided. 
summary_of_the_review: The paper proposes an interesting way for parametrizing latent-variable EBM and combining it with CRL. Experimental results are convincing that it is a promising approach to improving EBMs. It will be even strong if the paper could show the joint estimation method can also improve the learned features of the encoder.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to improve EBM training via contrastive representation learning (CRL).

In particular, it uses SimCLR framework to train the latent variable and implement latent-variable EBM with joint probability. The empirical results show that the proposed framework is more efficient and achieves lower FID scores and better OOD detection on visual datasets. 
strength_and_weaknesses: Strength:

The introduction of latent variable is important in statistical inference tasks. It is exciting to see that the latent variables help EBM in practice.
 
I appreciate that the endeavor of the authors to adapt EBM models to visual datasets, and the resulting generation quality and OOD detection have been improved over previous EBMs.


Weakness:

Although the introduction of latent variable is important, the CLEL just combine contrastive learning and EBM in a brute way, which is intuitive but incremental. What I am concerned about the most is the significance and novelty of this work, which is fully heuristic and direct in my opinion.  It is good to see that CLEL can improve image generation and OOD detection but I will really appreciate it if you can provide more insights about the “latent variables” rather than the downstream performance (which is also not strong enough). I must say that the paper can be significantly improved if it can provide enough evidence to show that what is the “latent variable learned by contrastive learning” and provide more statistical logics about the benefits.


The authors claim that the proposed model benefits from the introduction of true underlying latent variable. I believe that it is necessary to justify the why it is the true latent variable? Does it have explainable representations? Or can you prove that in what sense it is true.

I must say the generation quality is not the core issue of EBM’s training, thus I don’t think improving generation quality is significant enough, since we have many other choices.
clarity,_quality,_novelty_and_reproducibility: This paper is generally clear. The reproducibility can be guaranteed if the codes provided work properly.
About the novelty and quality, incorporating latent variable is important in statistical models. The authors have demonstrated a heuristic way to obtain the latent variable by contrastive learning, which shows the improvement in vision tasks (generation quality and OOD detection). Due to the heuristic motivation, it is better to provide more insights about the introduction of the latent variable beyond the downstream performance, which are better justifications for CLEL algorithms.
summary_of_the_review: In general, I think CLEL is a decent but incremental empirical work that verifies the benefits of contrastive latent variables in EBM training. The claims are justified by the downstream tasks and some ablation studies. But I do believe that there should be some results beyond the heuristic motivation and plain numerical results to justify this work so that it can be significant enough to convince the community. Thus, I encourage the authors to repackage the work and find out more merits about CLEL.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a latent variable energy-based model by introducing an energy form for the joint model p(x,z).
Intuitively, they augment data point x with z that comes from a separate encoder (that is trained using conservative representation learning) and train the joint energy model using contrastive divergence. They also use the z that minimizes E(x,z) as a negative sample to train the encoder.  
This setup enforces that the energy model assigns lower energy to the latent representation that is aligned with the external decoder. 

The authors show the capability of their methods using experiments on OOD detection, conditional and unconditional generation, and compositional generation. 
strength_and_weaknesses: The idea is novel and experimentally they show a great performance (w.r.t. to FID score) on CIFAR-10.  Although the performance is not better than the SOT in the field, it is impressive regarding other EBMs training methods. There are many applications in that EBMs have an advantage over other probabilistic methods such as score models, so better EBM training algorithms are important to the community.

The main weakness of the paper is that the authors didn't justify the joint energy model that they are using. 
For example, they say that "the remaining information in $\|| f_\theta(x)\||_2$ is used for modeling $p_\theta(x)$. What do you mean by that?
or when you say that "we separate f into direction and norm and we found that this separation reduces the conflict between $p_\theta$ and p\theta(z|x) optimization" what exactly do you mean? Are the direction of the gradient inconsistent? What else did you try?
Why does only a particular form of g work well? Why does using the identity function as g drastically reduce the performance of the model? The FID score when using the identity function is twice higher as using beta=0!

In Table 5, setting b does not use the negative example from p_theta for training the encoder. Does this mean that one can use a pre-trained encoder and distill the representation knowledge into E(x,z)? Have you tried something like that?


Typo: "into our EMB training"
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the idea is novel. 
summary_of_the_review: The paper is relevant to the community, novel, and shows an interesting connection between SSL representation learning and EBM training. 
The experimental results are strong and the authors did an extensive analysis of their methods. The main weakness is the lack of sufficient discussion regarding the used energy form and ablation study. For example, the authors show that the identity function is not a useful projector but not discussing why?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a very neat idea of embedding contrastive learning into energy-based model learning. On top of the normal latent-variable energy-based models, the authors introduce augmentation and contrastive representation learning to regularize the latent space. The general idea is to still keep the empirical KL divergence, regularized by energy regularization terms. Besides, this work adds contrastive learning, inspired by SimCLR, on the latent space modeling. The experiments give both qualitative and quantitative results. This work can generally outperform other energy or autoencoder-based methods.
strength_and_weaknesses: Pros:
1. This is a very interesting idea to embed contrastive learning into EBM. Indeed, contrastive learning has been widely used in a lot of scenarios. Employing it in EBM is quite novel.
2. In experiments, the paper demonstrates unconditional generation, conditional generation, out-of-sampling detection and conditional sampling. It's thus convincing that this work is versatile, and potentially can be used in many scenarios.

Cons:
1. Could you further explain the P in Eq. 9?
2. The directional projector lacks explanation or intuitions. It's better to explain the motivation and some insights behind this module.
3. I think one thing you claim in your paper is that your method can make the EBM training more stable. Maybe you can give more examples to illustrate that.
4. Some numbers are still worse than GAN-based numbers. Do you have any insights or ideas?
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and I think the structure starting with preliminaries makes the reading much easier. The authors provided the code and it looks good to me. Again, though the paper sticks to the raw EBM framework, the contrastive module is embedded neatly. I think this could potentially benefit the area.
summary_of_the_review: This is a good idea with many experimental validations. Some claims need to be further supported. And the gap with other type of models should be better explained. Some modules should be expanded to further facilitate understanding.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method to train latent EBMs using contrastive learning. The empirical results show that the proposed framework is more efficient and achieves lower FID scores and better OOD detection on visual datasets. 
The reviewers agree that the method is novel and could has significant implications for generative modeling. 

Reviewer nYLf points out a lack of theoretical insight about the approach. I agree that additional theoretical motivation for why the approach would work would be valuable. It would be also interesting to investigate what contrastive learning brings to the latent representation compared to the latent representation learned by a standard latent EBM.


However, I think that it is still interesting to see that CL can be used to train EBMs. CL incorporates additional information (through augmentation) which is not present in standard EBM training. Showing this additional signal to be beneficial empirically is still valuable.

For these reasons, and from the other reviewers' comments, I'm inclined to accept the paper for this empirical insight and to recommend adding more discussions about the limitations/ the open questions that are not addressed by the present work as suggested by Reviewer nYLf.






justification_for_why_not_higher_score: There are still some clarifications to be added about the theoretical motivations for using CL to train EBMs as pointed out by Reviewer nYLf.
justification_for_why_not_lower_score: Using CL in the context of generative modeling is novel and can have significant implications given the interest of the high community in both SSL and generative modeling.
Decision: Accept: notable-top-25%