Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper justifies why models trained with adversarial training are good generative models by proposing a probabilistic framework. The paper is heavily influenced by the JEM paper and proposes a generalized form of analysis in that paper to include the unsupervised scenario. The generalization defines a joint distribution $p(x, z)$ over data $x$ and representations $z$. For the supervised case, each class is represented by a "center" $w_y$ and for the unsupervised one, $p(x, z) = p(x) p(z|x)$ and $p(z|x)$ is reparametrized with augmentation.

Then the authors use this probabilistic framework to analyze the adversarial training. Specifically, they show PGD is a biased form of the Langevin dynamics in their formulation. Then they show how the maximum likelihood training of their probabilistic framework is similar to adversarial training. As a result, they conclude adversarial training leads to a model with high generative power. Finally, they propose refined sampling strategies from adversarially trained models.

In the last part of the paper, they use their framework to propose an unsupervised adversarial training method followed by sampling algorithms from the produced models.
main_review: The paper is technically sound and brings a novel perspective to adversarial training. The paper is the first to propose sampling from an unsupervised adversarially trained model. The technical aspects of the paper are overall very impressive. 

The experiment results support their claims well. The reported FID and IS of their method is also pretty good. The qualitative sample quality in Figure 1 and appendix seems worse than the JEM paper although they're getting a higher FID. 

There are a few minor typos:
In related works, the second line: while standard classifiers cannot?
In section 3, it is the first time the abbreviation EBM is appearing.
In page 8 there is a Table ?? in the paragraph before Comparison with other generative models


summary_of_the_review: Besides the few minor typos, I think the paper meets the quality bar for ICLR. The technical analysis has some valuable novel aspects that justify well why adversarially trained models are good generative models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a unified probabilistic framework, dubbed as Contrastive Energy-based Models, to understand the robustness and generative capability. The proposed CEM is a special case of EBM that models the joint distribution over two variables with a similarity function defined in a contrastive form. CEM could be instantiated into parametric form and non-parametric form, which work for supervised learning and unsupervised learning respectively. CEM could demystify adversarial training's generative capability in both supervised and unsupervised setting. Moreover, with CEM, adversarial training could be extended to unsupervised scenario.
main_review: This paper has 4 major contributions.
1. The proposed Contrastive Energy-based Models (CEM) gives a probabilistic interpretation for adversarial training, which explains the generative ability of adversarial trained models.
2. CEM's parametric and non-parametric form give probabilistic understanding of previous ST and AT in both supervised and unsupervised settings.
3. Under CEM, the equivalence between the IS and the InfoNCE loss of contrastive learning is established, which enables us to design principled adversarial sampling for unsupervised learning.
4. Experiments show that adversarial sampling methods derived under CEM outperforms or achieve comparable performance to SOTA.

However, more experiments using different models, different dataset and different attacks are needed.
summary_of_the_review: The proposed unified probabilistic framework CEM is fundamentally novel and interesting. It gives explanation on the generative capability of adversarial trained model. It also gives probabilistic understanding of ST and AT in both supervised and unsupervised learnings. The derived adversarial sampling method show a better sample quality. However, more experiments are needed to further justify the correctness of the proposed CEM, given that there are a lot of approximation in the derivations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission proposes a unified probabilistic framework to illustrate the generative capability of adversarial training. It also offers a unified perspective of adversarial training and sampling from both a supervised learning setting and an unsupervised learning setting. The proposed adversarial sampling strategy from the method is extensively demonstrated on different benchmarks, showing better quality compared with existing related works. 
main_review: The contribution is a unified probabilistic framework to explain adversarial learning, however, it is not clear whether this probabilistic framework can be beneficial for improving the training of adversarial models. 

The sampling strategy is not claimed as a contribution of the proposed work, however, the experiment part spends a lot space to demonstrate that the sampling in the proposed framework is better than existing works. The authors need to more clearly elaborate the contributions. 

The proposed probabilistic framework is generic, and thus should be able to work with different large-scale and large-resolution datasets. The authors only show results on a relatively small-scale and low-resolution dataset. The results presented are not ideal to explain the effectiveness of the proposed framework. 




summary_of_the_review: The problem studied is interesting, however, more investigation should be conducted to show the effectiveness and importance of the proposed probabilistic framework. Please also check the weakness part for more explanation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: By combining generative models and discriminative models in the framework of an energy-based model, this paper provides several theoretical insights about adversarial training, including that the softmax normalization term already provides a hint on adversarial training, and the importance sampling (with a justification) approach to unsupervised contrastive learning. The experiments show that their strategy with Langevin sampling improves over simpler adversarial training strategies.

main_review: In general, I like the direction in this paper. By formulating it as an energy-based model, we can look more into the normalization term of the softmax which is quite often overlooked. The connection drawn between Langevin dynamics and adversarial training is nice and new at least to me. I think the insight in contrastive learning is quite interesting. Although the so-called "hard negative mining" is an old idea in the computer vision community, it's not generally been casted in a theoretical framework. I enjoyed reading the paper which is not quite often (anymore) for my review batches in the ML conferences.

There is this small confusion in the notation that p_theta(x,y) defined in eq. (2) seems to be used equivalently with p_theta(y|x), it's better to clarify which part of f_theta(x,y) was attributed to the prior p(x), or my understanding would be eq.(2) should rather be a definition of p_theta(y|x) (where it doesn't take into account the prior).

I'm also a little bit confused initially about the sign between eq. (12) and eq. (7). In eq.(7), the goal for the minimization step is MLE, which means to minimize - logp (or equivalently, maximize log p). In eq. (12), we are talking about the AT loss being the same as negative gradient of the loss in eq.(7), which is correct, but in order to match the sign w.r.t. the MLE in eq. (7), seems that we should be running gradient ascent. This is clear in the end from the appendix, but it might worth mentioning a single word in the main paper as well.

The results are also very competitive, I think the unsupervised contrastive results are especially quite strong. It would be nice to have some additional technical explanations why the methods of moments approaches would work better than this current approach, rather than only a descriptive explanation in the current manuscript (although that is nice to have as well).

In terms of the results, it is interesting to see unsupervised Langevin performing better than supervised Langevin method. Any insights in that?

Although the paper focused on generative models, I also would be wondering whether the importance sampling approach might have consequences for discriminative models as well (the more common application in computer vision). It might be nice to show that as well since it shouldn't be a lot of work.

Nvasconcelos should be Vasconcelos.
summary_of_the_review: Overall, I like the insights provided in the paper and their experiments also proved their points well. Solid accept.
review_title: null

Meta-review: This paper presents a probabilistic framework that explains why models trained adversarially are robust generators. It received fairly high initial scores. The reviewers thought the work was novel and interesting. They liked that the analysis provided a way to derive a novel training method and sampling algorithms. Reviewers confirmed their support of acceptance and I think this paper is clearly above the bar. Respectfully, I’d prefer that the authors don’t ask the reviewers to “raise your score”. It is up to the reviewers to make that decision.
Decision: Accept (Poster)