Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: AR models are very common in many domains, especially language models. Maximum likelihood is the widely used approach for training these models. However, ML training of AR models causes some other issues such as exposure bias. Techniques such as scheduled sampling have been introduced before to address these issues. 
This paper introduces a new training algorithm for AR models by viewing it as an energy-based model.  
Here the authors define the model probability as $p(x_1, \cdots, x_k) = q(x_{<k}) \frac{e^{-Q(x_k, x_{<k})}}{Z_\theta}$. They defined $Q(x_k, x_{<k})$ using the logits before the softmax of the $k$th token, so $p(x_1, \cdots, x_k) = q(x_i, \cdots, x_k)$ (am I right?). 
However instead of ML training which normalize $e^{-Q(x_k, x<k)}$ with $Z_\theta$, they are training $p(x_i, \cdots, x_k)$ using contrastive divergence (CD). This formulation helps to better regulate the distribution on the $k$th softmax since CD training contrast $Q(x_k, x_{<k})$ with a sample from the model (using importance sampling given the introduced form for the energy) rather than the log partition function. 
The final loss function is a combination of cross entropy and CD.

The authors study the proposed method for different tasks of language modeling, machine translation, as well as image generation.


strength_and_weaknesses: 
Strength:
-The paper is well-written and easy to follow.
-The idea of viewing AR models as EBMs is not novel but the provided formulation is neat and interesting. 
-I like the extensive study of the proposed approach.

Weakness:
The improvement especially in the translation tasks is not considerable. Other training algorithms on top of ML training of AR can achieve similar improvement, for example, combining it with RL training. 
Bhattacharyya et al. (2021) reported a significant improvement over base AR-NMT by using an external EBM defined over the whole sentence (although some of the improvements come from using pre-trained language models).  
Also, their training algorithm can directly be applied to AR by viewing E as \sum_i E_i. 

Questions:
1) The author claims this approach can help with long-range dependencies, but I am not seeing how defining the energy model using the last softmax can help with that. Is that the effect of weights on training log q(x<k)? Even so, it is not similar to defining an explicit energy model over the whole sentence! 

Bhattacharyya S, Rooshenas A, Naskar S, Sun S, Iyyer M, McCallum A. Energy-based reranking: Improving neural machine translation using energy-based models. ACL 2021.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The proposed method is novel although some parts such as using importance sampling for estimating partition function have been introduced before for similar joint distribution form: p(x) = q(x) exp (-E(x)).



summary_of_the_review: The paper is interesting but the experimental results do not show considerable benefits for the approach. I don't think that the proposed training algorithm would be adopted by the community for training AR models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to use the softmax logits in the output of discrete autoregressive models as an energy function, converting an autoregressive model into an energy-based model. They then use contrastive divergence training, using an importance weighted estimate of the “negative phase” gradient. Through experiments on language modeling and translation tasks, they show that this results in more coherent and in-distribution samples, as compared with autoregressive baselines.
strength_and_weaknesses: **Strengths**

- **Relatively simple/general technique.** The basic idea of the proposed technique is to identify the softmax logits of an autoregressive model as an energy function, allowing them to replace the typical maximum log-likelihood training procedure with a contrastive-divergence-based training procedure. This is reminiscent of previous methods that have interpreted classifiers as energy-based models (e.g., Grathwohl et al., 2020). In this way, the model is explicitly trained by decreasing the likelihood of generated samples, hopefully reducing the effect of “exposure bias.” Notably, this procedure can, in principle, be applied to any discrete autoregressive model, making this a rather general technique.

- **Experiments on multiple datasets with relevant baselines.** The authors demonstrate their proposed approach on language modeling and translation tasks, with three language modeling datasets and one translation dataset with six translation pairs. The authors compare their approach with the baseline autoregressive model, as well as Residual EBM, another method that combines autoregressive and energy-based models. The proposed approach generally compares favorably.

- **Clear description, for the most part.** The descriptions within the paper are generally clear, with both text and mathematical terms well-defined. One way to improve the clarity of the paper even further would be to include some form of diagram.


**Weaknesses**

- **Additional computational overhead for sampling.** One downside of employing a contrastive divergence-based training method is that it requires sampling during training. Sampling from autoregressive models is typically quite costly, and I would guess that this significantly impacts training speed (although this isn’t discussed in the main paper).

- **Only applicable (at least currently) to discrete models.** The current method, as presented, is only applicable to discrete autoregressive models. While this encompasses a wide variety of applications, the authors have not demonstrated how to apply this technique to continuous autoregressive models, e.g., those that output Gaussian densities. In this sense, the title is a bit misleading, as the technique does not apply to all ‘autoregressive models.’

- **Only demonstrated on text data, i.e., no images or audio, etc.** The authors only demonstrate their technique on text data (language modeling and translation, i.e., conditional language modeling). However, even with discrete autoregressive models, this technique could be readily applied to image and perhaps audio modeling. Currently, the paper may only appeal to a subset of the machine learning community focused on language, but with these added application areas, the paper may receive wider attention.
clarity,_quality,_novelty_and_reproducibility: **Clarity** — The paper is generally clear, with the basic aspects of the approach well-described in mathematical expressions / equations. I appreciated the algorithm box, which clarified the training procedure. Results tables are also well-formatted and explained in the text.

**Quality** — The paper is fairly high-quality. The authors take a set of principled ideas stemming from energy-based models and apply them to autoregressive models to solve some well-known issues with the latter. Experiments on multiple datasets with relevant baselines demonstrate that the approach is beneficial, and further experiments demonstrate improvements with respect to exposure bias and coherence. In total, the paper tackles known challenges using principled ideas and clearly demonstrates an improvement in relevant empirical settings.

**Novelty** — The paper is somewhat novel. While many of the main ideas (treating non-energy-based models as energy-based models, contrastive divergence, importance weighting) are established to varying degrees, combining these components to train autoregressive models is a novel contribution, as far as I am aware.

**Reproducibility** — I am unsure of the reproducibility of the results, but I suspect that they are reproducible. The authors demonstrate their proposed approach on multiple datasets in various settings, although it’s unclear whether multiple random seeds are used. Generally, their proposed approach outperforms the relevant baselines, with more targeted analyses demonstrating these benefits. While I am unsure of how difficult it would be to reproduce these results, their consistency leads me to believe that they are.
summary_of_the_review: The paper presents a fairly general and sound technique for improving autoregressive models by training them as energy-based models. Results are demonstrated on several text datasets, comparing with relevant baselines. In total, the generality of the idea, combined with the paper’s convincing empirical demonstration, leads me to suggest acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for training autoregressive generative models in a way that casts these models into Energy-Based models. In essence, the method uses the logits from the Categorical output distribution of an autoregressive model to define an energy function; this is the basis for the subsequent treatment. As maximization of the energy function entails a contrastive divergence type of sampling, which may prove ineffective, the authors provide an importance sampling-based variant. 

strength_and_weaknesses: Pros: Interesting idea, extensive experimental evaluation against the standard baseline, correct derivations. 
Cons: The method is a minor extension upon existing techniques; the authors do not explain how more computationally demanding this method is compared to the baselines, and whether these extra costs are worth the extra accuracy. 
clarity,_quality,_novelty_and_reproducibility: The writing is clear and of good quality. 
The novelty is mediocre, but the method could be useful.
The experiments are reproducible quite well.
summary_of_the_review: The method is interesting, and the recasting of the categorical output distribution into the potential function of an energy-based model is nice. The training algorithm entails contributions of some merit. 

On the other hand, the experiments are not convincing at all; they do not elaborate on computational costs required for the extra accuracy, and do not perform an ablation study with different architecture choices which may affect performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is about using concepts from energy-based models to improve the training of auto-regressive generative models. Specifically, an energy-based model is defined in eq 3 for the joint distribution up to step k written as a product of the auto-regressive model up step k-1 times a Boltzmann term divided by a normalizer. The learning objective is the likelihood plus the KL between the energy based and auto-regressive model.

Promising results on language modeling and machine translation are presented.

(For this AC, it is not super clear why this approach per se should solve fundamental problems with purely auto-regressive modeling. The empirical results support this so as many other things in DL one perhaps just has to accept that.)

The reviewers are a bit mixed but overall leaning towards accept. The mixed opinions could possible have been resolved in a meeting, but unfortunately due to a too tight program this meeting was not organised. This AC's fault.

This is a bordeline paper and one can argue either way for accept or reject. Since the reviews are mixed and the average score is below the typical for accepted papers, rejection is recommended. The reviews are constructive and can help improve the paper for the next conference.
justification_for_why_not_higher_score: The novelty is not high enough.
justification_for_why_not_lower_score: The paper is properly done and shows merit of the proposed method. One can always argue about the significanceof the results - will they carry over to LLMs for example, but this is for future research to find out.
Decision: Reject