Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes E-ARM, aiming at integrating EBM into ARGM seamlessly in order to tackle two major existing issues on sequence modeling, i.e., exposure bias and incoherence problems. At each step, the joint distribution $p(x_k, x_{<k}$ is optimized by not only autoregressive cross-entropy loss, but also the EBM KL divergence. Moreover, the model uses importance sampling to bypass the MCMC as conventional EBM models do. Experiments on neural machine translation, language modeling and image samling all demonstrate that the proposed integration outperforms the base model.
main_review: Strength:
+ This paper is well-written and well-organized. Issues on conventional ARGMs are clearly pointed out, which lead to the motivation of this work.
+ Three experiments show the overall improvement of E-ARM compared with based ARGM.

Weakness:
- Training ARGM with EBM in a cooperative manner has early been proposed by
Learning Latent Space Energy-Based Prior Model, NeurIPS 2020
Cooperative Training of Descriptor and Generator Networks, TPAMI 2018
Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning, TPAMI 2021
These works, however, are essentially missed from related works and model comparisons.
- The experiments of this work are relatively insufficient.
  1. All experiments only demonstrate improvements compared with base model, while comparisons with other baseline models are missing. 
  2. Only basic evaluation metrics are used. Some other metrics, e.g. ROGUE, METEOR for NMT, FID for image generation, should also be listed.

summary_of_the_review: This work proposes to alleviate ARGM existing problems by leveraging the EBM objective. Such integration, however, is not essentially new and this work does not demonstrate its superiority compared with other methods.  Despite three experiments performed in this work, the evaluation and analysis are insufficient.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to improve the training of AR generative models by introducing a "regularizer" based on a decently designed energy-based model. The EBM is seamlessly integrated with the AR model and the sampling from the EBM can be transformed as an importance sampling from the AR model. Empirical results on text and image datasets verify the effectiveness of the proposed model. 
main_review: The paper is well-written and easy to follow. The results look promising and the idea of using EBM guidance in AR model training is interesting. I have some concerns as follows:

* Although the importance sampling avoids MCMC from the EBM, it does require sampling from $\tilde{q}(x_{<k})$ which goes through the autoregressive generation process and could be time-consuming. Would be informative if you could report FLOP in the paper. However, I do feel that it is way more complicated than a simple regularizer added on to the training of the autoregressive model.

* I still doubt that high variance is induced in the importance sampling as it is not clear how close $p_\theta$ and $\tilde{q}_\theta$ and another Monte Carlo estimate is required when computing the importance weight $w$. Do you observe training instability during training? Would be better to provide more analysis on this point.

* It would be more informative if you could report the results before the Top-K correcting by EBM in the inference stage, as it would clearly show if the AR model really benefits from the joint training.

* In the paper, it is claimed that the joint training improves the long-range coherence. Is there a metric to show this point empirically? 

* From table 4, it looks like the optimal solution requires a very small weight of the EBM KL. For what reason that a larger weight harms the performance? Theoretically, it should work with even only the EBM KL.

Minor: 
- Typo for "The first term *** in Eq. 7 is equivalent to ..." in the line after Eq.8: should be $-\mathbb{E}_{x<k \sim p_d}$
- Here is a related work of using NCE to learn EBM that you should consider to discuss or cite in the related work: 
"Flow Contrastive Estimation of Energy-Based Models", CVPR 20. 

summary_of_the_review: Overall speaking the paper is well-written and well-motivated. The direction of joint training of AR model and EBM is promising. However, I'm sort of concerned with the computational cost and training instability introduced by the extra EBM module, compared to the performance gain it brings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper designs a new globally normalized probability function to approximate the sequence data distribution, which is defined as the product of traditional autoregressive model distribution and energy-based model distribution. The authors also propose a new training objective that is different from maximizing log-likelihood. This new objective is composed of two KL divergence terms. To tackle the optimization issue due to the energy-based loss term, the authors propose to use importance sampling to sample from the autoregressive model to approximate the negative phase loss, so that MCMC is not required. Certain training tricks are required (e.g. pretraining the autoregressive model) to stabilize training.  Experiments on machine translation, language modeling, and image generation demonstrate the effectiveness of the proposed method.
main_review: Below are the detailed strengths and weaknesses (or questions):

*Strengths*:

1. The proposed method is interesting and novel. It is also well-motivated as a new energy-based model that does not require MCMC during training.  
2. The experimental results are generally good.


*Weaknesses*:

1. The proposed method only compares with vanilla autoregressive models but not other energy-based model baselines. For example, how would the method compare to (Deng et al. 2020)?  Deng et al. do not use MCMC either, thus an experimental comparison with them is important.   
2. One main merit of E-ARM is its removal of the “extremely time-consuming” MCMC process, however, this merit is not supported with experimental results in this paper. I feel E-ARM can be time-consuming due to sampling from autoregressive models in Eq. 11, thus a comparison with MCMC-based baselines is preferred. In addition to efficiency, performance comparison with energy-based model baselines is also important, otherwise, I am not sure about the advantage of E-ARM compared to other energy-based methods.   
3. I was confused when the authors suddenly note that the loss is summing over length K (Eq. 11). I think this point should be made earlier for clarity. For example, the sum over K should be added to Eq. 5 for correctness, right?  
4. Some experimental details are missing:   
(1) how many samples are used to approximate the expectations in Eq. 11?  
(2) how many samples are used to approximate the weight w?  
(3) do (1) and (2) share the same samples for approximation or need sampling twice?  
(4) Eq. 11 is summing over K and for each k there are expectations to be approximated. This process seems very time-consuming if resampling for every k. Please clarify these details.  

*Questions for the author:*

1. Is the positive phase loss in Eq. 12 also summing over K?  
2. The end of page 4 says “xx means improvements in this direction will be automatically taken care of as a result of xxx”. This sentence confuses readers – I am not sure whether the first term in Eq. 7 is actually included during training or not, please clarify.  
3. Appendix C says "for language modeling tasks, we fix the generation length as 50", why is generation required for perplexity computation?

summary_of_the_review: The proposed method is novel and interesting, but it only compares to traditional autoregressive models in experiments, and other energy-based model baselines are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach to integrate energy based models (EBM) into into general auto-regressive generative models ARGM. They propose to combine the negative log-likelihood loss of auto-regressive models with a second energy based loss, which serves as a kind of regularisation loss (steered with a hyper-parameter lambda). They claim that this approach mitigates the Exposure Bias which is frequently found in ARGM. A learning algorithm based on Hinton's wake-sleep algorithm is derived that approximates the EBM part of the loss with importance sampling thus avoiding expensive MCMC approximations.
Experiments form the NLP and the image generation domain are presented that show outperformance over selected baselines.  

main_review: The approach is interesting and new (to the best of my knowledge). It is motivated by the exposure bias or in other terms *"a discrepancy of the input context distributions between the training and inference stages"*, which ARGM are reported to suffer from. 
Accordingly, the experiments show improved performance when combining several ARGM with the proposed E-ARM method. 

That alone, however, does not yet make strong paper (rating 8 or higher) in my opinion, because the authors don't study the effect of their method in detail. Instead, they confine themselves to just reporting improved performance on selected, and not exactly the latest benchmarks (e.g. PixelCNN is from 2016).
What I would wish in order to increase my rating are
* experiments that reveal when and how exposure bias is actually a problem (e.g. depending on the model expressiveness)
* experiments that show that this bias is indeed decreased when using E-ARM
* comprehensive studies on the cost this decreased bias (=improved performance) comes with 

Being the devil's advocate, my suspicion could be that the improved performance is not actually due to $p_\theta$ (Eq. 4) but due to the heavily truncated back propagation in the second term of (5). A small footnote tells that BP is just two time steps back. The fact that this seems to be crucial for the method requires further investigation, and would make the paper much stronger. Besides, it would be easy to check by just using $\tilde q_\theta$ instead of $p_\theta$ in (5).     

The paper was ok to follow, but can still be improved in writing in the sense that the mathematical derivations are sometimes hard to follow. 
A lot of the separated equations contain just standalone terms instead of equations. Their references and the relation to them have to be found in the text, which hinders readability. 

Eq. 5 is misleading, although the intention of it might be clear. The optimisation is over a sum over all $k$  and under the expectation over $\mathbf x\sim p_d$. That should be made explicit for better understanding. 

The authors mention density estimation a couple of times, although the proposed method is clearly dealing with discrete distributions. In particular (footnote 1 would be wrong for densities).

Sec. 2.1, first line: $\mathbf x\in\mathbb R^K$, to be consistent with what follows

S2.3: *"Besides,autoregressive decoding typically greedily selects the most probable token at each time step, given
the ones previously selected.”*
I would disagree here. What you usually do is sampling the next token. That greedily decoding to the conditional mode is far from being ideal  is well known, and is by no means a shortcoming of autoregressive models. 

summary_of_the_review: An interesting idea that can lead to improved performance on certain benchmarks. However, deeper insights about the effect that the model addresses (exposure bias mitigation) as well as estimations of the increased overhead are missing in the experiments, thus limiting the scope of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of exposure bias and lack of long-range coherence in auto-regressive models. Sampling in such models is sequential, so the error would propagate through ancestral sampling. The authors propose to combine autoregressive models with EBM, which does not require the assumption of distribution factorization. Existing works use a two-stage optimization, where an EBM is trained on top of the auto-regressive models. The authors instead jointly minimize an energy-based learning objective and the vanilla MLE. They also present a way of estimating the gradient of negative samples through importance sampling. Extensive experiments on NLP and vision tasks show that the proposed learning objective reduces the discrepancy between training and inference.
main_review:  # Strong points

- The considered problem is important, and the authors' novel framing of it is well-motivated: using an EBM model that takes the global information into account to mitigate the gap between training and inference.

- A simple algorithm is derived under the defined joint distribution. They use the model distribution as the importance distribution, and It's faster to estimate the weights in the importance sampling than MCMC procedures in traditional EBM.

- Experiments are designed appropriately to showcase the resulting behavior of the model. The proposed method outperforms the base autoregressive models, and is compatible with other techniques and architectures.


# Weak points

- The design of the joint distribution seems a bit arbitrary. Could the author better explain the rationale behind the design?  For example, there is large flexibility for $\phi$ in Eq (4). If $\phi=-\log q(x_{\le k})$, then the importance weight is $w(x_{<k}) = \frac{q(x_{< k})}{E[q(x_{< k})]}$. It seems to me that this choice is a more intuitive one.  In addition, the joint distribution is essentially defined by conditional distributions. It seems to me that ``we compel the ARGM to fit not only the conditional distributions but also the joint distribution at each time step" overclaims the actual modeling of the joint distribution.

- Lack of comparison/discussion of relevant works. For exposure bias:  [1] adopt a discriminator as the energy-based model to correct the ancestral sampling in every step. For long-range dependency: [2] instead learns a global ordered representation, like PCA, and trains autoregressive models on top of the representation. ([2] would still incur exposure bias). The paper also does not compare to those ``two stages" methods in intro. 
 
# Minors

- It seems that there should be a $\sum_k$ in the RHS of Eq. (5). 

[1] Yuntian Deng and Anton Bakhtin and Myle Ott and Arthur D. Szlam, Residual Energy-Based Models for Text Generation, abs/2004.11714

[2] Yilun Xu and Yang Song and Sahaj Garg and Linyuan Gong and Rui Shu and Aditya Grover and Stefano Ermon, Anytime Sampling for Autoregressive Models via Ordered Autoencoding, abs/2102.11495
summary_of_the_review: There are some loose threads have been left by the authors. I would increase the scores if the authors address some of the points above.
review_title: null

Meta-review: This paper proposes a method to train autoregressive model that takes advantage of a well-designed energy-based learning objective model. With the importance sampling, the model can be trained efficiently without requiring an MCMC sampling. Experiments are conducted to verify the effectiveness of the proposed method.  The idea is interesting and well-motivated, but the experiments need to be improved. Reviewer FnWE’s major concerns include limited novelty, lack of discussion with closely related works, and insufficient experiments, and recommend rejecting the paper by assigning a rating of 3. Rebuttal doesn’t address his/her concerns. Reviewer in11 is concerned with the computational cost and training instability due to the extra EBM module and has a few unclear technical details that need to be clarified. The author’s reply along with additional experiments during rebuttal partially addresses the concerns of Reviewer in11, who eventually increases the rating to 6. Reviewer AQxn’s major concern is also about the lack of sufficient comparison with other relevant energy-based models.  Reviewer DZsJ pointed out that the more insightful analysis about the model is missing in the experiments. Even though the authors provide additional experiments for Reviewer DZsJ, they are not satisfied with the feedback because the additional results are not supportive of the claims made in the paper, and end up with a rating of 6. Reviewer SjXn’s concerns include the lack of comparison with relevant works and the unclear motivation of the design of the joint distribution. After the rebuttal, Reviewer SjXn’s concerns remain and assign a rating of 5 to the paper. The overall rating of the paper after rebuttal is marginally below the acceptance rate. Even though this paper proposes an interesting idea, the reviewers’ comments are not well addressed. As a result, AC cannot recommend accepting the paper.  The AC urges the authors to revise their paper according to the comments from the reviewers, and resubmit their work in a future venue.
Decision: Reject