E-Forcing: Improving Autoregressive Models by Treating it as an Energy-Based One
Authors: Yezhen Wang
Tong Che
Bo Li
Kaitao Song
Hengzhi Pei
Yoshua Bengio
Dongsheng Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: autoregressive models
exposure bias
language modeling
neural machine translation

Abstract: Autoregressive generative models are commonly used to solve tasks involving sequential data. They have, however, been plagued by a slew of inherent flaws due to the intrinsic characteristics of chain-style conditional modeling (e.g., exposure bias or lack of long-range coherence), severely limiting their ability to model  distributions properly. In this paper, we propose a unique method termed E-Forcing for training autoregressive generative models that takes advantage of a well-designed energy-based learning objective. By leveraging the extra degree of freedom of the softmax operation, we are allowed to make the autoregressive model itself an energy-based model for measuring the likelihood of input without introducing any extra parameters. Furthermore, we show that with the help of E-Forcing, we can alleviate the above flaws for autoregressive models. Extensive empirical results, covering numerous benchmarks demonstrate the effectiveness of the proposed approach.

Under review as a conference paper at ICLR 2023
E-FORCING: IMPROVING AUTOREGRESSIVE MODELS BY TREATING IT AS AN ENERGY-BASED ONE
Anonymous authors Paper under double-blind review
ABSTRACT
Autoregressive generative models are commonly used to solve tasks involving sequential data. They have, however, been plagued by a slew inherent flaws due to the intrinsic characteristics of chain-style conditional modeling (e.g., exposure bias or lack of long-range coherence), severely limiting their ability to model distributions properly. In this paper, we propose unique method termed E- Forcing for training autoregressive generative models that takes advantage of a well-designed energy-
learning objective. By leveraging the extra degree of freedom of the softmax operation, we are allowed to make the autoregressive model itself an energy- based model for measuring the likelihood of input without introducing any extra parameters. Furthermore, we show that with the help of E-Forcing, we can alleviate the above flaws for autoregressive models. Extensive empirical results, covering numerous benchmarks demonstrate the effectiveness of the proposed approach.
INTRODUCTION
By factorizing the joint distribution into the product of a series of conditional distributions, autore- gressive generative models (abbr. ARGMs) (Vaswani et al.) 2017]] Dai et al., 2019 van den Oord
let al. 2016a| b Salimans et al. 2017 Chen al. 2018)) simplify the difficult challenge of modeling high-dimensional joint distributions. They can be trained efficiently via maximum likelihood and
generate samples of exceptional quality, making this technique popular for modeling distributions,
especially for sequential data. Nonetheless, despite their potency and flexibility, and huge success, ARGMs still have inherent weaknesses due to the intrinsic characteristics of chain-style conditional modeling, especially when the training data is less diverse For example, ARGMs usually suffer from a discrepancy distributions of input contexts between the training and inference stages, which causes a consequent performance drop, i.e., Exposure Bias (Ranzato et al., 2016) [Bengio et al.) 2015). Besides, due to the nature of the greedy selection of beam search approximations, the decoded results from ARGMs may also lack long- longerange coherence (Deng et al. 2020).
Earlier work, both heuristic and theoretical, has been proposed to address these concerns. For instance, the exposure bias problem of ARGMs can be alleviated to some extent with scheduled sampling (Bengio et al. 2015; Mihaylova & Martins 2019), by mixing input contexts from both real data and autoregressive generation, during the training stage. However, this scheme introduces some new problems like the over-correctigg (Zhang et al. 2019) issue. In addition, at the inference stage, sampling methods such as beam search is employed to generate diverse candidates with high likelihoods, improving the quality of generated sequences. Nevertheless, these approaches result in only marginal improvements temporal coherence.
In this paper, we propose an elegant solution, i.e., E-Forcing, for the above problems of ARGMs by leveraging deep connection between ARGMs and Energy-based models (EBMs). EBMs are popular class of generative models that have demonstrated their effectiveness in modeling high-dimensional distributions in variety of machine learning applications, without requiring the transformation of the target distribution into a product of conditional distributions (Zhao et al. 2017
'When trained on massive datasets under which the underlying distribution is diverse enough, such as in large language models, this problem can he relieved because the training data covers lot of comer cases making the model much harder to go off distribution.

---

Under review as a conference paper at ICLR 2023
Arbel et al., 2021: Gao et al. ).21). As result, several studies (Deng et al.. 2020 Bakhtin et al. 2021: Durkan & Nash 2019) have made their attempts to benefit ARGMs from the advantages of EBMs. However, though some positive results were obtained, the existing works preferred two-stage optimization, which first obtained a well-trained ARGM and then trained an additional EBM based it. Such an optimization strategy not only introduced heavy training process for EBM but also did not enable ARGMs themselves to benefit from the properties of EBM in modeling the joint distribution in a temporally more coherent way, and required more training parameters to estimate energy scores, burdening the intricacy of the learning task.
Our method of combing ARGMs and EBMs takes a different approach, which seamlessly integrates energy-based models into autoregressive models by utilizing the extra degree of freedom within the final softmax layer of the model. We show that in this way the ARGM can be trained using an energy-based learning objective, which allows the ARGM to avoid those intrinsic concerns, such as exposure bias, with the help of energy-ban based models as former work did (Deng et al., 2020; [Bakhtin] ct al. 2021) whilst being free of increasing the learning model's complexity. This property makes our E-Forcing rather easy to be applied in the training process of any ARGM for any specific task, as no structural changes required.
Besides, we follow the predominant approach for training explicit density generative models to minimize the KL divergence between the (empirical) data distribution and model distribution, which gives rise to the gradient-basee contrastive divergence (CD) methods (Hinton 2002 Kim & Bengio, 2016) for energy-based models. Typically, these methods require Markov Chain Monte Carlo (MCMC) process to sample data from the EBM for the "negative phase" gradient estimation, which extremely time-consuming and, meanwhile, inapplicable for discrete data, such text. To solve this, we present a way to estimate those "negative phase" gradients through those samples generated with the network's autoregressive view instead of the EBM view, making the training feasible. Since our method combines the EBM and ARGM seamlessly as a whole, i.e., the ARGM is also an EBM itself, the exposure bias problem can be mitigated due to the fact that autoregressively sampled data involved in the "negative phase" of CD) methods. On top of it, unlike ARGMs, which factor the joint distribution into a product of conditional distributions, EBMs are able to model the joint distribution directly and score each input at the sequence level instead of at the token level, which makes them capable of modeling long-range coherence.
In summary, the following contributions are made to this paper: i) We introduce novel scheme by integrating the EBM view into autoregressive generative models seamlessly; ii) We proposed a novel method, named F-Forcing, for efficiently optimizing the energy-based autoregressive model via contrastive divergence based on importance sampling but not MCMC; iii) We successfully decrease the inherent flaws of autoregressive models exposure bias and weak temporal coherence by leveraging E-Forcing's 's two-phase optimization, which makes use of both real and generated data; iv) We demonstrate clear improvements of the proposed methods on various tasks such language modeling, machine translation, and image generation.
2 BACKGROUND AND RELATED WORKS
2.1 ENERGY-BASED MODELS
Let Pa denote the data distribution. Energy-based models (LeCun et al. 2006) are interested in learning an unnormalized energy function (x) that defines the density(mass) function na (x) as
exp( (x)())
Ty (x)
(I)
Za
where Eg : X' + R denotes an energy function which aims to map data sample from data space X' to an energy scalar, and Z00) = Σx ex)(-E((x)) denotes the normalizing constant, also known as the partition function, which can be barely estimated. Any function can be used as an energy function to represent an EBM as long as it can generate a single scalar given some input x and the normalizing constant is finite? Contrastive divergence algorithms are commonly used to optimize EBMs via maximum log- lioe (Hinton 2002 Kim & Bengio 2016 Grathwohl et al. 2020).
Without constraining the parametrization of Eg, this can be achieved by bounding the region of space in which a takes its allowed values.
2

---

Under review as a conference paper at ICLR 2023
Correspondingly, the gradient of the log- ligliiklihood, which needs to be maximized, with respect to can be expressed as
V Pa log log (x)] Exx(x) Vo Ex
Rpa(())
Vo [x)
(2)
The first term on the right-hand side of is usually called the "negative phase" term while the second term is called the "positive phase" term.
In general, due to the challenge of sampling from EBMs, training EBMs by contrastive divergence methods (Hinton 2002 Kim & Bengio 2016, Grathwohl al. 2021 is difficult, especially on
high-dimensional data. MCMC methods (Nijkamp et al. 2019 Du & Mordatch 2019 Grathwohl al. 2020) are usually adopted for data sampling. However, these methods require enormoos extra computing overheads and are not applicable when the input is discrete such as text sequences (Deng et al. 2020). As result, a variety recent works attempt to explore the strategy of training an FBM without MCMC. In particular, [Bakhtin et al. (202T): Xu et al. (2021) 1 Gao et al. (2020) optimize the
EBMs by using noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010: Ma Collins)
2018). Durkan & Nash (2019) estimate the intractable normalization component by utilizing ARGMs
and importance sampling. [engii et al.; Che et al. (2020); Wang et al. (202T) skirt the challenge of
collecting data in the nigh-dimensional data space by performing sampling using carefully crafted latent space, which improves sampling efficiency.
2.2 MODELING DISTRIBUTIONS AUTOREGRESSIVELY
Modeling high-dimensional data distributions directly is usually rather challenging task due to "the curse of dimensionality" (Bellman 1954). One alternative method to sequential the random variables and then factorize the joint probability distribution into the product of conditionals based on the sequence structure, which is the core idea of autoregressive generative models (ARGMs). ARGMs have been very successful, in particular for sequential data. For example, ARGMs have been widely used in language modeling (Vaswani et al. 2017 Dai et al., [209 Radford et al., 2019). audio synthesis (van den Oord et al. 2016a), and even image generation (van den Oord et al. 2016c b Salimans et al.] 2017).
However, the advantages of ARGMs are balanced some extent by issues of (1) exposure bias (Ran- zato et al. 2016: Bengio et al.] 2015 Song et al.) 2020). due to the discrepancy in input context distributions between the training and inference stages, and (2) weak long-range coherence (Deng
ct al. 2020), due to the inherent greedy selection of one token at a time without look-ahead.
2.3
THE MIXTURE OF EBMS AND GENERATIVE MODELS
The seminal idea of combing generative model and energy based model has been explored by
plethora of great works (Pang et al.] 2020 Durkan & Nash [219 Xie et al. 2019 2020: Xiao et al.
2021 Bakhtin al. 2021| Che et al. 2020 Arbel al.] [021] Deng al. 2020 Bakhtin et al.) 2021
Durkan & Nash 2019). In particular, Pang et al. (2020) aimed to learn an energy-based model (EBM)
in the latent space of a generator model, so that the EBM can act as prior model on the generator model's top-down network. VAEBM, a symbiotic composition of a variational auto-encoder and an EBM, was proposed by (Xiao al. 2021). Arbel et al. (20TT) proposed a novel training method for GAN/EBM combined model by leveraging the Donsker- Varadham representation of KL-divergence.
Among these works, Residual EBM (Deng al. 2020 Bakhtin al. 2021 Durkan & Nash 2019) and EBR (Naskar et al. 2020) may be the most related works to our paper. Authors of these works have made their attempt benefit ARGMs from the advantages of EBMs. However, different from our work, these works utilize a two-stage optimization scheme, which first obtained a well-trained generative model and then trained an additional EBM on top of Such an optimization strategy does not enable ARGMs themselves to benefit from the properties of EBM in modeling the joint distribution. Besides, in order to benefit from the EBM, complicated re- re-sampling or re-ranking schemes needed during inference time. It also increases parameters since it uses independent networks to represent the ARGM and the EBM, burdening the intricacy of the learning task. In contrast, we introduce the EBM inside the ARGM, treating the ARGM directly an EBM itself.
3

---

Under review as a conference paper at ICLR 2023
3 TREATING THE ARGM AS AN EBM
In this section, we present the overall framework of our E-Forcing method for training better autoregressive models. Let (XI,...,KK' be random sequence of length K drawn from the real data distribution X denote the random variable at time step k, and X<* represent the random subsequence before time step i.e. Xck (x1. X2... Xx,.Xz,... ................................ : xx-1). The general spirit of our design is to model the joint distribution Px(x,, x<k) by integrating an EBM inside the autoregressive model qo-
Formally, given an autoregressive model (x)...... xx) I go (x (x<k) parameterized by 0, we introduce independent energy-based models po (x: x<k) for each time step k: S K, with the formulation following
po (Xk. xxk) (x(x)))
Zg
where Zg equal to Eg Ex, (0x(Xx indicating the normalization constant, Φe (-) represents the energy function. Essentially, po (x x<k) is a product EBM, defined as the product of go and another EBM Φ0-
3.1 DEFINITION OF THE ENERGY FUNCTION
We define the energy function @e (Xx, Xxck using X corresponding component of network's ' output logits given the input context Xeck (e.g., given sequence "This is Friday." and assuming the corre- sponding index the token "Friday" the vocabulary is i, then the value of - ("Friday", "This is") is the i-th component of the output logit, namely, the input tensor of the final softmax layer).
The rationale behind such design of energy function is out of the extra degree freedom concealed inside the softmax transformation S RM (0, M which can convert an unnormalized vector with size M into probability distribution consisting of M probabilities
EF
S( zM])
(4)
M EM
Σ
It's easy to observe that the softmax operation is unaffected by the input vector's overall magnitude, that is, S(22),.... zM] S( [MM] + C), VC IR. Such property allows model the energy function by using the ARGM itself instead of introducing new network.
3.2 ENERGY-BASER LEARNING OBJECTIVE
Other than making the qo to match Pa. E-forcing has an additional training objective to make the K parametric distributions po (X x<k) to match the real data distribution Xx(x,,x<k at any time step l < K. This can be achieved by minimizing the Kullback-Leible (KL) divergence between the distributions for each time step of sequence,
DKL [u(xx x (lo wk K),
We attempt to use contrastive divergence methods (Hinton et al. 1995 Kim & Bengio 2016) to minimize the objective 3 descending the gradient w.r.t. / according to Eq. [ each time step. Specifically, given an arbitrary time step k, we have the corresponding gradient of objective 5 with respect to
VOLEBM-CD- Epp Vo Vo x<k)
Epo Vo x<t)
(6)
Positive Phase Gradient
Negative Phase Gradient
where Ex xxc) (Xx, xck) log qo (x<x).
Optimization via Eq. involves sampling data from the model distribution pe and can thus lead to the discovery of non-data- data-like like samples, whose likelihood is then explicitly reduced the corresponding energy increases during the training. E-Forcing is therefore not plagued by the exposure bias problem naturally. Besides, because model the joint distribution each time step, E-Forcing assess
4

---

Under review as a conference paper at ICLR 2023
the sequence up the current time step whole and generate more coherent data using energy sampling (Deng et al. 2020). However, the negative phase gradient is frustrating to compute, especially for discrete data (e.g. text) where common MCMC methods (Welling & Teh 2011 can not even be applied. Therefore, we propose a novel variant of contrastive divergence methods for E-Forcing's 's optimization in Section
OPTIMIZATION
The key obstacle of optimizing the objective Via contrastive divergence methods (Hinton, 2002 (i.e. descends the gradient of Eq. 6 is sampling data from the model distribution po for estimating the negative phase gradient. The common MCMC algorithms are not desirable for generating "negative" samples because they are rather consumnnn and not applicable to discrete data. In order to make the optimization process both efficient and feasible, modified the original CD methods by means of the importance sampling technique (Horvitz & Thommpon 1952). which holds two parts of gradient estimation.
4.1 POSITIVE PHASE GRADIENTS
Since the training set consists of i.i.d. samples sampled from the real distribution pa. the computing of positive phase gradients is not difficult. To be specific, by replacing Ex(xE, x<*) with the form
de (Xu, ( x<a)) log 4o (x<*) in Eq 6 the positive phase gradient g (0) with respect to parameter 0
can be written into
g (0) Eap Vocio (xk, xek) xx),
Vo log go (x<x)]
Since carrying out sample estimation of the expectation over the data distribution is viable, and the score de [k, x<c)) can be acquired by simply accessing the output logit of ARGM (according to the definition of de in Sec. 3 the first term of the positive phase gradient g can likewise be
readily computed. Besides, we can observe that the second term lp.l-Va Ve log qe (xxx)] of g (0) is the negative gradient of likelihood go (x<<)))ss logarithm, which is exactly the objective of maximizing the autoregressive generative model go log-likelihood.
4.2 NEGATIVE PHASE GRADIENTS
The estimation of negative phase gradients g Ep Vode X=c)) - Ve log (x=x)) on the other hand, is more involved. Sampling data from Pe is required for estimating the expectation Ey.. whereas po the introduced energy-based autoregressive model, which is an explicit autoregressive generative model and we can only access its modeled density(mass) function Po-
Inspired by the idea of importance sampling, we substitute the troublesome estimation of the ex- pectation over distribution pe with the expectation over distribution fo. which is the underlying autoregressive model that can generate samples considerably easier. Accordingly, the negative phase gradient E Ex Xxpxx Eo (Xx, x=x)] has the following form (See the detailed derivation in Appendix B).
8 (0) Eqo w(X<c) ) VX x<c) - Volog (xxxx)]
(8)
Ex. (x((xxx))))
where w(xc)) (xg(((22 [Ex (ox((xx-x2)))
(9)
According to Eq all the estimated expectations only need sampling from the autoregressive model do rather than the distribution po, and the reweighing weight w in Eq. p does not involve expectation computation over distribution Pa either. Generally speaking, producing data from an autoregressive model is simple ancestral sampling process and naturally suitable for discrete data, as compared with sampling straight from an explicit generative density estimator, which needs MCMC approaches (Durkan & Nash 2019). Besides, the term Ex Ex [(x<k) Vo log qo (xxx)) in Eq. can be regarded as a re- weighted gradient qo 's information entropy with respect to 9. This term be optimized similarly to the forcing training of the autoregressive model with
5

---

Under review as a conference paper at ICLR 2023
the "teacher" sequence generated autoregressively by the model itself. The scheduled sampling methods (Bengio et al. 2015; Ranzato et al. 2016 Mihaylova Martins 2019) are similar to this
term but without the re- weighting factor.
Moreover, the reweighing weight w of Eq. 9 can be further refined (see the derivation in Appendix B.3 and we observe that w w(x<k)
where (u(x<*) po(x<*)) (x<+)),
indicating the possibility of which distribution (pa or qo) the input context x most likely come from, Correspondingly, w(x< reflects the context x 's relative magnitude of p(x<a) compared with the average among all potential contexts-the larger the value of w(x< the more likely the context x in the data space coming from which is modeled by the product of autoregressive models and EBMs. During training, those imput sequences with contexts more likely from pe than qo will be assigned larger weights w while others will be assigned smaller weights w.
4.3 FINAL OPTIMIZATION OF E-FORCING
Algorithm 1 Optimizing ARGMs with B-Forcing
Given: a training dataset ~ Pa random-initializeed autoregressive model go. K E is the generation length
for iteration = 1; <max iterations;i + do
Sample minibatch B {(c,, s:) E if i then
8; is of length K, ca is the context of s;
D. After N iterations, we start applying E-Forcing
Volotal
Σ K ===1 VoC (B)
else
Autoregressively generate [B] samples from qo conditioned c. denoted as B VoLtoral t Ek VoL + AE (( (B
end if
Update +-0 - 10-VVoo V
17 denotes learning rate
end for
Finally, with the help of the above estimation of gradients regarding two phases of Eq. 6 we are able to optimize the product EBM pe via descending the estimated gradient of contrastive divergence loss V VoLERM-CD for any time step k
(k) V
EBM CD (
(8) (() (0)
(10)
Eq. 10 can be easily estimated by using "positive" samples from the given training dataset and autoregressively generated "negative" samples from go.
Nevertheless, training the model from scratch with the energy-based learning objective alone can not work well in practice. The reason is simple: at the initial stage of the training process, what we have is just randomly initialized network that can barely generate anything meaningful. This fact indicates disjoint supports between the real distribution pa and modeled distribution pg. Importance sampling fails in this case. Hence, to make the optimization more feasible, we pre-train the entire model with autoregressive loss CAR by teacher-forcing for few epochs before introducing the energy-base based learning objective. In sum, the final gradient concerning parameter 0 at each update iteration is
V Leoota
Σ V Coc (R) A A=1
Vo VOCEBM-CD())
(11)
where ' adjusts the ratio between the two objectives for the time step k. The intact optimization We found that an exponentially descending configuration of
procedure is shown in Algorithm| | coefficients A according the order of time steps works well. One possible reason that such set of coefficients can remedy the imbalanced training signal by negative phase gradients in Eq. 8 among time steps.
We take K to he the length of segment of the transformer.

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
To empirically corroborate the effectiveness of E-Forcing and show its broad applicability, we have conducted extensive experiments on applications, such as language modeling and machine translation. In this section, will first introduce these experimental setups and analyze the obtained results. Besides, we also carried out a series of experiments to further show our E-Forcing method's ability to resolve ARGMs' inherent flaws(i.e. exposure bias and incoherence generation). More experimental settings as well as analytical experiments are shown in Appendix A and
5.1 APPLICATION TO LANGUAGE MODELING
For the language modeling task, three different datasets. WikiText-103 (Merity et al. 2017), Toronto Book Corpus (Zhu et al. 2015 Kiros et al., 2015), and CC-news (Mackenzie et al.] 2020), are chosen as testbeds; two autoregressive network structures are used to evaluate the effectiveness: vanilla Transformer (Vaswani et al. 2017) ("Tr-Base" for short) and Transformer-XL. (Dai et al., 2019)
("Tr-XI." for short). We regard the vanilla training with the teacher foreing technique as the baseline method. Besides, we also compared our E-Forcing with the residual EBM Deng et al. (2020) method, which is a typical method to improve the performance of language models by utilizing EBMs. In order to benefit from the EBM, the residual EBM method requires new network to estimate the energy scores and imposes a Top-K energy resampling scheme during inference
The final results reported in Table T We can from the results that E-Foreing outperforms two pure autoregressive models with clear margins over all three benchmarks. Specifically, on the Wikitext-103 benchmark, our B-Foreing improves the performance of the Transformer-BBal model and Transformer-X XL model by 0.62 PPI. points (from 30.56 to 29.94) and 0.30 PPI. points (from 24.20 23. 23,90) respectively; on CC-news and Toronto Book Corpus benchmarks. our method obtains 0.51 ppl and 0.47 ppl performance gain respectively and gets further improvement once energy resampling technique was applied. Besides, though residual EBM's learning parameters are twice as ours and their method is unable to directly benefit autoregressive models without Top-K energy resampling, our E-Forcing achieves comparable results to them, even slightly better on Toronto Book Corpus and Wikitext- 103 benchmarks.
PPL +
Method
CC-News
Toronto Book Corpus
WikiText103
Baseline (Tr-Base)
18.29
17.57
30.56
Baseline (Tr-XL)
24.20
Residual EBM (Tr-Base)
15.57-15.58
16.98-17.00
29. 29.88-22 29.99
Residual EBM (Ti-XI.)
23 23.85-23.88
E-Forcing (Tr-Base)
15.78
17.10
29.94
E-Forcing (TrXL))
23.90
E-Forcing E.R. (Tr-Base)
15.63-15.67
16.89-16.93
29.81-29.84
E-Forcing E.R. (Tr-LL)
23.79-23.82
Table Language modeling performance different models on three benchmarks. Evaluation is conducted using perplexity (PPL). E.R. is the abbreviation of Energy Resampling technique (Bakhtin et al. 2021 which serves as necessary module of Residual EBM.
5.2 APPLICATION TO NEURAL MACHINE TRANSLATION
We further evaluate E-Forcing's effeectiveness on neural machine translation (NMT), which can be regarded as a conditional generation task. We mainly analyze E-Forcing on the IWSLT14 dataset, which includes six different language pairs (fGerman, Spanish, Italian] English and English - (German, Spanish, Italian]) (Hereafter we abbreviate English, German, Spanish, Italian as "EN",
'It is worth noting that Top-K energy resampling can not get the PPI. directly. Bakhtin et al. 2021 provides way to approximate PPL., which leads to estimated interval of PPL.
7

---

Under review as a conference paper at ICLR 2023
Label Smoothing
Scheduled Sampling
Beam Scarching
BLEU 1
Method
Arg.
DE
EN->
32,44-0.86
27.92=003
30,48+0008
3861=011
35,42+0.00
33,62:0007
27.411000
28.7220.04
31.3910005
39,55
36.38400077
32.85
33.68=0.03
27.62+0004
28.81
31,42=007
39.85=0.13
36.71
33.02
Base
31.61
28/1610006
29.72.0.10
$2,2910003
10.64.0.077
37.4810.08
33.87
34,23:40.06
27.964:008
19.26:00.11
31,9340.08
40,16
37.21:00:44
B
35,10.00.04
28.731004
29.9710.07
52,6410.12
10.91.10.0
57,9310.10
34.21
32.99-10.10
27.1510.08
26.33:20.12
51.1310:44
39,56.00.11
36,07.10.2
32.51
34.06=00.86
27.97
29 26
31,90 #:0.13
40.30 #0.3
36.92 1:000
33,40
E-Forcing
B
34.93 -0.05
28.91 +0.12
30.04 +0.11
32,56 +0004
41.01 +0.06
37.73 +0.12
34.20
34.58
0.009
28.38
1:012
29.56 10.10
32.11
1:000
40.95
10.03
37,56 1:0.0
33.85
-0.05
+0.04
+0.11
41.58
Table 2: Comparison of BLEU scores between approach E-Forcing and the base ARGM trained just with cross-entropy loss six translation pairs of IWSLT14 datasets. We "_" denote that the training trick not used while "V" indicates we use "5 B" represents we use beam searching with beams.
"DE", "ES", "IT"). In addition, we also reported the result of B-Forcing over the WMT16 (English German) benchmark, which is a relatively larger dataset, in Table
Results concerning IWSLT14 are shown in Table where we use six-layer vanilla transformer as the base autoregressive model. We test not only the pure performance of E-Forcing but also the compatibility with other techniques. In detail, we can observe that (1) without any partic- ular engincering, E-Forcing outperforms the vanilla training with cross-entropy singly(teacher- forcing) by 0.62 (31.92 + 32.54) BLEU points on average, especially on three translation pairs- 38.61 39. 39.56 on Spanish-to- tii--tt--Ennhhhhh 30.48 - 31.13 on Valiinn-o-EEnglish, 35,42 + 36.07 Bnglish-to-Spanisle (2) E-Forcing is compatible with other techniques like scheduled sampling, which can help alleviate the exposure bias problem to some extent. They not mutually clusive and can work together to further improve the performance of the base ARGM. (3) How- ever, since scheduled sampling can reduce exposure bias and beam search can somewhat allevi- ate the flaws caused by greedy selection at each time step, the performance gain of E-Forcing when all these tactics are combined only 0.34 (34.21 - 34.55), which lower than the 0.62 (31.92 32.54) obtained when the model is purely trained without these other techniques.
Additionally, 3 shows the performance of E-Forcing on the WMT16 English - German task. For two different model sizes, enabling label smoothing (L.S.) improves model perfor- mance by 0.52 and 0.35, respectively. The per- formance of the base transformer model further increases to 28.36 BLEU points when scheduled sampling (S.S.) is used, while the larger model improves to 29.23 points. E-Forcing paired with label smoothing and scheduled sampling yields the highest scores of 28.62 and 29.44, respec- tively. Overall, our training strategy outperforms ARGM's vanilla sorcher-foreig training and can have uniformly favorable impacts across dif- ferent models and dataset sizes.
Model
L.S.
S.S.
E-Forcing
BLEU
27.56
28.04
Tr-Base
v
28.36
v
28.62
29.05
Tr-Large
29.23
29.44
Table 3: Translation performance of proposed F- Forcing WMT16 English-Gennen evaluated with BLEU. We uniformly use heams when applying beam search. "L.S." denotes Label Smoothing and "S.S." " de- notes Scheduled Sampling.
5.3 EFFECT ON THE EXPOSURE BIAS
We follow the analytic experiments in the work (Zhang et al., (019)) to show that our E-Forcing is capable alleviating the exposure bias problem. To be concrete, we randomly select IK pairs from the training data for each translation pair and use the trained autoregressive model which applied E-Forcing to decode the source sentences, and then count the ground truth words whose probabilities in the predicted distributions produced by baseline and denote the number as M. The ratio N the total number of words tested calculated. The detailed results shown in Table 4 We find that the results all different tasks greater
E-Forcing are greater than those produced by the
8

---

Under review as a conference paper at ICLR 2023
than 50%, which demonstrates the ability of our E-Forcing in alleviating the exposure bias problem to some extent.
Trans. Pairs
DE
EN
EN
DE
EN
IT
I- EN
ES
EN
EN
ES
14203
14554
14976
13952
16021
15359
Total
22148
23057
23654
23744
23860
22775
Ratio
64.12%
63.12%
63.31%
59.76%
68.33%
67.43%
Table 4: The effect of E-Forcing the exposure bias problem. Fach test set of translation tasks contains 1K sentences selected randomly. A denotes the ground truth words whose probabilities in the predicted distributions produced by E-Forcing are greater those produced by baseline.
5.4 EFFECT ON INCOUERENCE GENERATION
Translation
Scheduled
E-Forcing Training
Targel Sentence Length
All Test
Task
Sampling
[0, 25)
[25, 49)
[50, oo)
37.72 +0004
33.24 +0006
30.86 +0.07
34.61 +0.08
De-+
38,20 10007
33.76 10.03
31.08 10.66
35.10
38.37 11006
33.92 10.09
31.43 10.04
35.36 10.05
1:0.03
32.73 10.02
26.86 +0005
32.29 1:003
It- En
35.52 :0009
33.25 +0.08
26.95 +0.14
32.64 10.12
35.56 1-0.10
33.33 +0.13
27.21 +0007
32.82 +:0.1
43.37 +0.05
39.67 10.08
37.14 10.06
40.64 10.07
43.61 +0.09
40.00
+0.04
37.38 +:006
40.91 +0006
43.84 10.10
40.35
10.05
38.07 10.04
41.58 10.07
Table 5: Performance comparison on the IWSLT114 test set the different lengths sentences on three translation tasks (German to English, Italian to English, and Spanish to English). Performance is evaluated by the BLEU score.
We also attempted quantitatively validate that our E-Forcing benefit ARGMs by improving the coherence of generation. SableBishows the BLEU scores of generated translations on the IWSLT14 test set with respect different lengths of the source sentences. Intuitively, due to the cumulative effect of greedy selection at each time step, the collection of samples with longer sentences ought to be more plagued by the incoherence of generations problem. Our approach can outperform the vanilla training in all three length intervals, especially in the lengthy sentence interval [50, oo), indicating that our E-Forcing can resolve the incoherence problem to a degree.
CONCLUSIONS AND FUTURE WORK
this paper, we propose a novel training method dubbed E-Forcing for ARGMs by treating them as EBMs. This is achieved by defining the energy function using the softmax operation's 's extra degree of freedom within an autoregressive network. We further design a unique way to improve the training of E-Forcing using importance sampling. Experimental results demonstrate the effectiveness of E-Forcing to alleviate exposure bias and incoherence problems of ARGMs. the future, we expect extend E-Forcing other sequential generation tasks (e.g. text summarization, audio generation) and incorporate the proposed methodology into other advanced autoregressive architectures.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In 9th Interna- tional Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b.
Anton Bakhtin, Yuntian Deng. Sam Gross, Myle Ott, Marc' Aurelio Ranzato, and Arthur Szlam. Residual energy-based models for text. J. Mach. Learn. Res., 22:40:1-40:44 2021.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsc Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, jun 2005.
Richard Ernest Bellman. The Theory of Dynamic Programming, Santa Monica, CA, 1954.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1171-1179, 2015.
Yoshua Bengio, Grégoire Mesnil, Yann N. Dauphin, and Salah Rifai. Better mixing via deep representations. In Proceedings of the 30th International Conference on Machine Learning. ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28, pp. 552-560.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved au- toregressive generative model. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 863- 2018.
Junyoung Chung. Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1442.3355 2014.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguisties, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 2978-2988, 2019.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy- based models for text generation. 8th International Conference on Learning Representations, ICLR 2020, 2020.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp- 3603-3613, 2019.
Conor Durkan and Charlie Nash. Autoregressive energy machines. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, volume 97, pp. 1735-1744, 2019.
10

---

Under review as a conference paper at ICLR 2023
Ruiqi Gao, Erik Nijkamp, Diederik P. Kingma, Zhen Xu, Andrew M. Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In 2020 IEEE/CVE Conference on Computer Viston and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 7515-7525, 2020.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning energy- based models by diffusion recovery likelihood. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Will Grathwohl, Kuan-Chich Wang. Göm-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy-based models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrasto estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. 297-304, 2010.
G. Hinton, P. Dayan, B. Frey, and R. Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 268 158-61,, 1995.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 1478)1177) 2002.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from finite universe. Journal of the American statistical Association, 47(260):663-60 685, 1952.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. CoRR, Ne2116666003339 2016.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 3294-3302, 2015.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, jul 2004.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 3698 3707, 2018.
Joel M. Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. Ce-news-en: A large english news corpus. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pp. 3077-3084, 2020.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19(2):13-330, June 1993.

---

Under review as a conference paper at ICLR 2023
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Tsvetomila Mihaylova and André F. T. Martins. Scheduled sampling for transformers. In Fer- nando Emilio Alva-Manchego, Eunsol Choi, and Daniel Khashabi (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 August 2, 2019, Volume 2, pp. 351-356, 2019.
Subhajji Naskar, Amirmohammad Rooshenas, Simeng Sun, Mohit lyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using mergy-based models. CoRR, abs/2009.13267, 2020.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5233-5243, 2019.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint wrXiv:166600528, 2016.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonsirations, 2019.
Bo Pang,
Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning. volume 80 Proceedings of Machine Learning Research, pp. 4095 4104, 10-15 Jul 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Kaitao Song, Xu Tan, and Jianfeng Lu. Neural machine translation with error correction. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3891-3897, 2020.
Aäron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA, 13-15 September 2016, pp. 125, 2016a.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4790 2016b.
12

---

Under review as a conference paper at ICLR 2023
Aäron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1747-1756, 2016c.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp- 1747-1756. PMLR, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open-world uncertainty modeling for confidence calibration. CoRR, 0e//2107.1266 2021.
Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning. ICML 20/1, Bellevue, Washington, USA, June 28- July 2, 201 pp. 681-688 2011.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between variational autoencoders and energy-based models. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
J. Xie, 7. Zheng, X. Fang, S. Zhu, and Y. Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1-1, mar 2019. ISSN 1939-3539.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chin Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Trans. Pattern Anal. Mach. Intell, 42(1):27-45, 2020.
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1, pp. 4334 4343, 2019.
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 19-27, 2015.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL SETTINGS
In this section, we introduce detailed setups of different benchmarks as well as the information of corresponding datasets.
A.1 DATASETS
We conducted our experiments based 7 datasets over different learning tasks:
1. WikiText-103 comprises 103 million training tokens from 28 thousand articles, with average length of 3.6 thousand tokens per article.
2. Toronto Book Corpus consists of fiction books in 16 different genres, totaling about half billion words.
3. CC-news is de-duplicated subset of the English portion of the CommonCrawl news dataset, which totals around 16 Billion words.
4. IWSLT14 contains about 170k training sentence pairs, 7k valid pairs, and 7k test pairs. has six different domains of language, and each two of them can consist of a translation pair.
5. WMT16 contains 103M training tokens from 28K articles, with an average length of 3.6K tokens per article, which allows testing the ability of long-term dependency modeling.
6. MNIST is a large collection of and a test set of 10,000 examples.
digits. It has training set of 60,000 examples
7. CIFAR-10 subset of the Tiny Images dataset and consists 60000 32x32 color images. The images are labeled with one of 10 mutually exclusive classes.
A.2 IMPLEMENTING SETUPS
Translation-IWSIII
Translation-WMe
Language Modeling
Tr-Base
Tr-Base
Tr-Large
Tr-Base
Tr-XI
Number Layers
12
Hidden Embed Size
512
512
1024
512
410
FC-Layer Embed Size
1024
2048
4096
2048
2100
Attention Heads
4
8
16
8
10
Dropout
0.3
0.3
0.1
0.1
Learning Rate Ir scheduler
5e-4
le-3
le-3
5e-4
2.5e-4
inverse sqrt
inverse sqri
inverse sqrt
inverse sqrt
cosine
Warm up Updates
4000
4000
4000
4000
10000
Weigth Decay
le-4
0.0
0,0
le-2
0.0
E-Forcing Start Epoch
15
15
10
15
10
Table 6: Hyperparameters of different model structures and datasets. "Tr-Base", "Tr-Large", and "Tr-LL" indicate Transformer-Bast Transformer-Large and Transformer-X respectively
We uniformly use the Adam optimizer. The training will be stopped once the model has not obtained better performance for 20 epochs the validation set. For translation tasks, the length of generated fake sentences, which is used for the computing of the negative phase Eq. T0 is dependent on the source sequence whilst for language modeling tasks, we fix the length of generated fake sentences as 50 during training. The model structures for language modeling and machine translation tasks are shown in Table 6 As for the model structures of the image generation task, we use the official structure reported by PixelCNN (van den Oord et al.) (016c) and Gated PixeICNN (van den Oord cet al. 2016bb without modification. The source code will be released once upon acceptance. We use the same batch of samples generated autoregressively to approximate both the expectations in 10 and weight w (i.e., shared), which does not need to sample twice. The number samples in batch is dynamic while the maximum number of the total tokens in batch is fixed (4096 in experiments). If the length of sequences batch is 32, then includes 4096 32 128 samples in total. It is common strategy in language generation tasks and has been used in many frameworks(e.gg
14

---

Under review as a conference paper at ICLR 2023
Fairseq (Ott al. 2019)). We generate samples autoregressively as many as the number of sequences in the current batch at each update iteration.
B DERIVATION OF THE NEGATIVE PHASE GRADIENT
In this section, we show the detailed derivation of Eq. 8 Formally, as shown in Sec. given an autoregressive model go (x=+) I qo (x/ xx<) (k denotes the time step) with parameters 8, define a product the autoregressive model and an EBM as follows
pa (xk, xxk) (o(xxe)-
(12)
Zo
where (x(x<k) =
I go(x) (x(xxxxe1) Zo is the normalization term and equal to
Ex'en rago Exx e (xc. w.r.t 0 of "negative phase" is
The optimization of (x Xick) includes two phases, and the gradient
Ex Exxerpp Vg lo (xxxx) Exx, Xeampo Veog(Xx,
(13)
Next, we will show the specific derivation about how to transform Eq.
13 into Eq.
B.1 DERIVATION OF THE FIRST TERM
The first term Ex V log (x(xx)) can be processed as follows
E Excaropo (ologg((X<k))
Σ po (x V (x
Xek
po (xk: xxck V log qo (xx<))
Xek Xt
(14)
Ex, e (xx(xxxx)))
Σ g0(x<c))
V log (x(c))
Zo
Xck
(xxx=~9(xxee) ((<x< log (x
Exp c(sspiee)))
where we have w(xek)
because
Bele (oo(te) Eny -pplestes)
Ex, e (-((x((x)))
Ex, e ((x))())))
w(xc))
Zo
Exar Exi (0(x<x)) (0(
Ex, e (0xxxxc))
(15)
Exer (x(x)c)) Exe e (6(xxxee)))
Ex, (-olxxxcct)
Exexrgo(Xec) Exx, x-xe
15

---

Under review as a conference paper at ICLR 2023
B.2 DERIVATION OF THE SECOND TERM
Then, we tackle the second term Ex
L Voon Xcx)] as follows
Ep [oo00(XR x=*)]
Σ
po (ck
Vodu Xcck
Σ po(X,, X(ck)
90 (x, x<c)
Vo0o (xx, X<k))
90 (x(x xzk)
Xe.
90 (X(n) (0(xx(xcA)
Σ qo(Xx: Xcck)
(xsulXx,, XxR)
Zp 90 (Xk, xxck
Xi.Xcck
do(xx(xcc)
1 Vooo(xx,, xxx)]
Ex (xxpxeer
go (xER) (x(xxx))))) go go(xA ((ckk
Zo
= - (x(xx)
Σ
qu((
(XX,(xx;XCN))
Zo
Xick
Xx
Σ 10((xc Σ
e (e-do(NEXEE))
Vacha (x (x),
Zo
Nick
Xi.
(xc(x))))
Vode(Xx,
Zu
Xt
Exi xx(xxxx))))
(xp(x))))
Vooo(xx: xxx)]
Exi e (xxo
Zo
Exc(xex)
Σ g0 ( (xdO xx+)]
Xx
= Ego (x
((< V eoo (Xt, xx+)]]
Exx. (x (x Vooo (o Xx, x=k)]
(16)
where w(x<x) is also equal to Ex, a oix))ee)
Combining Eq. 14 and Eq. 16 we can obtain an
Zo
equivalent form of the gradient of the negative phase without any expectation over as
(xxxssn(xxx)) (w(xx)) Vo log (x(xx))] + Ex x xxx.
xck)), (17)
Exp ((N()()))
where w(xek)= (xxcx) EEx, e colxx
(18)
B.3 FURTHER REFINEMENT OF W
The reweighing weight w can be further deduced as
Σx, (x(xxxEE))
pe 2x(x Exe xt ge(xcN)
w(x<x)
E Ex ro (x2x)
(Ex, (ou((x
Ex ~ao (x 1Exa xx(x)
golxec)
(19)
pol xxck
m(x<x)
xxce)
Ex ruge (xxc)) ) (x.c))
(a(xcc))
Ex (XECK)
where μ(x<)) )) is defined as zo (x<e))
(x(
C MORE EXPERIMENTAL ANALYSIS
C.I ANALYSIS TO TOP-K K ENERGY RE-SAMPLING
Top-K K energy samplig in the inference stage is introduced by Bakhtin et al, (202I) which collects many candidate sequences generated autoregressively in the inference stage and then re-samples
16

---

Under review as a conference paper at ICLR 2023
Trans. Pairs
DE +
EN-> DE
EN-
IT- EN
ES-
34.93
28.91
30.04
32.56
41.01
37.73
34.97
28.92
30.08
32.60
41.07
37.71
10
34.95
28.95
30.07
32.59
41.03
37.75
Table 7: The effect Top-K correction in the inference stage. We tested BLEU scores of using different & on different translation pairs IWSLT14 dataset.
from them depending on their energy scores estimated by the network. To measure the effectiveness of the Top-K energy re-sampling towards our method, we conduct an ablation study neural machine translation task by selecting different K = {0, 10). The results are shown Table 7 and performances are evaluated by using the BLEU score. From Table 7 we observe that the benefits brought by Top-K sampling is minor (K=(5, 10)). when compared with the model without Top-K sampling (K=0). This together with the results shown in Table T shows that E-Forcing can considerably benefit the base autoregressive model even without the energy resampling technique.
C.2 APPLICATION TO IMAGE GENERATION
In order to illustrate the effectiveness and generality of our method in processing different modality tasks, we further show the results of applying E-Forcing to image generation in this section. We apply E-Forcing Pixel-CNN (Van Oord al. 2016) and its variant Gated Pixel-CNN (Oord et al. 2016). Experiments are carried out on the MNIST and CIFAR-10 datasets.
Table summarizes the quantitative results measured by per-pixel nega- tive log-likelihood (NLL). We can see that with the help of our E-Forcing, both the Pixcl-CNN and the Gated Pixel-CNN can obtain improvements in all datasets (0.17 0.15 and 3.14 - 3.07 for Pixel-CNN on MNIST and CIFARIO respectively and 0.14 + 0.12 and 3.03 + 2.97 for Gated Pixel-CNN on MNIST and CIFARIO respectively). This is further evidence in favor of the energy- based learning objective for improving autoregressive models.
Test (Train) NLL |
Model
MNIST
CIFAR-10
PixeI-CNN
0.17 (0.13)
3.14 (3.08)
Pixel-CNN (w/E-Foreing)
0.15 (0.12)
3,07 (2.98)
Gated PixeI-CNN
0.14 (U.11)
3.03 (2.90)
Gated Pixel-CNN (w/F-Forcing)
0.12 (0.10) 2.97 (2.87)
Table 8: Performance of E-Forcing with different base net-
works on MNIST and CIFAR-10 10 in bits/dim (lower is better), training performance in brackets.
C.3 THE EFFECT OF DIFFERENT START EPOCHS OF E-FORCING
In addition, we have studied the effect of different start epochs of E-Forcing on the performance of language mod- eling, which can be seen in Table D From this, we may deduce that start- ing E-Forcing training at the 15th and 10th epoch yields the best results for Transformer- Base and Transformer- XL respectively, whereas starting ear- lier or later yields performance de- cline. It is reasonable because, if E- Forcing was introduced too early, the autoregressive model may not have been optimized well at that moment. As result, the quality of generation for the "negative phase" would be terrible, making energy-based based training unstable. On the other hand, the underlying autoregressive model can be modified only marginally if E-Forcing was introduced when the ARGM training is virtually complete.
Model
Start Epoch E-Forcing
Structure
5
10
15
20
25
Tr-Base
30.38
30.12 12
29.94
30.05
30.29
Tr-XI.
24.12
23.90
23.96
24.05
24.16
Table 9: Exploring the effect of different start epochs of E-Forcing Wikitext103 benchmark. Performances evaluated by perplex- ity (PPL).
17

---

Under review as a conference paper at ICLR 2023
BaseLine E-ARM
BaseLine E-ARM
crogs entropy loss
tra sarning
opoch
(a)
(b)
Figure 1: (a) Cross entropy loss curves on IWSLT14 Spanish to English translation task on training set. The blue and orange colors represent base model and E- E-Forcing respectively; (b) Cross entropy loss curves on IWSLT14 Spanish - English translation task test
C.4 ANALYSIS TO MODEL'S CONVERGENCE
In this section, we will investigate the convergence of E-Forcing. To begin, we first train a base Transformer model ("Tr-Base" architecture shown in Table on the IWSLT14 Spanish to English training set for baseline and E-Foreing method respectively, and then record the training loss and test loss (in cross-entron cross-entropy) at the end of each epoch. The loss curves are plotted in Figure From Figure we can see that (1) at the start the training, our F-Forcing converges slightly faster than the baseline. (2) As the training process progresses, the cross entropy of the baseline on the training set will gradually decrease,
faster rate than E-Forcing. On the other hand, the test loss curve of the baseline will fall initially and then slowly rise after 50 epochs while E-Forcing always remains stable convergence. This phenomenon also shows that our E-Forcing method can effectively prevent over-fitiee over-fitting so that obtaining better generalization.
C.5 ABLATION STUDY WITH DIFFERENT ARCHITECTURE CHOICES
Training Methods
GRU
LSTM
ENAS
DEQ
Tr-XL
TNet
Teacher-Forcing
92.48
78.93
58.60
57.10
54.55
54.19
F-Forcing 90.12
76.97
56.89
55.55
53.49
53.24
The ablation study of E-foreing over different choices of the architecture AR models with the comparison of vanilla fcacher-forcing training. We tested PPL scores using different AR models on the Penn Treebank dataset
Table 10:
In this section, we conducted ablation study investigate our E-Foreing model's generalization ability over different sequential models. We tested over 6 different sequential models, which are GRU (Chung et al. 2014). LSTM (Hochreiter & Schmidhuber 1997). ENAS (Pham al., 2018).
TrelisNet(TNE: (NET for short) (Bai et al.][2019] DEQ (Bai et al. 2019a) and Transtormer-2 XL (Dai
al. 2019) on Penn Treebank (Marcus al. 1993) dataset, which is relatively small dataset and widely used in machine learning for NLP (Natural Language Processing) research. In general, we can observe that our E-Forcing can achieve improvement over all base AR models applied, which indicates it is a universally applicable training method for AR models.
18

---

Under review as a conference paper at ICLR 2023
C.6 CASES STUDIES
To better understand the advantages of our method in correcting error tokens, we also prepare some translation cases in IWSLT14 German + EEglish, as shown in Table TT
Source Sentence(German)
Prodicted Target SenteneetEnglish)
wenn ict ihoca 600 zeitschiddenn zeige und siz in 10 kategorien anileile oder ich ihnen 400 zeitschriften zeige, und diese 20 kategorien sufleile, dunn glauben dass ihnen mehr unswah) und eine bessere wawahlerrfhrrnn pepeben habe, als ich ihmen dis 400 gegeben bitte gegentber dom, wenn ich thnen die 600 gegeben hime..
TruthdIruth: if show you 600 magazines and divide them up into 10 show you 400 magazines divide them up into given you chuice a better chucoing experience if gave you the 400 than if gave you the 600.
calegories, calegurriss you believe
Baseline: if show you 600 magazines and i aplit them 10 categoriies i'm showing them 400 magazines, and I'm going spáit them up into 20 casegories. you think I've given them more choices and better choice tham would have given them the 400 over the time that i gave them the 600.
give you 400 magazines can split categosies, then you think I've gives you more chosce and better selection than would have given you the 400 of which if gave you the 600.
Baseline S.S.. if show you 600 mugazines categories,
give you 400 magazines,
up
Ours: if show 600 magazines and divide them into categoriies or divide them given you more choices and better solection experience than gave you the 400 of whom if gave you the 600.
show you 400 magazines,
catogories, you think i've
und ich definiriv, für mich- sitution gefihtlich whir, anzufangen, diesen dunklen pfad versutunng sorusages henusterzu- sickern - besonders in dem umstand, in dem ich mich in meiner karriere gerale befinde.
GroundTruth: and definitely know would very dangeroms for me start sort of leaking down that dark path of assumpcioo, particularly given the circumstance that I'm in right now in my career.
my in my situation it
Baseline: and i know definitely, for would be very dangerons begin to this dark puth of suspect- expecially the circumstance that i'mm in my cancer right now.
Baseline S.S.; and i know definitely would be dungerous to to kind settle down dark puih presumption expecially the circnmetance in which I'm my catter right now.
situation very
Ours: and definitely know that it's for me- in my sitution very danger- stirt sickle down this dark puth suepection, in particular, the of where i'm my
right
wir hiben das licht ausgeschaltes, logten es eis vakuum und saugten die ganze Juft aus und kublien es bis fast zum jecat, ganz alleine im sufiig. war stick fiei, verhalton wie wolle..
Ground Truth: we camed off the lights, and then we pur it a vacumm and sucked out all the air, and then we cooled it down new, all alone in the elevator. little chank metal is
wanted.
Baseline: we turned the light off, put a vacuura and sacked out the ur und cooled it up unill almost now, all the way alone, the piece metal was open hehove it was.
vacuum, and caoled all the way up to now, all over the place, the piace of metal was free behave whatever it wanted
Baseline S.S.. tumed the lights off, pur it
sncked
Ours: we tamed off the lights, we into vacomm and we sucked all the coeled up almost now, all alose the elevator, the piece
metal was free to beluve whotevvr wanted.
und im grunde kumnen sie das betrachien, wissen sie, als eine tyrancci des crin- nernden solbst, und sie konnen sich das erinnernde selbst denken als das souusapen das erlebende selboi schleept durch erfahrungen, die das erlebende selbot nicht brancht.
Ground Truth and busically you can look at this, you know, as a tyranny of the remembering self, and you can think of remembering self experiencing dhoouh experiences need.
dragging the experiencing self
Buseline: and busically, you can think of this, you know, as a tyranny of self. and you think the memorable self as that kind weaves the living self through experiences that don't
the life itself.
Buseline- S.S. and busically, you can look at this, you know, a tyrannei of memorial self, and you can chink of the memorial self one that kind of sucks living self through experiences that don't need the living self.
Ours: and busically. you can look al that, you know. as tyrancy remenbesing self, and you can think of the memary itself is one, which sout of dragging the living self through experiences that the living self doesn's need. Ground Truch: we're and yet i actually think we'dl have to go buck hundreds, 300 years, before the enlight infighhtmment, to find a time when we buttled progress, when we fought about fronts, than
sind in schoelie erstaunlccien, esstunliches ereigaissen anf vielan gebieten. und doch danke ich wirklich, does wir hunderte, 300 juhre vor die aufklirang zurlick gehen mitsteen, um eine zeit zu findes, in der wir fortschiitt bekämmpft haben, der wir über diese dinge beftiger getritien baben, mehr fronten jezzi.
the verge amazing, amozing
many fields.
these things more vigoruusly,
Baseline: we are at threshold of amazing, amazing events in many oreas, and yet i really think that we have go back hundreds and 300 years before the enlighienment to find time when we have fought progress which driven
things
Buseline S.S. we's the threshold amaring, amazing events in many mreas. and yet, i really think that we hare to go back hundreds and hundreds. of before the enlighsenment to find time when have struggled with progress which we have driven on things more powerful, than now.
fronts
Ours: we' 're the threshold to amazing. amazing events in many areas, and yet really think thai have back hundreds and 300 years before the enlightenment find
frught progress, where we've 'vee fightng about these things to more fronts than we have now.
Table 11: Translation cases on IWSLT14 De-En test set, generated by the bascline method, baseline with scheduled sampling and our E-Forcing. The italic font means the mismatch translation
19

---

Under review as a conference paper at ICLR 2023
C.7
EVALUATION WITH OTHER METRICS
Scheduled Sampling
E-Forcing Training
Metrics ROUGE-LT
Trans. Pairs
ROUGE-1
ROUGE-21
METEOR
BLEU
66.51
43.69
63.69
64.35
34.61
De En
66.83
44.08
64.02
64.61
35.10
v
v
67.46
44.77
64.78
65.13
35.36
62.18
32.29
It En
64.73
40.97
61.94
62.51
32.64
65.27
41.51
62.49
62.80
32.82
71.10
49.47
68.78
68,
40.64
Es + En
49.53
68.96
69.28
40.91
71.91
50.17
69.65
69.63
41.58
Table 12: Comparison of ROUGE-I, ROUGE-2, ROUGE-L, METEOR, and BLEU scores between our approach E-Forcing and the base ARGM trained just with cross-entropy loss three translation pairs of IWSLT14 datasets, The value expressed percentage. use "Tr-Base" as the network architecture.
To further evaluate the effectiveness of our proposed E-Forcing, we also evaluate our method by using other metrics, such ROUGE Lin (2004) and METEOR [Banerjee & Lavie (2005) for neural machine translation. The results are shown in Table 12 In Table [2] the improvements of E-Forcing in different metrics consistent with the conclusion Table 2 which further prove the effectiveness of our E-Foreing method.
20


Reviews:
[{"review_id": "tIwPhXi3Iil", "reviewer": "Reviewer_W653", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for training autoregressive generative models in a way that casts these models into Energy-Based models. In essence, the method uses the logits from the Categorical output distribution of an autoregressive model to define an energy function; this is the basis for the subsequent treatment. As maximization of the energy function entails a contrastive divergence type of sampling, which may prove ineffective, the authors provide an importance sampling-based variant. \n", "summary_of_the_review": "The method is interesting, and the recasting of the categorical output distribution into the potential function of an energy-based model is nice. The training algorithm entails contributions of some merit. \n\nOn the other hand, the experiments are not convincing at all; they do not elaborate on computational costs required for the extra accuracy, and do not perform an ablation study with different architecture choices which may affect performance. ", "strength_and_weaknesses": "Pros: Interesting idea, extensive experimental evaluation against the standard baseline, correct derivations. \nCons: The method is a minor extension upon existing techniques; the authors do not explain how more computationally demanding this method is compared to the baselines, and whether these extra costs are worth the extra accuracy. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XToTURAEHt", "reviewer": "Reviewer_FpT4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "AR models are very common in many domains, especially language models. Maximum likelihood is the widely used approach for training these models. However, ML training of AR models causes some other issues such as exposure bias. Techniques such as scheduled sampling have been introduced before to address these issues. \nThis paper introduces a new training algorithm for AR models by viewing it as an energy-based model.  \nHere the authors define the model probability as $p(x_1, \\cdots, x_k) = q(x_{<k}) \\frac{e^{-Q(x_k, x_{<k})}}{Z_\\theta}$. They defined $Q(x_k, x_{<k})$ using the logits before the softmax of the $k$th token, so $p(x_1, \\cdots, x_k) = q(x_i, \\cdots, x_k)$ (am I right?). \nHowever instead of ML training which normalize $e^{-Q(x_k, x<k)}$ with $Z_\\theta$, they are training $p(x_i, \\cdots, x_k)$ using contrastive divergence (CD). This formulation helps to better regulate the distribution on the $k$th softmax since CD training contrast $Q(x_k, x_{<k})$ with a sample from the model (using importance sampling given the introduced form for the energy) rather than the log partition function. \nThe final loss function is a combination of cross entropy and CD.\n\nThe authors study the proposed method for different tasks of language modeling, machine translation, as well as image generation.\n\n", "summary_of_the_review": "The paper is interesting but the experimental results do not show considerable benefits for the approach. I don't think that the proposed training algorithm would be adopted by the community for training AR models. \n", "strength_and_weaknesses": "\nStrength:\n-The paper is well-written and easy to follow.\n-The idea of viewing AR models as EBMs is not novel but the provided formulation is neat and interesting. \n-I like the extensive study of the proposed approach.\n\nWeakness:\nThe improvement especially in the translation tasks is not considerable. Other training algorithms on top of ML training of AR can achieve similar improvement, for example, combining it with RL training. \nBhattacharyya et al. (2021) reported a significant improvement over base AR-NMT by using an external EBM defined over the whole sentence (although some of the improvements come from using pre-trained language models).  \nAlso, their training algorithm can directly be applied to AR by viewing E as \\sum_i E_i. \n\nQuestions:\n1) The author claims this approach can help with long-range dependencies, but I am not seeing how defining the energy model using the last softmax can help with that. Is that the effect of weights on training log q(x<k)? Even so, it is not similar to defining an explicit energy model over the whole sentence! \n\nBhattacharyya S, Rooshenas A, Naskar S, Sun S, Iyyer M, McCallum A. Energy-based reranking: Improving neural machine translation using energy-based models. ACL 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K-u53XvvHx", "reviewer": "Reviewer_nMbm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use the softmax logits in the output of discrete autoregressive models as an energy function, converting an autoregressive model into an energy-based model. They then use contrastive divergence training, using an importance weighted estimate of the “negative phase” gradient. Through experiments on language modeling and translation tasks, they show that this results in more coherent and in-distribution samples, as compared with autoregressive baselines.", "summary_of_the_review": "The paper presents a fairly general and sound technique for improving autoregressive models by training them as energy-based models. Results are demonstrated on several text datasets, comparing with relevant baselines. In total, the generality of the idea, combined with the paper’s convincing empirical demonstration, leads me to suggest acceptance.", "strength_and_weaknesses": "**Strengths**\n\n- **Relatively simple/general technique.** The basic idea of the proposed technique is to identify the softmax logits of an autoregressive model as an energy function, allowing them to replace the typical maximum log-likelihood training procedure with a contrastive-divergence-based training procedure. This is reminiscent of previous methods that have interpreted classifiers as energy-based models (e.g., Grathwohl et al., 2020). In this way, the model is explicitly trained by decreasing the likelihood of generated samples, hopefully reducing the effect of “exposure bias.” Notably, this procedure can, in principle, be applied to any discrete autoregressive model, making this a rather general technique.\n\n- **Experiments on multiple datasets with relevant baselines.** The authors demonstrate their proposed approach on language modeling and translation tasks, with three language modeling datasets and one translation dataset with six translation pairs. The authors compare their approach with the baseline autoregressive model, as well as Residual EBM, another method that combines autoregressive and energy-based models. The proposed approach generally compares favorably.\n\n- **Clear description, for the most part.** The descriptions within the paper are generally clear, with both text and mathematical terms well-defined. One way to improve the clarity of the paper even further would be to include some form of diagram.\n\n\n**Weaknesses**\n\n- **Additional computational overhead for sampling.** One downside of employing a contrastive divergence-based training method is that it requires sampling during training. Sampling from autoregressive models is typically quite costly, and I would guess that this significantly impacts training speed (although this isn’t discussed in the main paper).\n\n- **Only applicable (at least currently) to discrete models.** The current method, as presented, is only applicable to discrete autoregressive models. While this encompasses a wide variety of applications, the authors have not demonstrated how to apply this technique to continuous autoregressive models, e.g., those that output Gaussian densities. In this sense, the title is a bit misleading, as the technique does not apply to all ‘autoregressive models.’\n\n- **Only demonstrated on text data, i.e., no images or audio, etc.** The authors only demonstrate their technique on text data (language modeling and translation, i.e., conditional language modeling). However, even with discrete autoregressive models, this technique could be readily applied to image and perhaps audio modeling. Currently, the paper may only appeal to a subset of the machine learning community focused on language, but with these added application areas, the paper may receive wider attention.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject