YOUR AUTOREGRESSIVE GENERATIVE MODEL CAN BE BETTER IF YOU TREAT IT AS AN ENERGY-BASED ONE
Authors: Yezhen Wang
Tong Che
Bo Li
Kaitao Song
Hengzhi Pei
Yoshua Bengio
Dongsheng Li
Venue: ICLR 2022 Submitted (2022)
Keywords: autoregressive generative model
exposure bias
energy-based model

Abstract: Autoregressive generative models are commonly used, especially for those tasks involving sequential data. They have, however, been plagued by a slew of inherent flaws due to the intrinsic characteristics of chain-style conditional modeling (e.g., exposure bias or lack of long-range coherence), severely limiting their ability to model distributions properly. In this paper, we propose a unique method for training the autoregressive generative model that takes advantage of a well-designed energy-based learning objective. We show that our method is capable of alleviating the exposure bias problem and increase temporal coherence by imposing a constraint which fits joint distributions at each time step. Besides, unlike former energy-based models, we estimate energy scores based on the underlying autoregressive network itself, which does not require any extra network. Finally, thanks to importance sampling, we can train the entire model efficiently without requiring an MCMC process. Extensive empirical results, covering benchmarks like language modeling, neural machine translation, and image generation, demonstrate the effectiveness of the proposed approach.

Under review as a conference paper at ICLR 2022
YOUR AUTOREGRESSIVE GENERATIVE MODEL CAN BE BETТER IF YOU TREAT IT AS AN ENERGY-BASED ONE
Anonymous authors Paper under double-blind review
ABSTRACT
Autoregressive generative models are commonly used, especially for those tasks involving sequential data. They have, however, been plagued by a slew of inherent flaws due to the intrinsic characteristics of chain-style conditional modeling (e.g.. exposure bias or lack of long-range coherence), severely limiting their ability model distributions properly. In this paper, propose unique method for train- ing the autoregressive generative model that takes advantage of a well-designed energy-based learning objective. We show that our method is capable of allevi- ating the exposure bias problem and increase temporal coherence by imposing a constraint which fits joint distributions at each time step. Besides, unlike former energy-based models, we estimate energy scores based on the underlying autore- gressive network itself, which does not require any extra network. Finally, thanks to importance sampling, we can train the entire model efficiently without requiring MCMC process. Extensive empirical results, covering benchmarks like lan- guage modeling, neural machine translation, and image generation, demonstrate the effectiveness of the proposed approach.
INTRODUCTION
By factorizing the joint distribution into the product of a series of conditional distributions, autore- gressive generative models (abbr. ARGMs) (Vaswani al.! 2017 Dai et al., 2019: van den Oord et al. 2016a b) Salimans et al.) 2017 Chen et al. 2018) simplify the difficult challenge of modeling high-dimensional joint distributions. They can be trained efficiently via maximum likelihood and generate samples of exceptional quality, making this technique popular for modeling distributions, especially for sequential data. Nonetheless, despite their potency and flexibility, ARGMs still have inherent weaknesses due to the intrinsic characteristics of chain-style conditional modeling. For example, ARGMs usually suffer from discrepancy of the input context distributions between the training and inference stages, which causes consequent error propagation (i.e., Exposure Bias (Ran- zato et al. 2016; [engio et al.) 2015)). Besides, due to the nature greedy selection of beam search approximations, the decoded results from ARGMs may also lack in longerange coherence. We consider one approach by which ARGMs could be adapted to reduce these concerns.
Earlier work, both heuristic and theoretical, has already been proposed with those goals. For in- stance, the exposure bias problem of ARGMs can be alleviated to some extent with scheduled sampling (Bengio et al. 2015) Mihaylova & Martins 2019), by mixing input contexts from both real data and autoregressive generation, during the training stage. However, this scheme suffers from over-correcting problem (Zhang al.. 2019). In addition, at the inference stage, beam search makes it possible to choose more diverse candidates, improving the quality of generated se- quences. Nevertheless. this results in only marginal improvements temporal coherence, since ARGMs can only leverage previous decoded contexts without consideration of the whole sequence information. Moreover, setting aside the difficulty in training them, energy-basee based models (EBMs) have demonstrated their effectiveness in modeling chine learning applications (Zhao et al. 2017; Arbel et al. 2022 Gao al. 202T], without requiring the transformation of the target distribution into a product of conditional distributions. As result, several studies (Deng et al. 2020; Bakhtin et al. 2021 I Durkan Nash 2019) attempt to combine EBMs with ARGMs, expecting benefit from the strengths of both approaches. However, though some positive results were obtained, the existing works preferred a two-stage optimization, which first obtains well-trained ARGM and then trains an additional EBM based on it. Such optimiza-
-ige--imensional distributions in a variety of ma-

---

Under review as a conference paper at ICLR 2022
tion strategy does not enable ARGMs to benefit from the properties of EBM in modeling the joint distribution in a temporally more coherent way.
In this paper, we present novel design for seamlessly integrating Energy-based models into AutoRegressive Models (E-ARM). Our training is based on an energy-based learning objective, which forces ARGMs training fit the joint distribution along with the conditional one at each time step. Thanks to our well-designed energy function, the two involved models can share a single base network without additional parameters, that is, the base network not only serves as a generator that provides fake data to facilitate the training of EBMs like previous works (Che et al., 2020 Xiao al., 202T] Durkan & Nash, 2019 Deng al. 2020), but also plays the role of modeling the en- engy surface. This property makes it easy to plug E-ARM into the training of any autoregressive
generative models.
Intuitively, the exposure bias in ARGMs is caused by the fact that the model is trained on real data rather than data generated by the model. On the other hand, the EBM's optimization process for modeling joint densities, the negative phase of wake-sleep algorithms (Hinton) 2002 Kim & Bengio 2016) requires sampling data from the EBM itself. Along with the fact that our method combines the EBM and the ARGM seamlessly as a whole, E-ARM can reduce the discrepancy between input data of the training and inference stage, which mitigates the exposure bias problem of the ARGM. On top of it, unlike ARGMs, which factor the joint distribution into a product of conditional distributions, EBMs are able to model the joint distribution directly and score each input the sequence level instead of at the token level, which makes them capable of modeling long- range coherence. Additionally, in order to optimize the proposed energy-based learning objective efficiently via gradient-based wake-sleep algorithms (Kim & Bengio 2016), we present way to estimate the negative phase gradient (which is a necessary component in the gradient-based wake- sleep algorithms) through those samples generated with the autoregressive view instead of the EBM view, which would require an expensive Markov Chain Monte Carlo (MCMC) process. This allows us to sidestep extremely time-consuming MCMCs, thus accelerating training.
In summary, the following contributions are made with this paper: i) We introduce novel scheme, E-ARM, to integrate the EBM view into autoregressive generative models seamlessly; ii) we attempt to reduce the intrinsic problems of autoregressive models, such exposure bias and weak temporal coherence, by optimizing an energy-based learning objective, which uses samples autoregressively generated; iii) We demonstrate how to efficiently optimize our model constructed from single network, using wake-sleep algorithms without MCMC; iv) In a number of applications, such as language modeling, neural machine translation, and image generation, our model can achieve better results in comparison with relevant baselines.
BACKGROUND
2.1 ENERGY-BASED MODELS
Energy-based models LeCunn et al. 2006 can express any probability p(x) for x E RK as
(x(x) exp( - -Ex(x))
(I)
Zo
where Eg : D - Th. denotes an energy function which aims to map a D-dimensional datapoint to scalar, and Z(0) Σx exp( exp(-E((x)) Ex(x)) denotes the normalizing constant, also known as the partition function. Any function can be used as an energy function to represent an EBM as long as it can generate single scalar given some input x and the normalizing constant is finite Wake-sleep algorithms are commonly used to optimize EBMs (Hinton 2002: Kim & Bengio 2016 Grathwohl al.) (020)) via gradient- based approximate maximum likelihood. Specifically, the gradient of the log- logllikel,oood, which needs to be maximized, with respect to 0 can be expressed as
E Ex) 00
log po(x)) (x)
Eo Ex(x
Epp(())
Eg Eo(xx)
08
00
Without constraining the parametrization of Eg, this can be achieved by bounding the region of space in which x takes its allowed values.
2

---

Under review as a conference paper at ICLR 2022
The first term in the right hand side of Eq. 2 is the negative phase term while the second term is called the positive phase term. MCMC methods have been used (Hinton) 2002; WWeliig & Teh, 2011a) to approximately sample from (xx).. for estimating the negative phase term.
2.2 MODELING DISTRIBUTION AUTOREGRESSIVELL
can decompose any joint distribution p(x) into a prod- uct of conditional distributions using the product rule of probability by ordering those random vari- ables within the joint distribution and characterizing each random variable given all variables pre- ceding it in that order. Formally, we use X<k to denote the vector variable covering all random variables before the time step & and use denote the random variable at time step k. Then we have
Autoregressive generative models (ARGM 2
K
p(x)= II p(xx(x<))
(3)
In general, modeling distributions autoregressivell has achieved remarkable accomplishments in numerous areas (Vaswani et al., 2017 Radford et al. 2019) van den Oord et al. 60 (Salimans) itt al. 2017)) thanks to its ability to avoid the challenging target of modeling joint sigh--iimnssoonl distributions directly. We primarily focus on autoregressive language models in this paper, but also conduct experiments on image generation to validate the generality of our method.
2.3 EXPOSURE BIAS AND INCOHERENCE PROBLEMS IN AUTOREGRESSIVE MODELS
In the discussion about the defects sequential autoregressive generative models, the exposure bias problem (Bengio et al. 2015 Ranzato et al.) 2016) is an important issue, which greatly affects the model's deployment performance. During training stage, the autoregressive model always conditioned on ground truth token sequences. In generation stage, however, the model has to rely on its own previously generated tokens to predict the next token, when the model is deployed. an incorrect token is selected, this error can be amplified in following steps because the next prediction will be made using an unusual input (one unlike those in the training set). Besides, out of consideration of efficiency, autoregressive decoding usually selects the most probable token each time step, given the ones previously selected. Such a scheme assumes the largest joint probability of be achieved by separately choosing the most probable next token (given its
the whole sequence previous context) over all time steps, which is only the local optimum. Correspondingly, the chosen sequence can not always be the model's optimum result.
INTEGRATE EBMs INTO AUTOREGRESSI MODELS SEAMLESSLY
For long time, result of compromises for improving training stability and efficiency (e.g., modeling a joint distribution by decomposing it and using a teacher- forcing training strategy), con- ventional autoregressive generative models have suffered from flaws such as the exposure bias and the lack of long-range coherence. To tackle these issues, we attempt to seamlessly integrate Energy- based models into RutoRegres Models (E-ARM), which can be regarded toregressive generative models blending with an energy-based learning objective. Given joint sequential distribution, E-ARM also addresses autoregressively. that is, tackling tokens step by step under a specific order. However, what differs from conventional ARGMs is that we attempt model both the conditional and the joint distributions simultaneously each time step. In this way, E-ARM can model distributions conveniently in autoregressive manner while avoiding those potential problems brought by ARGMs.
variant of
Formally, given a sequence of random variables (æ1, 22, FK) with length K, we introduce parametric autoregressive model (x(x)))) II qo (x((x=) (k denotes the time step) with pa- rameters 0. Particularly, we define qe (Xxx) ) = Ii (x(x))) II q(I, Xxw),, which means only those conditional distributions @o xx)) ) of the most recent k time steps are involved in the current update of parameters 0 while those distant conditional distributions q( are
In this paper, the term "autoregressive model" is sometimes used to denote the autoregressive generative model for convenience.
3

---

Under review as a conference paper at ICLR 2022
treated fixed (The rationale behind such design will be elaborated in See Then, define Po(Tk, as a product of the autoregressive model and an EBM as follows,
(o( (o(x<*)
(4)
Zo
where the energy function (T X<ck defined the 2y 's negative corresponding component of the base network's & output logit with the input prefix context X< ( ........: #k-11 (e.g., given a sequence "This Friday." and assuming the corresponding index of the token "Friday" in the vocabulary is i, then the value of ("Frddrda,"", "This is") is the i-th component of the output logit, which is the straight input tensor of the final softmax layer), and the normalization term Ze
Ee, e 2x)))
Ex
Our primary goal is to make the distribution go (<)))<<))))))) ) to approach the real conditional pa (x*)) whilst maintaining (r tc)) as close the real joint pa(a X.ck as possible at each time step, which can be achieved by minimizing the Bullback-Leibbee (KL) divergence between the distribu- tions,
K
+ AD (pa(Fx, XCx (ex))lp((r (x))) x. (5)
0 = arg min Σ
D (pa(ta/x<a))loooer)
where À adjusts the ratio between the two objectives. In Eq. the first objective at each time step can be easily optimized by cross entropy while the second objective is optimized in the sense of EBMs by wake-sleep algorithms (Hinton et al., 1995| Kim & Bengio (0T66), which minimizes the objective by descending the following gradient of ( according to Eq.
Ex Bxxxxcerpaateeee xex)
20
Ea Ex (IR, (xx)
(12,
30 Eo xcc)
(6)
Positive Phase
Negative Phase
where we have Ex x<*) = do (T&,, Xck) - log (x(xx<)) Optimization via Eq. 2 or 4 involves sampling data from the model and thus lead to the discovery of non-data-like samples, whose likelihood is then explicitly reduced by the energy function. E-ARM is therefore not plagued by the exposure bias problem. Besides, because model the joint distribution throughout the training process, E-ARM can assess the entire sequence as a whole and generate more coherent data using energy sampling (Deng al. 2020).
OPTIMIZATION
In this section, we present how efficiently optimize E-ARM. To begin with, we optimize the first objective in Eq. [5 as with conventional autoregressive models by reducing the per time-step cross- entropy loss. As for the second objective, resort to descend the estimated gradient shown in Eq. 6 Thanks to the importance sampling technique and our well-defined energy function, we now show that an improved version of Eq, has simple and symmetric form that can be easily estimated whilst not requiring an expensive MCMC.
Specifically, by replacing Eg (#&, x<ck) with the specific form de (IR, xck) log go (xx)), the gra- dient w.r.t. 0 in the positive phase of Eq. ca be written into
-Exex ppl log qo (x=x)] + Ex xxxx 00 (I xxx)].
D0
00
Similarly, we can get the negative phase gradient
9 d log qo (x(x))) + ExL. ro (Tk, Xx+)].
θ
Excerpol
(8)
00
The first term -Exes~pu as log qo (x)))) in Eq. equivalent the log-likelihood gradient of qe (x=)), which means improvements in this direction will be automatically taken care of as a re- sult of steps arising from the gradient of the first KL-divergence in Eq. 5 albeit at the expense of
here, we take minimization version of the Eq. Thus the sign before each phase is converse.
4

---

Under review as a conference paper at ICLR 2022
changing the weight given to the second vs. the first KL, X. Besides, because the estimation of the expectation operator over the data distribution Pa is easy, and the score ( ( can be acquired simply accessing the output logit of ARGM (see the definition of Φn in Sec. B}, the second term can likewise be readily estimated and optimized. As a result, the positive phase optimization is both feasible and efficient.
The negative phase gradient estimation, on the other hand, is more involved. In Eq. g sampling data from pa is required for estimating the expectation Egg, whereas pa parametric joint probability involving an energy-based unnormalized probability estimator that may require time-consuming MCMC methods to generate data. However, thanks to importance sampling, we can substitute the troublesome computation of the expectation over the distribution Pe with the expectation over the distribution qo. which can generate samples autoregressively without MCMC. Formally, the 0 (I x=+)] is equivalent to the following formulation (See
negative phase gradient IE x a the detailed derivation in Appendix
2 log (x(xxx)] + E Bes (w(xx)) 00
-x Poxxxmo((Xex (x (w(x<*)
to (#x, x<k)]; (9)
00
Em E p(zeiece)
where w(xck)= =
(10)
Bxxandolxet) Em coooxxxee)
According to Eq. all the estimated expectations only need sampling from the autoregressive model rather than the joint distribution, and the reweighing weight w in Eq. I0 also does not involve expectation computation over distribution po- Generally, producing data from autoregressive model is a very simple ancerstral sampling process, as compared with sampling straight from an EBM, which needs MCMC approaches (Durkan & Nash 2019). On account of that, the optimization process can be much more efficient.
Besides, the term (xxxx=(((x=x)) (xxx)) 2o log (X<x)] in Eq. 9 is equivalent to re-weighted version of the gradient of qs is information entropy with respect to 0. This term can be optimized similarly to the faaherr-frreing training of autoregressive model with the "teacher" sequence gener- ated autoregressively by the model itself. Actually, the scheduled sampling methods (Bengio et al.) 2015 Ranzato et al. 2016 Mihaylova & Martins 2019) are similar to this term but without the re- weighting factor. Furthermore, it is worth noting that for a sequence with total length K, since we add constraint to fit the joint distribution pe at each time step k. Eq. 9 actuully has K counterparts with different time steps. If we use the qo (xxxx) directly instead of go ( <k) in the Eq. 4 to define (rrk xx),,, due to the fact that the distribution go (x<x)) modeled by an autoregressive model can be naturally broken up into pieces, simply summing up these K gradients results in the term
K Bxpoxxee) [[(x
K K+1-1
① log go (x2x)]
=
(xx( (w(x<x) log go (x) (11)
00
00
where indicates the specific index of the current token in the entire sequence. As result, earlier time steps (smaller 1) will get stronger training signals (larger K + 1 - I, indicating more gradi- ent terms), giving rise imbalanced training for different time steps. To solve this, we introduce
ge (xxxx) as I qo (x(((x=:) Imm=1 q(mm, ) to define p0( x<x) shown in Sec. allowing gradients only back propagate through conditional distributions w.r.t. a few recent tokens This explains our proposal of using qe (x))k) to define po(Ik; Xck).
Ultimately, combining Eq. and Eq. g at each time step kt we can optimize pe( x<k) via descending the estimated gradient of 0 as follows,
i 20
i
Exer~pal
log qo (xx*)]
Ex Exer~p(xxe)) [
log qo (x<+)]
00
i
+ Emg, 2 do (k ck)
Exx Xex do x (w(x////////// 2 (Ik, xxx)
00
00
Positive Phisse
Negative Phase
(12)
'In practice, we find that using recent tokens worked best.
5

---

Under review as a conference paper at ICLR 2022
that the only difference between two phases is that in the negative phase, the expectation over de have a reweighing weight w for each sample. The reweighing weight w in Eq. andddd Eq. 12 can be further refined (see the derivation in Appendix B and we can observe that
From Eq. 12
x((x)))
w(X<))
(13)
Ex (((xER)
where (((xx))= = most likely come from, the distribution pe or the distribution qe. Correspondingly, w(x<*) reflects
pa(x)))
[x(x=(] indicates the possibility of which distribution the prefix context xe is
the context X<x's s relative magnitude of μ(x²x) compared with the average among all potential contexts- the larger the value of w(x the more likely the context x in the data space coming
from po. which is modeled by the product of autoregressive models and EBMs. During training, those input sequences with contexts more likely under Pe than qo will be assigned larger weights w while others will be assigned smaller weights w.
In general, E-ARM ought to be viewed as a new learning pattern for autoregressive models that en- sures our base autoregressive network stays close to the real distribution pa. We found that training from scratch with the energy- based learning objective of in Eq alone did not work well. The reason is that at the initial stage of the training process, what we have is just randomly initial- ized autoregressive network which outputs sequences with random values given any context. This indicates disjoint supports between the real sequence's s distribution pa and distribution po modeled by ARGMs. If we only use the energy-based learning objective of Eq. 12 the whole gradient Ep,(x)]00 & log (x)] in Eq 2 would be 0 due to disjoint supports between pa and pg. As result, in order to make the optimization more feasible, we must maintain the cross-entropy loss low through- out training and pre-train as a pure ARGM for a few epochs before introducing the E-ARM objective. Actually, the starting epoch of E-ARM is a hyper-paree prree-pprraeter, and we discuss it in the Sec. 5.2
Following the excellent work of Deng et al. (2020); Bakhtin al. (202I) I. we also adopt Top-K energy re-sampling in the inference stage, which means that in the generative process, we first gather multiple candidate sequences generated autoregressively, and then re-sample from them based on their energy scores estimated by the network's 's logit at the last time step where the entire sequence has been processed. Since we employ the EBM to model the joint distribution at each time step, such re-sampling strategy mitigate the undesirable impact of the greedy selection of one token at a time, and we found this variation to increase the coherence of generated samples.
5 EXPERIMENTS
To empirically corroborate the effectiveness E-ARM and show its broad applicability, we con- duct extensive experiments covering three machine learning applications, which are neural machine translation (NMT), language modeling, and image generation. In this section, we will introduce the three corresponding experimental setups, followed by an analysis of the obtained results. We will release the source code once upon acceptance.
5.1 APPLICATION TO NEURAL MACHINE TRANSLATION
E-ARM is first evaluated in the context of neural machine translation (NMT), which is conditional generation task and is important in the natural language processing (NLP) field. We first analyze E-ARM on the IWSLT14 dataset, which includes six different language pairs ([German, Spanish, Italian) > English and English - {German, Spanish, Italian)). In addition, we test E-ARM on the WMT16 (English -> German) benchmark to make sure we evaluating E-ARM on larger dataset. Hereafter we abbreviate English, German, Spanish, Italian as "En", "De". "Es", "It". The weight Eq. 5 is set as 0.05 for all translation tasks. We use one size of transformer ( ("Base-IWSLT") for the IWSTT14 benchmark and two sizes of transformer ("Base-WMT") "Large- MTT") for the WMT16 benchmark [5] Scheduled Sampling is carried out following Mihaylova & Martins 2019).
The results of IWSLT14 tasks are shown in Table I We test not only the pure performance of E-ARM but also the compatibility with other techniques. Specifically, we can observe that (1) without any particular engineering, E-ARM outperforms the base autoregressive translation model
'The implementation is developed Fairseq OOt al. 2019

---

Under review as a conference paper at ICLR 2022
Label Smoothing
Schedmled Sampling
Beam
BLEU
Model
Arg.
Searching
DE- EN
EN- DE
EN- IT
IT EN
ES- EN
EN-
32,4410.06
26.6410.10
27.92.10.03
30.481008
38.6110.11
35.42.0.09
31.92
33.6210.07
28.72.10.44
33.68:0046
27.62
28.81
31.42=007
39.85.1113
36.71
33.02
34.61.1.00
28.4610.06
29.72.10.10
32.29110.3
40.6410.077
37,48.0005
33.87
34.23:10.06
27, 27.96.10.03
29.26:10.11
31.93
40.16:1..03
37.21
33.46
V
35.10:41.44
29.97=0007
32.64
32.99+0.10
27.15
28.33=0.12
31.13+004
39.5 39.56+1..11
36.07=0.02
32.54
5
34.06-+:0.06
07.97+0.00
29,26+00009
31.90 +0.11
40.30 +0.03
36.92 =0.09
33.40
E-ARM
33.97 1:0.08
28.03 10004
29.13 10002
31.84 1:0.11
40.32 10.03
36.96 00.07
33.38
5
34.93 10.03
28.91 10.12
30.04 10.11
32.56 10:01
11.01 10.06
37.73 1..12
34.20
37.56
35.36 1.0.55
29.11
1.0.4
30,25
10.09
32.82 10.11
41.58 10007
38.19
00.0
34.55
Table 1: Comparison BL.UU scores between our approach E-ARM and the base ARGM trained just with cross-entropy loss on six translation pairs of IWSLT14 datasets. We use "-" to denote that the training trick is not used while "v" indicates use it. "5 B" represents we use beam searching with 5 beams.
trained with cross-entropy singly by 0.62 (31.92 - 32.54) BLEU points in average, especially three translation
- 39.56 on Konnishtto-Englihhe 30.48 - 31.13 Italian-to- English, 35,42 - 36.07 on English-to-Spanish (2) E-ARM compatible with other techniques extent. They are not mutually exclusive and can work together to further improve the performance of the base ARGM. (3) However, since scheduled sampling can reduce exposure bias and beam search can somewhat alleviate the flaws caused by greedy selection at each time step, the performance gain of E-ARM when all these tactics combined is only 0.34 (34.21 34.55), which is lower than the 0.62 (31.92 + 32.54) obtained when the model is purely trained without these other techniques.
like scheduled sampling. which help alleviate the exposure bias problem to
Additionally, Table 2 shows the per- formance of E-ARM on the WMT16 English - German task. For two different model sizes, enabling bel smoothing (L.S.) improves model performance by 0.52 and 0.35, re- spectively. The performance of the base transformer model further in- creases to 28.36 BLEU points when scheduled sampling (S.S.) is used, while the larger model improves to 29.23 points. E-ARM paired with label smoothing and scheduled sam- pling yields the highest scores of 28.62 and 29.44, respectively. Over- all, our training strategy outperforms ARGM's vanilla teacher-forcing training and can have uniformly favorable impacts across different models and dataset sizes.
Model
L.S. S.S. w/E-ARM
BLEU
27.56
28.04
Base-WMT
28.36
28.62
28.70
29.05
Large-WMT
29.44
Table 2: Translation performance of proposed E-ARM WMT16 Anglish->German, evaluated BLEU, We uniformly use 5 beams when applying beam search. "L.S." denotes Label Smoothing and "S.S." denotes Scheduled Sampling.
5.2 APPLICATION LANGUAGE MODELING
To further demonstrate E-ARM's consistency in reducing flaws of autoregressive generative models, we also conduct language modeling experiments. The WikiText-103 dataset (Merity et al.) 2017). which the largest word-level language modeling benchmark with long-term dependency, was chosen as the testhed. It comprises 103 million
Model
iPaaras
PPL
Tr-Base
156M
30.56
Tr-Base (w/E-ARM)
156M
29.89
Standard Tr-XL
151M
24.20
Standard Tr-XL (w/E-ARM)
151M
23.81
Table 3: Language modeling performance of different models on WikiText Evaluation conducted using perplexity (PPL..

---

Under review as a conference paper at ICLR 2022
training tokens from 28 thousand articles, with an average length of 3.6 thousand tokens per article, which allows model to evaluate the ability of modeling long-term dependency. Two network structures are mainly tested, which are Transformer-Bar (Vaswani et al. 2017) and Transformer-XI. (Dai et al. 2019) (Tr-Base and Tr-XL for short respectively hereafter).
The final results are reported in Table 3] We can see from the results that E-ARM outperforms baselines with clear margins for different types of models. Specifically, the Transformer-Base improves perfor- mance by 0.67 PPL points (from 30.56 to 29.89), while the Transformer-Xl. improves model by 0.20 PPL points (from 24.20 to 23.81). Our strategy does not change the structure of the base network nor introduces any additional module or learnable pa- rameters, therefore we can conclude that the perfor- mance boost is solely from the introduced energy- based learning objective.
Start Epoch
15
25
0.00
30.56
30.56
30.56
0.01
30.48
30.12
30.22
A
0.05
30.43
29.89
30.16
0.1
30.60
30.03
30.14
0.5
30.71
30.36
30.47
Table 4: How different and the E-ARM start epoch (when we introduce the E-ARM into the training on WikiText103) affect performance eval- uated by perplexity (PPL). The Tr-Base model structure used and is train 40 epochs in total.
In addition, study the effect of hyper- parameetrr settings on the performance of language modeling, which can be seen in Table From this, we may de- duce that starting E-ARM training at the 15-th epoch yields the best results, whereas starting earlier later yields performance decline. It is reasonable because, if E-ARM was introduced too early, the autoregressive model may not have been opti- mized well that moment. As a result, generative quality would be terrible, and make energy-based training unstable. On the other hand, the underlying autoregressive model can be modified only marginally if F-ARM introduced when the ARGM training virtually complete. Besides, from the vertical perspective which presents the impact of different X, we can observe that the best À in Eq. is 0.05. The first line of the table indicates the bascline of training the autoregressive model with pure cross-entropy loss.
5.3 APPLICATION TO IMAGE GENERATION
In order to illustrate the effectiveness and generality of our method in processing different modality tasks, we further show the results of applying E-ARM to image generation in this section. We apply E-ARM to Pixel-CNN (Van Oord et al. 2016) and its variant Gated PixeI-CNN (Oord et al. 2016 Experiments are carried out the MNIST and CIFAR-10 datasets.
Test (Train) NLL.
Model
MNIST
CIFAR-10
Pixel-CNN
0.17 (0.13)
3.14 (3.08)
Pixel-CNN (w/E-ARM)
0.15 (0.12)
3.07 (2.98)
Gated Pixel-CNN
0.14 (0.11)
3.03 (2.90)
Gated PixeI-CNN (w/E-ARM)
0.12 (0.10)
2.97 (2.91)
Table S: Performance of E-ARM with different base
networks on MNIST and CIFAR-10 in bits/dim (lower better), training performance brackets.
Samples of CIFAR-10 from Gated Pixel-CNN (w/B-ARM).
Figure/ 1:
Table 5 summarizes the quantitative results measured by per-pixel negative log-likelihood (NLL), while Figure ddgure|deppicts depicts some of the generated samples. We can see that with the help of our E-ARM, both the Pixel-CNN and the Gated Pixel-CNN can obtain improvements in all datasets (0.17 + 0.15 and 3.14 > 3.07 for Pixel-CNN on MNIST and CIFAR10 respectively and 0.14 - 0.12 and 3.03 + 2.97 for Gated Pixel-CNN on MNIST and CIFARIO respectively). This is further evidence in favour of the energy-based learning objective for improving autoregressive models.
8

---

Under review as a conference paper at ICLR 2022
6
RELATED WORKS
6.1 AUTOREGRESSIVE GENERATIVE MODELS
Modeling nigh-dimensional data distributions directly is usually a rather challenging task due to "the curse of dimensionality" (Bellman 1954). One alternative method is to sequentialize the ran- dom variables and then factorize the joint probability distribution into the product of conditionals based on the sequence structure, which is exactly the core idea of autoregressive generative models (ARGMs).
ARGMs have been very successful, in particular for sequential data. For example, ARGMs have been widely used in language modeling (Vaswani et al., 2017: Dai et al., 2019] Radford et al. 2019), audio synthesis (van den Oord al. 20T6a), and even image generation (van den Oord al.
2016c b Salimans et al. 20)7). The advantages of ARGMs are however balanced by issues of (I) exposure bias (Ranzato al., 2016; [engio al. 2013; Song et al. 2020), due to the discrepancy in input context distributions between the training and inference stages, and (2) weak long-range coherence, due to the inherent greedy selection of one token at time without look-ahead.
6.2 ENERGY-BASED MODELS
In the field of generative modeling, energy-based models (EBMs) have been widely used (Zhao ctt al.) 2017| Arbel et al. 202T Gao et al. 202I). The primary idea behind EBMs is to decompose the dependencies between variables (e.g. images and labels) through different terms of an energy function, assigning low energies to proper configurations found in the dataset, while assigning high energies to incorrect or unseen ones (LeCun et al., 2006).
Due to the challenge of sampling from EBMs, training EBMs by wake-sleep algorithms (Hinton 2002 Kim & Bengio [016 Grathwohl et al. 202T), which require expensive MCMC approaches, has been notoriously difficult, especially on hig-ddimensional data like images or texts. Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Tch, 2011a) is a frequently used gradient- -based MCMC approach that injects noise into parameter updates and anneals the step size during the course of training, and which has been adopted in numerous prior works (Nijkamp al. [2019] Du Mordatch 2019) 2 Grathwohl al.) 2020). However, these gradient- MCMC methods require enormous extra computing overheads and are not applicable when the input is discrete like for text sequences (Deng et al. 2020).
As result, variety of recent works attempt explore the strategy of training an EBM without MCMC. In particular, [akkhtin et al./ (202T); Xu et al. (2021a); Gao et al. (2020) optimize the EBMs by using noise contrastive estimation (NCE) (Gutmann & Hyvärinen] 2010 Ma & Collins 2018).
Durkan & Nash (2019) estimate the intractable normalization component by utilizing ARGMs and importance sampling. Che al. (2020); Wang et al. (2021) skirt the challenge of collecting data in the high-dimensional data space by producing data in the fower-dimensional feature space, which improves sampling efficiency.
7 CONCLUSIONS AND FUTURE WORK
In this paper, we propose a novel method dubbed E-ARM integrate energy-based models into au- toregressive generative models seamlessly, with an energy-based training objective that exploits an underlying autoregressive model. This achieved by defining the energy function from the output logits of the base autoregressive network, to model the unnormalized joint distribution of the subse- quence up to each time step. We also found ways to improve training of E-ARM using importance sampling, avoiding the requirement of MCMC for the energy-based training. Experimental results on two language tasks and one vision task demonstrate the effectiveness of E-ARM to alleviate ex- posure bias and incoherence problems of ARGMs. In the future, expect to extend E-ARM on other sequential generation tasks (e.g. text summarization, audio generation), and incorporate the proposed methodology into other advanced autoregressive architectures.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In 9th Inter- national Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc' Aurelio Ranzato, and Arthur Szlam. Residual energy-based models for text. J. Mach. Learn. Res., 22:40:1 2021.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, jun 2005. Association for Computational Linguistics.
Richard Ernest Bellman. The Theory of Dynamic Programming. RAND Corporation, Santa Monica, CA, 1954.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp- 1171-1179 2015.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad. and Pieter Abbeel. Pixelsnail: An improved au- toregressive generative model. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüssann, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 863-871, 2018.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 2978-2988, 2019.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Mare' Aurelio Ranzato. Residual energy- based models for text generation. In 8th International Conference on Learning Repre- sentations, ICLR 2020. OpenReview.net, 2020.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 3603-3613, 2019.
Conor Durkan and Charlie Nash. Autoregressive energy machines. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97, pp- 1735-1744, PMLR, 2019.
Ruiqi Gao, Erik Nijkamp, Diederik P. Kingma, Zhen Xu, Andrew M. Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 7515- 7525, 2020.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning energy- based models by diffusion recovery likelihood. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Will Grathwohl, Khan- Wang, Jöm--Herrk Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky Your classifier is secretly an energy based model and you should treat it like one. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
10

---

Under review as a conference paper at ICLR 2022
Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy- based models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp- 297-304, 2010.
G. Hinton, P. Dayan, B. Frey, and R. Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 268 5214:1 5214:1158-61 1995.
Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771-1800) 2002.
Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. A distributional approach to controlled text generation. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. CoRR, abs/ ttss/1606.00 (00.6006633330 03439, 2016.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, jul 2004. Association for Computational Linguisties.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for condi- tional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 3698-3707, 2018
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Tsvetomila Mihaylova and André F. T. Martins. Scheduled sampling for transformers. Fer- nando Emilio Alva-Manchego, Funsol Choi, and Daniel Khashabi (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 -Aggust 2, 2019, Volume 2, pp. 351-356, 2019.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy- based model. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5233-5243, 2019.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv: 1606.0550. 2016.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Bo Pang, Tian Han, Erik Nijkamp. Song- Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
11

---

Under review as a conference paper at ICLR 2022
Tetiana Parshakova, Jean-Marc Andreoli, and Mare Dymetman. Global autoregressive models for data-efficient sequence learning. In Proceedings of the 23rd Conference on Computational Nat- ural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, pp. 900-909, 2019a.
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional reinforcement learn- ing for energy-based basedd sequential models. CoRR, abs/ 19.00951. 2019b.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th Interna- tional Conference on Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Kaitao Song. Xu Tan, and Jianfeng Lu. Neural machine translation with error correction. In Proceed- ings of the Twenty-Nie -inth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp- 3891-3897, 2020.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA, 13-15 September 2016, pp. 125, 2016a.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural In- formation Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4790-4798, 2016b.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1747-1756. JoL.. 2016c.
Aaron Van Oord. Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756. PMLR, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008 2017.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open- world uncertainty modeling for confidence calibration. CoRR, 15s221077.1628 2021.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28- July 2, 201 1, pp. 681 668, 201 la.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Wushington, USA, June 28 July 2, 201 pp. 681 168-688, 201
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between vari- ational autoencoders and energy- based models. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
12

---

Under review as a conference paper at ICLR 2022
J. Xie, Z. Zheng, X. Fang, S. Zhu, and Y. Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1-1, mar 2019. ISSN 1939-3539.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chin Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Trans. Pattern Anal. Mach. Intell., 42(1):27-45, 2020.
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
Yilun Xu, Yang Song, Sahaj Garg. Linyuan Gong, Rui Shu, Aditya Grover, and Stefano Ermon. Anytime sampling for autoregressive models via ordered autoencoding. In 9th International Con- ference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021b.
Wen Zhang, Yang Feng, Fandong Meng. Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Anna Korhonen, David R. Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1, pp. 4334 4343, 2019.
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial net- works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
13

---

Under review as a conference paper at ICLR 2022
A THE DERIVATION OF THE NEGATIVE PHASE GRADIENT
In this section, we show the detailed derivation of Eq. Formally, as shown Sec. given an autoregressive model (x(*<*) - ( xx1) (k denotes the time step) with parameters 0, we define product of the autoregressive model and EBM follows
vo(#,xx*k) = (x(x<x)
(14)
Za
((,,(xen) Under such definition, only those con- ditional distributions go (Zy xxa) of the most recent k time steps are involved in the current update of parameters while those distant conditional distributions q(5,, (xe) are treated as fixed. We have explained the rationale and intuition in Sec Zg is the normalization term and equal to
where (x<x))
M (0 xxi) I===
Ex (xx(x))) m, (=((x))()))) The optimization of pa( Xcc) includes two phases, and the gradient w.r.t 0 of negative phase is
i log go (x<x))) + E x ~po 00
Ex ck -po 00
th (Ixx, x<x)]
(15)
Next, will show the specific derivation of these two terms in Eq. 15 so that the entire Eq. be transformed into Eq.
15 can
A.1 THE DERIVATION OF THE FIRST TERM
The first term Ex Low
&
log (xxek]] can be processed as follows
Ex Bxxcppp log go (xxx)] =
Σ po (x
log go (x<1)
00
30
Xick
i
po (Jrk: xcck
log (x(x)))
36
Nick Zb.
(16)
Ex, 6 -o((srx
Σ (X<c)) Xec
log go (x<1)
Zo
20
θ
= -Ex To (x(()))) (w(c))) log go (x<x)),
DO
Eng ((
where we have wxxc))
because
Rel wi (xea) Dig a -olls 2
w(xc+)= En, e e-d(U(xxE))
Σn, (o(EEE)))
Zo
Exar Exp g0 )e (o(
Σn, (loxxEc)))
(17)
Exer qo(x<c)) Exg e (ololxxxee))
En, (arLxec))
Rx Eay, (-0olttxxct)
14

---

Under review as a conference paper at ICLR 2022
A.2
THE DERIVATION OF THE SECOND TERM
Then, tackle the second term Ex,,xer~~o
100 (T xx*) as follows
① Epol 00 (T xxk)] = Σ
Po xck)
00 (Kx x<k)
00
00
qo (k x<h) 8
Σ
Po
the (IRR, xeck
40 (Kk xcc) 00
10 (2(xx()) ) - x0( d
Σ The Xack
qo(#;, Xck)
00 (0 xc1)
Zp To( Xeck)
dU(EL(EE)) 1
Ee (cxxxx))((()xxx
20 (u xxri
20 (x(
Zo 00
e =u(=x(xe))
1 2
E0p(x Σ
jo (I< (x<))
d8 (Its xck)
(0(xx((<c))
Zg
00
1 2
Σ (x((x<c)
Σ (o(()))2))
2el (2tk xck)
Zg 00
Xick
c (cN.
0
(xc(()e))
0 (Ik,
Zo
00
(o(
En e -o(2xxxx)) 2
(xp(()e))
0g (x2,
Zo
Σ
go (2t X=1)) (x
0o Xch)l
00
(xj((xe)) [a (xce) w (x<x)
( xcx)]]
00
-Exe Sxxxeeineexxxx /w (x
6o (I (IK, Xck)
00
(18)
where w(xck) is also equal Een olni)ee)) equivalent form of the gradient of the negative phase without any expectation over pe
Combining Eq. and Eq. 18
we can obtain an
Zo
- /w (k) ②
2
log ão (x<x)) + E xcx (wweeeeee)w/(e/c (w(xx))
(0(
(19)
00
00
En e (((E
where w(xc)) Ry
(20)
Recer ~oo (xce)) EEnr e colosixee))
B THE FURTHER REFINEMENT OF w
The reweighing weight w can be further deduced as
Σn, e ((22)xe))
En
po(ze
w(x<)) Exer ~ (x(x)))) EBp e (-Po((xxx))))
Bak (o(xcc)) (o(
Ex ~o (xxx)) 1Ezz
(o(xcN)
(21)
pe(xcc))
p(x=k)
(x(xcc)
Ex (x 4o (xcck)
Ex,, (((x<<)))
where p(xxR) is defined (x(x)) Ne (x(x)))
15

---

Under review as a conference paper at ICLR 2022
C EXPERIMENTAL SETTINGS
In this section, we introduce the specific setup of different benchmarks in 6 We uniformly use Adam optimizer. The training will be stopped once the model has not obtained better performance for 20 epochs on the validation set. For translation tasks, the length of generated fake sentences, which is used for the computing of negative phase in Eq. [2] is dependent on the source sequence whilst for language modeling tasks, we fix the length of generated fake sentences as 50 during training. As for the model structures of the image generation task, we use the official structure reported by PixeICNN (van den Oord et al. (0T6c) and Gated PixelCNN (van den Oord et al.) 2016b) without modification. The source code will released once upon acceptance. We use the same batch of samples generated autoregressively and weight w (i.e., shared), which does not need to sample twice. The number of samples in a batch is dynamic while the maximum number of the total tokens in a batch are fixed (4096 in our experiments). If the length of sequences in a batch is 32, then includes 4096 32 = 128 samples in total. It is a common strategy in language generation tasks, and has been used in many frameworks(e.gg Fairseq (Ou et al. 2019)). We generate samples autoregressively as many as the number of sequences in the current batch at each update iteration.
approximate both the expectations in Eq [2
IWSLT14
WMT16
WiKiText100
Hyper-Parameters
Th-Base
Tr-Base
Tr-Large
Tr-Base
Tr-XL
Number of Layers
12
12
12
16
Hidden Embed Size
512
512
1024
512
410
FC-Layer Embed Size
1024
2048
4096
2048
2100
Attention Heads
4
16
8
10
Dropout
0.3
0.3
0.1
0.1
Learning Rate
5e4
le-3
Ic-3
5e-4
2.5c-4
Ir scheduler
inverse sqrt
inverse sqit
inverse sqrt
inverse sqrt
cosine
Warm up Updates
4000
4000
4000
4000
10000
Weigth Decay
lc-4
0.0
lc-2
0.0
Coefficient
0.05
0.05
0.05
0.05
0.02
E-ARM Start Epoch
15
15
10
15
10
Table 6: Hyper-Parammeers of different model structures and datasets. "Tr-Base", "Tr-Large", and "Tr-II." indicate Transformer-Base, Transformer-Large and Transformer-XI. respecivelly
D MORE EXPERIMENTAL ANALYSIS
D. EFFECT ON INCOHERENCE
In order to validate the effectiveness of our E-ARM for ameliorating the long-range coherence of generations, we undertake an experiment to assess the model's 's performance under different test sets with varying sentence lengths. We divided the test set of IWSLT14 (German - English, Italian English, Spanish + English) translation dataset into three subsets (IO, 25), [25, 50], and [50, oc)) based on the target sentence lengths. Then, we incrementally applied scheduled sampling technique and our E-ARM above the base transformer network, and tested their performances on these three subsets. Generally, the subset of samples with longer target sentences ([50, oc) should have been more affected by the long-range incoherence problem (lower BLEU score). In practice, we uni- formly applied label smoothing and beam searching (with beams) strategy for all experiments in Table 7
Specifically, Table 7 shows that the base translation model improved performance for all three test sets with varying target sentence lengths after using the scheduled sampling technique, especially for the two sets [0. 25) and [25, 50) which had relatively short target sentence lengths (e.g. On German to English task, 38.20 37.72 = +0.48 points and 33.76 33.24 = 0.52 points for [0, 25) and [25, 50) test sets respectively). We consider that this performance boost was achieved through alleviating the exposure bias problem, since scheduled sampling approaches (Ranzato et al., 2016 Zhang et al. 2019; Mihaylova & Martins (019)) have been verified in mitigating the exposure bias problem. Besides, after applying our F-ARM together with the scheduled sampling technique, the
16

---

Under review as a conference paper at ICLR 2022
Translation Task
Scheduled Sampling
E-ARM Training
Target Sentence Length
[0, 25)
[25, 49)
[50, oo)
37.72 1:004
33.24 10006
30.86 10.07
34.61 10.08
De-+ En
38.20 1:007
33.76 10.03
31.08 10.06
35.10 10.04
v
38.37 #0.06
33.92 +0.09
31.43 2004
35.36 +0.05
35.20 1:003
32.73 1000
26,86 10.05
32.29 10.03
It En
35.52 #:0.09
33.25
10.08
26.95 10.14
32.64 +:012
35.56 10010
33.33 10.13
27.21 10.07
32.82 #0.11
45.37 +0.05
39.67
+0.08
37.14 +0.06
40.64 +0.07
Rs- En
43.61 +0.09
40.00
+004
37.38 +0.06
40,91 40.66
43.84 10.10
40.35
10.05
38.07 10.04
41.58 10007
Table 7: Performance comparison on the IWSLT14 test set with respect to different lengths sentences on three translation tasks (German to English, Italian to English, and Spanish to English). Performance evaluated by BLEU score.
base model can further obtain additional performance gain. Specifically, the improvement on the longer sentence is more evident, since model can obtain large improvements On German to English task, 31.43 31.08 +0.35 points for [50, bo) test sets) than short sets [0, 25] and [25, 50] (e.g. On German to English task, 38.37 38. 20 +0.17 points and 33.92 33.76 + 0.16 points for [0, 25) and [25, 50) test sets respectively). This phenomenon indicates that our E-ARM can resolve the incoherence problem to some extent.
the [50, oo) (e.g.
D.2 EFFECT ON EXPOSURE BIAS
Trans. Pairs
DE + EN
EN- DE
EN-
IT
IT-
EN
ES
EN
EN
ES
W
14203
14554
14976
13952
16021
15359
Total
22148
23057
23654
23744
23860
22775
Ratio
64.12%
63.12%
63.31%
59.76%
68.33%
67.43%
Table 8: The effect of B-ARM on the exposure bias problem. Each test set of translation tasks contains 1K sentenees selected randomly. N denote the ground truth words whose probabilitics in the predicted distributions produced by E-ARM are greater than those produced by the baseline.
We follow the analytic experiments in the work (Zhang et al. 2019) show that E-ARM is capable of alleviating the exposure bias problem. Specifically, we randomly select 1K pairs from the training data for each translation pair and use the trained autoregressive model which applied E-ARM (Label Smoothing with smoothing factor 0.1 is applied during training while scheduled sampling is not used) to decode the source sentences, and then count the ground truth words whose probabilities in the predicted distributions produced by our E-ARM are greater than those produced by the baseline and denote the number A. The ratio of A the total number of words tested is calculated. The detailed results are shown in Table g We find that the results on all different tasks are greater than 50%, which demonstrate the ability of our E-ARM in solving exposure bias problem.
D.3 ANALYSIS TO MODEL'S CONVERGENCE
In this section, We will investigate the convergence of our E-ARM. To begin, we first train a base Transformer model ("Tr-Base" architecture shown in Table 6) on the IWSLT14 Spanish to English training set for baseline and E-ARM model respectively, and then record the training loss and test loss (in cross entropy) at the end of each epoch. The loss curves are plotted in the Figure 2 From Figure 2, we can see that (1) at the start of the training, our E-ARM converges slightly faster than the baseline. (2) As the training process progresses, the cross entropy of the baseline on the training set will gradually decrease, with faster rate than E-ARM. On the other hand, the test loss curve of the baseline will fall at initially and then slowly rose after 50 epochs while E-ARM always remains stable convergence. This phenomenon also shows that our E-ARM model can effectively prevents over-fitting and produce better generalization.
17

---

Under review as a conference paper at ICLR 2022
BaseLine E-ARM
BaseLine E-ARM
entropy tra carning
crogs entropy
opoch
(a)
(b)
Figure 2: (a) Cross entropy loss curves on IWSLT14 Spanish to English translation task on training set. The blue and orange colors represent base model and E-ARM respectively; (b) Cross entropy loss curves on IWSLT14 Spanish + English translation task test
D.4 ANALYSIS TO TOP-K RE-SAMPLING
Trans. Pairs
DE
EN
EN
DE
EN-1
IT
IT-
EN
ES
EN
EN
ES
34.86
28.73
29.91
32.44
40.88
37.59
5
34.93
28.85
30.04
32.56
41.01
37.66
34.88
28.91
29.96
32.41
40.90
37.73
Table 9: The effect of Top-K correction in the inference stage. We tested BLEU scores of using different & on different translation pairs of IWSLT14 dataset.
Top-K energy re-sampling in the inference stage is introduced by Bakhtin et al. (202I). which collects many candidate sequences generated autoregressively in the inference stage and then re- samples from them depending on their energy scores estimated by the network. To measure the contribution of the Top-K energy re-sampling in our method, we conduct ablation study to verify by selecting different K (0, 5, 10). The results shown in Table 9 by using BLEU score. From 9 we observe that the benefits brought by Top-K sampling is minor (K=(5, 10}), when compared with model without Top-K sampling (K=0). These results also indicate that the perfor- mance improvements of our E-ARM are mainly from our joint-training, rather than Top-K energy samplig...
D.5 EVALUATION WITH OTHER METRICS
Scheduled Sampling
E-ARM Training
Metries ROUGE-L?
Trans. Pairs
ROUGE-1
ROUGE-21
METEOR;
BLEU
66.51
43.69
63.69
64.35
34.61
De En
44.08
64.61
35.10
67.46
44.77
64.78
65.13
35.36
64.50
40.65
62.18
32.29
It + En
64.73
40.97
61.94
62.51
32.64
65.27
62.49
62.80
32.82
71.10
49.47
68.78
68.94
40.64
71.36
49.53
68.96
69.28
40.91
71.91
50.17
69.65
69.63
41.58
Table 10: Comparison of ROUGE-I, ROUGE-2, ROUGE-L, METEOR, and BLEU scores between our ap- proach E-ARM and the base ARGM trained just with cross coos-eytroppy loss on three translation pairs of IWSLT14 datasets. The value is expressed in percentage. We use "Tr-Base" as the network architecture.
To further evaluate the effectiveness of the our proposed E-ARM. we also evaluate our method by using other metrics, such ROUGE Lin (004) and METEOR [Banerjee & Lavie (2005) for neural
18

---

Under review as a conference paper at ICLR 2022
machine translation. The results are shown in Table I0 In Table 10 the improvements of E-ARM in different metrics is consistent with the conclusion of Table 1 which further prove the effectiveness of our E-ARM model.
D.6 EFFICIENCY STUDY
Our E-ARM has the advantage of being able to optimize an energy-based learning target using maximum log-likelihood, without the usage of MCMC procedures. The requirement to sample data from the autoregressive model at each update step, on the other hand, remains a possible element that could slow down the training process. Nonetheless, the extra overheads are still acceptable when compared to sampling data using MCMC algorithms. The reasons are provided in below:
Assuming that the forward processes of the Transformer, with length sentence as the input, have the time cost т. We tested the time cost of gradients back propppropggttion for Transformer on Tesla V100 GPU. We found that the time cost of the backward process approximately twice as the forward process, which is marked as 2r. Therefore, the time cost of one step update is approximate 3r. Autoregressively generating sequence of length n by Transformer necessitates n feedforward processes, as each predicted token must use all previously created tokens input. One fact is that we simply need to use the previously produced tokens as the input at each time step and gradient back pockprroaggation is not required during the generation.
As a result, the time cost of generating a fake sentence for energy-based training in Eq 12 is actually a 2=17 n_17. Considering the IWSLT14 German to English transla- tion task, which has train- ing samples with length 20 in average, the time cost of generating fake data in each iteration is roundly 9.5T. Furthermore, the generated fake sentence will be fed into the trans- former and included in the overall loss computation, resulting in an extra for- ward and backward proce- dure apart from the update of original input. Thus, the total time cost of our E-ARM's ss one update is 15.5T, which is about 5.2 times great vanilla training. Table [1 shows the time cost of train- ing a 12 layer transformer with 100 iterations. The time cost of our E-ARM roughly coincides the extra time cost modeling, which usually have sequences consisting of hundreds tokens, we randomly truncate a continuous sequence with length 50 for energy- based training in E2
Model
S.S.
w/E-ARM
Sec./100 iter.
27.3
30.1
Tr-Base
Autoreg.
145.8
Autoreg.
149.2
20 steps SGLD
630.6
50 steps SGLD
1452.3
Table 11: Efficiency performance on IWSLT14 German+ English, evalu- ated with BLEU. We uniformly use 12 layer "Tr-Base" in Table 6 "S.S." denotes Scheduled Eq 1y sampling fake data from autoregressive models. "e steps SGLD" represents optimizing our E-ARM with Eq[6] fake data sampled at the first transformer layer's output by SGLD with steps.
"Autoreg." indicates optimizing E-ARM with
analyzed above. For long sequence tasks like image generation and language
When it comes to the MCMC sampling, one problematic issue is that for sequential data like text, the intrinsic discrete property prevents it from applying MCMC in the data space, which forces us to apply it in the latent feature space. Here, we take the SGLD (Welling & Teh, 220 Ib) algorithm for example. Assuming that we apply the SGLD the first layer of the network, then the time cost of one SGLD iteration is about 37 either. Since the SGLD process requires I: iterations to reach convergence, the total time cost of one update of our E-ARM with MCMC process is (3k + 6) In practice, the & is usually set as 100 for stable training Grathwohl et al. (2020). which results in the time cost being (3x (3x100+6)r
102 times large the vanilla training. For short-run SGLD, which takes as 20 with a sacrifice performance, it still leads to the time cost being 22 times as large as the vanilla training.
3r
19

---

Under review as a conference paper at ICLR 2022
D.7 CASES STUDIES
To better understand the advantages of our method in correcting error tokens, we also prepare some translation cases in IWSLT14 German + EEglish, as shown in Table T2
Source Sentence(German)
Prodicted Target SenteneetEnglish)
wenn ich ihnen 600 seischriiten zeige und sic in 10 kasegozien aufteile oder thnen 100 zeitschrifien zeige, und diese 20 kategorien sufteile, dann glauben daes ich ihnen mehr eegeben habe, als ich itnen die 400 gegeben hises gegentiber dear wenn ich ihnen die 600 gegeben batte.
Ground if show you 500 magazines and i divide them up into 10 calegories,
show you 400 magazines and divide them up into 20 car- egories, you believe that have given you more chuice and expesience if gave you the 400 than if gave you the 600.
bessere snwahleefahrung
choosing
Baseline: if show 600 magazines and i split them in 10 cutegories, i'm showing them magazines, and I'm poing to split
up aaso
categories. you think I've given them more choices and better chosce than i would have given them the 400 over the time that gave them the 600. give you 400 magazines can split
Baseline S.S.; if show you 600 magazines categories, and give you 400 maguzines. categorizs, then you think L've given you mare chosce and better selection than would have given you the 400 of which if gave you the 600.
up
Ours: if show 600 magazines and i divide them into categoriis, show you 400 magazines, nnd divide them given you more choices and better selection experience than i gave you the 400 of whom if gave you the 600.
categories, you think i've
und ich definiriv, für mich- meiner situtiin gefibtlich wire, unzufisggon, clesses dunklea pdad vermuting sornagen herunterza- sickem - besonders in dem umstand, in dem ich mich in meiner karriene genade befinde.
Ground Truth- would he very dangerous for me start sort of lesking down that dark path of assumpsioon particulatly given the circumstance that I'm in right now in my career.
definitely know that, my in my sinution it
Baseline: and i know definitely, for would be very dungerons begin to this dark puth of suspect- expecially the circumstance that i'm in my career right now.
situation very down dark path presumption pacially circumstance which I'm my career right now.
Baseline S.S.; and know definitely it would be dangemuus start, kind
Ours: and definielly know that it's for me in my situation very danger- start sickle down this dark puth suspection, in particular, the of
i'm my
right
war hiben das licht nasgeschnller, legton es ein vakuum und sangten die ganze laft aus und kühlten bis fast zum jesat, ganz alleine im safzig. war das stick frei, verhalton wie wolle..
Ground Truth. we turaed off the lights. and ten we put in a vacumm and suckeed out all the and then we cooled it down now, all alone in the elevator. wanted.
little chank metal is
Baseline: we tumeed the light off, par a vaceum and tacked out air and cooled it up umill almost now, all the way alone, the piece metal was behove it was. Baseline S.S.;
tumed the lights off, pur it
and the way up to now, over the place,
sucked the air, and we cooled
the piece of metal was free in behave whitever wasted.
Ours: we tumed off the lights we put isto vaccum and we sucked all the
cnoled up until almost now, all alone the clevator, the piese
metal was free to beluve wioteever it wanted.
und im grunde kennen sie das betrachien, wissen sie, als eins ly lyrameei des erin- nernden solbst, und sie können sich das erinnernde selbst denken als das suumagenn das erlebende selbsi schleppt durch erfahrungen, die das erlebende selbot nicht brancht.
Ground and busically you can look at this, you know, as tynmny the remembering self, and can think the remembering self sort dragging the experiencing self through experiences that experiencing self doesa's it need.
Baseline: and basically, you can think of this, you know, as tyranny of self, and you think the memorable self as that kind of self through experiences that don't
the living
the life itself.
Buseline S.S. and basically. you can look at this, you know, as a tyrannei of memorial self, and you can think of the memorial self one that kind of sucks living self through experieccss that don't need living self. Ours: basocally, you can
at that. you know. mensbering self, and you can think of the memory itself as one, which soct of drapging the living self through experiences that the living sell doesn't it need. Ground Truth. we'r
tyranny of
sind an schwelle orstmnliches, enstonniches oreigaissen anf vielen gebieten. und doch denke ich wirklich, does wir hunderte, 300 juhre var die aufklirung zurtick gehen missstes, um eine seit zu finden, in der wir fortshrin bekömmp haben, der wir über diese dinge beftiper getritten baben, an mehr fronten jezzi.
in many fields, and yet i actually think we'dl have to go back hundreds, 300 years, before the enlightenment. to find a time when we hottled progress, when we Eought about these things vigoruusly
the varge ge amazing, asazing
fronts, than
Raselline: we aee at threshold of amazing, smazing events in many aneas. and yet i really think that we have go back hundreds and 300 years before the enlightenment to find time when we have fought progress which driven
these things
Buseline S.S. we're al the ihreshold amazing, amazing events in munyy sreas, and yet. i really think that we have to go back hundreds and hundrads before the enlighsenment to find when have struggled with progress which we have driven on things more powerful, than now.
fronts
Ours: we's 'r the threshuld to amazing, yot really think that we have to go back hundreds and 300 years before the time when fought progress, where we' righting. about these things to more fronts than we have now.
events in many areas, and
lightenment
Table 12: Translation cases on IWSLT14 De-EE test set, generated by the baseline method, bascline with scheduled sampling and our E-ARM. The italic font means the mismatch translation
20

---

Under review as a conference paper at ICLR 2022
E MORE DISCUSSION OF RELATED WORKS
The seminal idea of combing a generative model and an energy-based model has been explored by plethora of great works (Pang al. 2020 Durkan & Nash 2019 Xie al. 2019 2020 Xiao al.) 2021 Bakhtin et al. 2021). Our E-ARM can be considered as member of this family of mod- els general. but has different mechanism and goal than the others. In particular, Pang et al. (2200) aimed to learn an energy-based model (EBM) in the latent space of a generator model, so that the EBM can act as a prior model on the generator model's ' top-down network. They believe that the energy-based correction of the prior noise distribution will benefit the subsequent generator's generating process. Furthermore, Xie et al. (2019) attempted to learn the conditional distribution of a high-dimennional output given input by combining the efforts of fast thinking initializer, which generates the output and a latent vector, and a slow thinking solver, which learns an objective function in the form of conditional energy function, so that the output mizing the objective function, or more rigorously by sampling from the conditional energy-based model. A similar work is GAMs (Parshakova et al. 2019a Khalifa al. (02I), which combine an autoregressive component with a log-linear component, allowing the use of global a priori fea- tures compensate for lack of data. Moreover, VAEBM, symbiotic composition of a variational auto-encoder and an EBM, was proposed by (Xiao et al. (021). It can use state-of-the-an VAE to capture the general mode structure of the data distribution while relying on its EBM component to explicitly eliminate non-data-like regions from the model and refine the generation samples. In addition, Bakhtiin et al. (202T) designed novel mechanism to train an unnormalized energy- -bssdd models for modeling joint sequence by working in the residual of pretrained locally normalized language model and training using noise contrastive estimation. All of the above models require an additional network to learn the energy scores, which prevents the base autoregressive model from benefiting from EBM's properties modeling the joint distribution in a more temporally coherent manner. In contrast, by carefully constructing an energy-basek learning objective and its corre- sponding optimization procedure, we are able to smoothly integrate energy surface learning into autoregressive networks that do not require additional learnable parameters. Rather than proposing a new generative model, our method is more likely to a novel training pattern for training better autoregressive model. Recently, instead of constructing an autoregressive model in the data space, Xu et al. (2021b) have proposed unique way which uses autoregressive models in the latent space followed by decoder which decodes the autoregressively generated latent feature into the original data space. They attempt to learn a structured representation space where dimensions are ordered based on importance and trade off the sample quality for computational efficiency by truncating the dimensions of latent generations. Their work is orthogonal to ours. We think the combination be- tween our E-ARM with anytime sampling is also a valuable work which worth exploration in the future.
be generated by opti-
21


Reviews:
[{"review_id": "lSfEjl4xBDX", "reviewer": "Reviewer_AQxn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below are the detailed strengths and weaknesses (or questions):\n\n*Strengths*:\n\n1. The proposed method is interesting and novel. It is also well-motivated as a new energy-based model that does not require MCMC during training.  \n2. The experimental results are generally good.\n\n\n*Weaknesses*:\n\n1. The proposed method only compares with vanilla autoregressive models but not other energy-based model baselines. For example, how would the method compare to (Deng et al. 2020)?  Deng et al. do not use MCMC either, thus an experimental comparison with them is important.   \n2. One main merit of E-ARM is its removal of the “extremely time-consuming” MCMC process, however, this merit is not supported with experimental results in this paper. I feel E-ARM can be time-consuming due to sampling from autoregressive models in Eq. 11, thus a comparison with MCMC-based baselines is preferred. In addition to efficiency, performance comparison with energy-based model baselines is also important, otherwise, I am not sure about the advantage of E-ARM compared to other energy-based methods.   \n3. I was confused when the authors suddenly note that the loss is summing over length K (Eq. 11). I think this point should be made earlier for clarity. For example, the sum over K should be added to Eq. 5 for correctness, right?  \n4. Some experimental details are missing:   \n(1) how many samples are used to approximate the expectations in Eq. 11?  \n(2) how many samples are used to approximate the weight w?  \n(3) do (1) and (2) share the same samples for approximation or need sampling twice?  \n(4) Eq. 11 is summing over K and for each k there are expectations to be approximated. This process seems very time-consuming if resampling for every k. Please clarify these details.  \n\n*Questions for the author:*\n\n1. Is the positive phase loss in Eq. 12 also summing over K?  \n2. The end of page 4 says “xx means improvements in this direction will be automatically taken care of as a result of xxx”. This sentence confuses readers – I am not sure whether the first term in Eq. 7 is actually included during training or not, please clarify.  \n3. Appendix C says \"for language modeling tasks, we fix the generation length as 50\", why is generation required for perplexity computation?\n", "summary_of_the_paper": "This paper designs a new globally normalized probability function to approximate the sequence data distribution, which is defined as the product of traditional autoregressive model distribution and energy-based model distribution. The authors also propose a new training objective that is different from maximizing log-likelihood. This new objective is composed of two KL divergence terms. To tackle the optimization issue due to the energy-based loss term, the authors propose to use importance sampling to sample from the autoregressive model to approximate the negative phase loss, so that MCMC is not required. Certain training tricks are required (e.g. pretraining the autoregressive model) to stabilize training.  Experiments on machine translation, language modeling, and image generation demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed method is novel and interesting, but it only compares to traditional autoregressive models in experiments, and other energy-based model baselines are missing.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gMfGYBGgsaX", "reviewer": "Reviewer_FnWE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper is well-written and well-organized. Issues on conventional ARGMs are clearly pointed out, which lead to the motivation of this work.\n+ Three experiments show the overall improvement of E-ARM compared with based ARGM.\n\nWeakness:\n- Training ARGM with EBM in a cooperative manner has early been proposed by\nLearning Latent Space Energy-Based Prior Model, NeurIPS 2020\nCooperative Training of Descriptor and Generator Networks, TPAMI 2018\nCooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning, TPAMI 2021\nThese works, however, are essentially missed from related works and model comparisons.\n- The experiments of this work are relatively insufficient.\n  1. All experiments only demonstrate improvements compared with base model, while comparisons with other baseline models are missing. \n  2. Only basic evaluation metrics are used. Some other metrics, e.g. ROGUE, METEOR for NMT, FID for image generation, should also be listed.\n", "summary_of_the_paper": "This work proposes E-ARM, aiming at integrating EBM into ARGM seamlessly in order to tackle two major existing issues on sequence modeling, i.e., exposure bias and incoherence problems. At each step, the joint distribution $p(x_k, x_{<k}$ is optimized by not only autoregressive cross-entropy loss, but also the EBM KL divergence. Moreover, the model uses importance sampling to bypass the MCMC as conventional EBM models do. Experiments on neural machine translation, language modeling and image samling all demonstrate that the proposed integration outperforms the base model.", "summary_of_the_review": "This work proposes to alleviate ARGM existing problems by leveraging the EBM objective. Such integration, however, is not essentially new and this work does not demonstrate its superiority compared with other methods.  Despite three experiments performed in this work, the evaluation and analysis are insufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "XZY0qRkpPsW", "reviewer": "Reviewer_DZsJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach is interesting and new (to the best of my knowledge). It is motivated by the exposure bias or in other terms *\"a discrepancy of the input context distributions between the training and inference stages\"*, which ARGM are reported to suffer from. \nAccordingly, the experiments show improved performance when combining several ARGM with the proposed E-ARM method. \n\nThat alone, however, does not yet make strong paper (rating 8 or higher) in my opinion, because the authors don't study the effect of their method in detail. Instead, they confine themselves to just reporting improved performance on selected, and not exactly the latest benchmarks (e.g. PixelCNN is from 2016).\nWhat I would wish in order to increase my rating are\n* experiments that reveal when and how exposure bias is actually a problem (e.g. depending on the model expressiveness)\n* experiments that show that this bias is indeed decreased when using E-ARM\n* comprehensive studies on the cost this decreased bias (=improved performance) comes with \n\nBeing the devil's advocate, my suspicion could be that the improved performance is not actually due to $p_\\theta$ (Eq. 4) but due to the heavily truncated back propagation in the second term of (5). A small footnote tells that BP is just two time steps back. The fact that this seems to be crucial for the method requires further investigation, and would make the paper much stronger. Besides, it would be easy to check by just using $\\tilde q_\\theta$ instead of $p_\\theta$ in (5).     \n\nThe paper was ok to follow, but can still be improved in writing in the sense that the mathematical derivations are sometimes hard to follow. \nA lot of the separated equations contain just standalone terms instead of equations. Their references and the relation to them have to be found in the text, which hinders readability. \n\nEq. 5 is misleading, although the intention of it might be clear. The optimisation is over a sum over all $k$  and under the expectation over $\\mathbf x\\sim p_d$. That should be made explicit for better understanding. \n\nThe authors mention density estimation a couple of times, although the proposed method is clearly dealing with discrete distributions. In particular (footnote 1 would be wrong for densities).\n\nSec. 2.1, first line: $\\mathbf x\\in\\mathbb R^K$, to be consistent with what follows\n\nS2.3: *\"Besides,autoregressive decoding typically greedily selects the most probable token at each time step, given\nthe ones previously selected.”*\nI would disagree here. What you usually do is sampling the next token. That greedily decoding to the conditional mode is far from being ideal  is well known, and is by no means a shortcoming of autoregressive models. \n", "summary_of_the_paper": "The authors propose an approach to integrate energy based models (EBM) into into general auto-regressive generative models ARGM. They propose to combine the negative log-likelihood loss of auto-regressive models with a second energy based loss, which serves as a kind of regularisation loss (steered with a hyper-parameter lambda). They claim that this approach mitigates the Exposure Bias which is frequently found in ARGM. A learning algorithm based on Hinton's wake-sleep algorithm is derived that approximates the EBM part of the loss with importance sampling thus avoiding expensive MCMC approximations.\nExperiments form the NLP and the image generation domain are presented that show outperformance over selected baselines.  \n", "summary_of_the_review": "An interesting idea that can lead to improved performance on certain benchmarks. However, deeper insights about the effect that the model addresses (exposure bias mitigation) as well as estimations of the increased overhead are missing in the experiments, thus limiting the scope of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QJi3xnHQdGZ", "reviewer": "Reviewer_SjXn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " # Strong points\n\n- The considered problem is important, and the authors' novel framing of it is well-motivated: using an EBM model that takes the global information into account to mitigate the gap between training and inference.\n\n- A simple algorithm is derived under the defined joint distribution. They use the model distribution as the importance distribution, and It's faster to estimate the weights in the importance sampling than MCMC procedures in traditional EBM.\n\n- Experiments are designed appropriately to showcase the resulting behavior of the model. The proposed method outperforms the base autoregressive models, and is compatible with other techniques and architectures.\n\n\n# Weak points\n\n- The design of the joint distribution seems a bit arbitrary. Could the author better explain the rationale behind the design?  For example, there is large flexibility for $\\phi$ in Eq (4). If $\\phi=-\\log q(x_{\\le k})$, then the importance weight is $w(x_{<k}) = \\frac{q(x_{< k})}{E[q(x_{< k})]}$. It seems to me that this choice is a more intuitive one.  In addition, the joint distribution is essentially defined by conditional distributions. It seems to me that ``we compel the ARGM to fit not only the conditional distributions but also the joint distribution at each time step\" overclaims the actual modeling of the joint distribution.\n\n- Lack of comparison/discussion of relevant works. For exposure bias:  [1] adopt a discriminator as the energy-based model to correct the ancestral sampling in every step. For long-range dependency: [2] instead learns a global ordered representation, like PCA, and trains autoregressive models on top of the representation. ([2] would still incur exposure bias). The paper also does not compare to those ``two stages\" methods in intro. \n \n# Minors\n\n- It seems that there should be a $\\sum_k$ in the RHS of Eq. (5). \n\n[1] Yuntian Deng and Anton Bakhtin and Myle Ott and Arthur D. Szlam, Residual Energy-Based Models for Text Generation, abs/2004.11714\n\n[2] Yilun Xu and Yang Song and Sahaj Garg and Linyuan Gong and Rui Shu and Aditya Grover and Stefano Ermon, Anytime Sampling for Autoregressive Models via Ordered Autoencoding, abs/2102.11495", "summary_of_the_paper": "This paper considers the problem of exposure bias and lack of long-range coherence in auto-regressive models. Sampling in such models is sequential, so the error would propagate through ancestral sampling. The authors propose to combine autoregressive models with EBM, which does not require the assumption of distribution factorization. Existing works use a two-stage optimization, where an EBM is trained on top of the auto-regressive models. The authors instead jointly minimize an energy-based learning objective and the vanilla MLE. They also present a way of estimating the gradient of negative samples through importance sampling. Extensive experiments on NLP and vision tasks show that the proposed learning objective reduces the discrepancy between training and inference.", "summary_of_the_review": "There are some loose threads have been left by the authors. I would increase the scores if the authors address some of the points above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "66Tv6EfiqVW", "reviewer": "Reviewer_in11", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The results look promising and the idea of using EBM guidance in AR model training is interesting. I have some concerns as follows:\n\n* Although the importance sampling avoids MCMC from the EBM, it does require sampling from $\\tilde{q}(x_{<k})$ which goes through the autoregressive generation process and could be time-consuming. Would be informative if you could report FLOP in the paper. However, I do feel that it is way more complicated than a simple regularizer added on to the training of the autoregressive model.\n\n* I still doubt that high variance is induced in the importance sampling as it is not clear how close $p_\\theta$ and $\\tilde{q}_\\theta$ and another Monte Carlo estimate is required when computing the importance weight $w$. Do you observe training instability during training? Would be better to provide more analysis on this point.\n\n* It would be more informative if you could report the results before the Top-K correcting by EBM in the inference stage, as it would clearly show if the AR model really benefits from the joint training.\n\n* In the paper, it is claimed that the joint training improves the long-range coherence. Is there a metric to show this point empirically? \n\n* From table 4, it looks like the optimal solution requires a very small weight of the EBM KL. For what reason that a larger weight harms the performance? Theoretically, it should work with even only the EBM KL.\n\nMinor: \n- Typo for \"The first term *** in Eq. 7 is equivalent to ...\" in the line after Eq.8: should be $-\\mathbb{E}_{x<k \\sim p_d}$\n- Here is a related work of using NCE to learn EBM that you should consider to discuss or cite in the related work: \n\"Flow Contrastive Estimation of Energy-Based Models\", CVPR 20. \n", "summary_of_the_paper": "This paper proposes to improve the training of AR generative models by introducing a \"regularizer\" based on a decently designed energy-based model. The EBM is seamlessly integrated with the AR model and the sampling from the EBM can be transformed as an importance sampling from the AR model. Empirical results on text and image datasets verify the effectiveness of the proposed model. ", "summary_of_the_review": "Overall speaking the paper is well-written and well-motivated. The direction of joint training of AR model and EBM is promising. However, I'm sort of concerned with the computational cost and training instability introduced by the extra EBM module, compared to the performance gain it brings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to train autoregressive model that takes advantage of a well-designed energy-based learning objective model. With the importance sampling, the model can be trained efficiently without requiring an MCMC sampling. Experiments are conducted to verify the effectiveness of the proposed method.  The idea is interesting and well-motivated, but the experiments need to be improved. Reviewer FnWE’s major concerns include limited novelty, lack of discussion with closely related works, and insufficient experiments, and recommend rejecting the paper by assigning a rating of 3. Rebuttal doesn’t address his/her concerns. Reviewer in11 is concerned with the computational cost and training instability due to the extra EBM module and has a few unclear technical details that need to be clarified. The author’s reply along with additional experiments during rebuttal partially addresses the concerns of Reviewer in11, who eventually increases the rating to 6. Reviewer AQxn’s major concern is also about the lack of sufficient comparison with other relevant energy-based models.  Reviewer DZsJ pointed out that the more insightful analysis about the model is missing in the experiments. Even though the authors provide additional experiments for Reviewer DZsJ, they are not satisfied with the feedback because the additional results are not supportive of the claims made in the paper, and end up with a rating of 6. Reviewer SjXn’s concerns include the lack of comparison with relevant works and the unclear motivation of the design of the joint distribution. After the rebuttal, Reviewer SjXn’s concerns remain and assign a rating of 5 to the paper. The overall rating of the paper after rebuttal is marginally below the acceptance rate. Even though this paper proposes an interesting idea, the reviewers’ comments are not well addressed. As a result, AC cannot recommend accepting the paper.  The AC urges the authors to revise their paper according to the comments from the reviewers, and resubmit their work in a future venue.
Decision: Reject