Residual Energy-Based Models for Text Generation
Authors: Yuntian Deng
Anton Bakhtin
Myle Ott
Arthur Szlam
Marc'Aurelio Ranzato
Venue:  (2020)
Keywords: energy-based models
text generation

Abstract: Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

Published as a conference paper at ICLR 2020
RESIDUAL ENERGY-BASED MODELS FOR TEXT GENERATION
Yuntian Deng', Anton Bakhtin", Myle Ott", Arthur Szlam", Marc' Aurelio Ranzato² Harvard University? Facebook Al Research
harvard. haw..aaadddaa...aad.....ee. dengyunttian@sean
aszlam, ranzato]efb.com
ABSTRACT
Text generation ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based locally normalized models which predict one word time. While these work remark- ably well, they are plagued by exposure bias due to the greedy nature of the gen- cration process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second train using noise contrastive estimation. Further- more, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such BERT and RoBERTa. Our ex- periments on two large language modeling datasets show that residual EBMs yield lower perplexity compared locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.
INTRODUCTION
The dominant approach to parametric text generation is based on large neural auto- regrressseee mod- els (Radford al.) 2019). These models can trained efficiently via maximum likelihood and they can efficiently generate samples of remarkable quality. Key to their success is local normalization, i.e. they are defined in terms of product conditional distributions, one for each token the se- quence. Such distributions relatively cheap to compute with modern hardware given the limited vocabulary size common sub-word units like BPE (Sennrich et al. 2015).
Unfortunately, local normalization also brings some drawbacks. First, the designer of the model needs to specify the order in which tokens are generated. Second, at training time the model is conditioned ground truth context while at test time it is conditioned on its generations, discrepancy referred to as exposure bias (Ranzato et al., (016). Finally, while heuristics like beam search somewhat help rescore at the sequence level, generation generally lacks long-range coherency because it is produced by the greedy selection of one token at a time without lookahead.
Energy-based models (EBMs) (Hinton 2002 LeCun al. 2006 Ranzato et al. (007)) are a more general framework which potentially address all these issues, as they do not require any local nor- malization. They only require the definition of an energy function defined over the whole input sequence. Training aims at shaping the energy function such that regions of high density of training data points have lower energy than elsewhere. In principle, EBMs are ideal for modeling text as they can score the whole input once, they are not prone to label bias (Bottou] 1991) and they may enable generation large chunks of text, which should help improve coherency.
However, so far EBMs had limited application in text generation, because sampling from the model intractable, and so is maximum likelihood training. The problem is that shaping the energy func- tion is accomplished by updating the model parameters such that the energy is decreased at the training data points (a.k.a. positive examples) and increased at other data points (a.k.a. negative examples). In maximum likelihood training negatives plication we cannot use gradient-basee MCMC methods (Teh et al., 2003 Du & Mordatch, (019) and Gibbs sampling (Welling al., 2005) is too slow to be practical. Generating negatives by local
generated from the model. but in text ap-

---

Published as a conference paper at ICLR 2020
perturbations of the ground truth would be efficient but hardly useful for generation purposes, when at test time the model needs to generate from scratch.
Recently, Bakhtin et al. (0119) carefully studied the problem of training discriminator to dis- tinguish human written text from language model generations. They experimented with different language model and discriminator architectures, training/test time corpora and concluded that the discriminator can generalize rather well to weaker language models when the training/test corpora match. [akhtinn al. (2019) found that the learned discriminator is not robust to random perturba- tions, and argued that the discriminator operates in the "residual" space of the language model.
Concumently, Grover et al. (2019) proposed general approach "de-bias" generator, by simply training a discriminator and using its output for importance sampling.
In this work, we build upon these two works. First, we formalize the residual interpretation by Bakhtin et al. (2019) and use a generative model of the form:
Po(x) x PLM (z) exp( Eo(x))
where (zM(2) is locally normalized language model which is fixed during training, and Es is the energy function parameterized by 0. The resulting model (x(x) is globally normalized due to the energy term. Note that the same residual formulation was also used in Rosenfeld et al. (2001): Wang & Ou 2018b): Parshakova et al. (2019).
This formulation has multi-fold benefits. First, by incorporating a locally normalized language model, we can leverage recent advancements in locally normalized language modeling. Second, the language model provides natural proposal distribution for training (Bakhtin et al.) 2019)), and training can be made efficient by using the conditional noise contrastive estimation objective Gut- mann & Hyvärinen 2010) as we shall see in {3] Lastly, this formulation enables efficient evaluation and generation via importance sampling (Horvitz & Thompson 1952 Grover et al. 2019).
In some sense, this last point is perhaps the central contribution of the paper, as it allows estimating perplexity of the residual EBM, and thus allows these EBMs to be compared in a standard way to other models. Indeed, in U we show that our joint model decreases perplexity on two large datasets, when compared to various auto-regressive language model baselines. Finally, the EBM generations are significantly preferred by humans according to our qualitative evaluation. To the best of our knowledge, this is the first time that an EBM has demonstrated improved generation ability against very strong auto-regressive baselines, both in terms of estimated perplexity and through human evaluation.
2 RELATED WORK
Energy-based models have a long history in machine learning (Hopfield 1982 Hinton 2002, LeCun et al., 2006; Ranzato al.) 2007). The key challenge of training mining for good negatives. This can be accomplished explicitly by fantasizing inputs where the energy should be increased implicitly via global constraints such as sparsity (Ranzato et al., 2007). Methods attempting maximizing the likelihood of the data require to sample from the distribution induced by the model. Unfortunately, gradient-based MCMC approaches like Hybrid Monte Carlo (Teh et al. 2003) and
Langevyn dynamics (Ranzato et al.) 20077] Du & Mordatch 2019:] Xie et al., 2016 2017 2019
2018 Gao et al., 2018, Nijkamp al. 2019) are not applicable when the input is discrete like in text applications. Other approaches like Gibbs sampling (Hinton) 2002) were applied to binary inputs but do not scale well large dictionaries once the energy function is a large bidirectional transformer model like the one used in this work. Several variants of auto-encoders have also been investigated for representing and generating text (Bowman et al. 2016 Zhao et al., 2018), but they have not shown significant improvements in terms of perplexity and they have far been applied to relatively small datasets only.
Our approach appears similar to discriminative reranking approaches used in the parsing and ma- chine translation community (Shen et al.) 2004). However, our approach provides a generative are directly tuned close the gap between the model dis-
model, and tribution and the data distribution, rather than relying on surrogate ranking losses. This approach is also related to other sequence level training objectives (Edunov al. 2018), with the major differ-
2

---

Published as a conference paper at ICLR 2020
ence that in those works training aims at improving the baseline model, but generation at test time is still greedy.
Energy Networks have been used for sequence modeling (Rosenfeld et al. 2001 Wang al. 2015 2017 Wang & Ou 2017; Sa Parshakova et al. 2019). In particular, our residual modcling form and the training algorithm is the same in Wang & Ou (018b), where they used LSTM as the generator and a CNN-LSTM as the energy function, and showed significant gains compared to LSTM baselines in speech recognition. Our work builds on these prior works and develops new lower and upper bounds for the log- log-probability under the joint model, which makes it possible to show that the residual EBM approach gets better perplexity. We also develop an importance weighting sampling scheme used at generation time, which is focused on conditional generation as opposed to rescoring in speech recognition (Wang Ou 2018b). The residual EBM formalism makes it very natural to use BERT for language modeling, and we show that empirically this type of approach can outperform modern state-of-the-a language modeling baselines, both in terms of perplexity, and through human evaluation.
Generative Adversarial Networks (Goodfellow et al., 2014) also relate to EBMs, except that in EBMs the generator is implicit and negatives samples are produced by the discriminator itself. In our work, the pretrained locally normalized language model can be seen as a fixed generator, like in Bakhtin et al. (2019). AAzai et al. (2018) also share our same goal but their generator is not locally normalized and they propose to improve the sampling from the generator by using the discriminator for rejection sampling. Similar to work. Grover al. (2019) propose to the discriminator to de-bias the pretrained generator using importance sampling. We adapt this work to the application of text generation. In particular, we adopt the conditional noise contrastive estimation (NCE) objec- tive (Ma & Collins 2018 Gutmann & Hyvärinen, 2010) to our residual model energy function and then sample from the joint model using importance sampling. We want note that the same for- mulation has been proposed in (Wang & Ou 20186 Parshakova et al. 2019). While Ma & Collins) (2018) used conditional NCE to predict the next word in a sequence, we apply to produce a whole sequence at once with the pretrained auto-regressive language model as the noise distribution.
3 RESIDUAL ENERGY-BASER MODELS
We study the problem of conditional generation of discrete sequences. Given a prefix "1.-.. , Tp with By E V where V is the vocabulary, we want to model the probabilities of generating a sequence of total length T > The generative model is:
Po(Epti, ... FT pr12.... xp) (LMM(TT1I.
.T 11, ... "II) exp( (ane--Ee(x)) Ee(21. ZTT)
Zo Zo(, xpp))
where Zo( xpp) is a normalizing factor known as partition function. Computing the partition function is intractable in our case since it involves a sum over |''''> terms which grow expo- nentially with the sequence length: in our experiments the size of the vocabulary is 50,096 and the length of the generation is 40 tokens. We call Po the joint model, and Eg the residual energy function since PLM is fixed throughout training. The goal of training to learn the parameters of the energy function such that the joint model distribution gets close to the data distribution. For the sake of reducing clutter in the notation, we will drop the conditioning variables in the following discussion.
3.1 TRAINING
When the partition function is intractable, Maximum Likelihood Estimation (MLE) requires samples from the model distribution, which is usually approximated with Monte Carlo sampling or mean field inference (Hinton 2012: LeCun al. 2006) for globally normalized models. Unfortunately, both approaches are too computationally expensive for text applications when using large bidirectional transformer models. For instance, we were to employ Gibbs sampling exactly, we would need to perform at every position as many forward passes as words in the dictionary to compute each conditional distribution. On large datasets where training locally normalized models on multiple machines already takes days, having such additional overhead means that the model would learn
I We assume a fixed T for simplicity of analysis and implementation, but our method generalizes to varying length generation with end-of-sequence symbol.
3

---

Published as a conference paper at ICLR 2020
from much less data for the same amount of time, and this is seldom a beneficial strategy for learning models that generalize well. Therefore, we do not use either MCMC nor mean field methods, as the latter would introduce additional variational parameters an inference network which anyway yields an approximation to MLE learning.
Instead, we train our residual energy function using Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen 2010), and more specifically its conditional version (Ma & Collins 2018). NCE requires two distributions: The model distribution and noise distribution. In our case, the model distribution is the joint model of Eq. 2 Po. while the noise distribution is the pretrained language model. PLM. NCE then trains a binary classifier on the difference of log-probability scores of these two models. Since our joint model is the product of the energy function (whose parameters we want to learn) with PLM. the difference reduces to: log Po log PLM = Fig. Therefore, under these modeling assumptions of residual learning and noise model, the objective function becomes:
1
max EE Paaca log exp(Eg(x+)) + EEg_-PM log exp( exp((-E((x_)
(3)
where x4 is a positive sequence taken from the human generated training set, and x_ a negative sequence drawn from PLM (for given ground truth prefix). In other words, training the energy function reduces to training a binary classifier to discriminate between real text and text generated by an auto-regressive language model. The aim of training is to assign negative energy pos- sible to real data, and as positive energy as possible to machine generated data. Interestingly, the role of positive and negative samples is totally symmetric in this loss function, | will discuss the consequences of this.
With the theoretical guarantee of NCE, can show that the optimum of the above objective is reached at data distribution with infinite amount of data and model with enough capacity, which is also proved in Ma & Collins (2018 2
Theorem 1. If PLM has the same support Pdatas then the objective fanction in Eq. reaches its maximum at log PLM(()) - Fo(x)) = log Pautas if there exists such
Proof. This theorem directly follows from the proof in Gutmann & Hyvärinen (010). Note that optimum. PT.M (ar) exp(-Eo(x)
) is self-normalizing: instead of Pa(x) x PI.M (xr exp(-Ex(x)) have (a(a)) = PLAM (x) exp( exp((Eo(xx)) (x))) However, we still need estimate the partition function
throughout the rest of this paper, since we cannot guarantee that this optimum can be reached.
3.2 EVALUATION
commonly used protocol for evaluating generative sequence models, especially language models, is perplexity (PPL), which is equal to 2 -ー Ei=pt> log, P(a, (a.-1.- ,d1)) PPL can be interpreted as the average number of tokens the model is uncertain at every time step. Since the log-likelihood required by PPL relies on estimating the partition function Ze Σx PLMM( exp(- Eg(x))
EgwPry Ar exp( - Eg (z)) (z)). we derive two estimators for the log part-aiatiion function log Zg based on the work of Nowozin (2018).
Theorem 2. Denote T, as the empirical estimate log EEMPIM exp(-E(xr) -Ex)) with n samples ~
PLMMii 1,..., n): Tn log Σ exp(-E(x,)) then VE 0. NN 0 such that Vn > N we have
Z ETTT)) 7 [2 1)Tn - 2(n 1)T.-1] < Zo +
The proof is given in Appendix A.2
We can use the above two estimators to estimate the lower and upper bounds of the partition function, but we want to emphasize that they are true only asymptotically (when is sufficiently large). We also want to note that to get lower variance estimates we use leave-one-out strategy to estimate Tz-1- See Nowozin (2188) for implementation details and methods to improve numeric stability.
Similarly to locally normalized models, we can also factorize the probabilities of an entire sequence step by step, as Pzr) = II=_, ((r.( and evaluate the PPL for each generation step. By
From Ma & Collins 2018) Assumption 2, for conditional NCE the model needs he flexible enough such that the noff-ooraiiiiig property can be satisfied conditioned any prefix.
4

---

Published as a conference paper at ICLR 2020
Algorithm 1: Top-k Joint Sampling
Input:
number samples a drawn from PLM, value of top-k
// Get a set of samples from PLM
sample samples {z' ..... x²) from PLM with top-k sampling calculate energies
F (xx)) for each x' {x+, ,""""
// Resample from the set of LM samples
sample = I' with probability exp(-a8)
(xp(-sa)
return x
marginalizing over the future, we can derive the following per step probabilities:
p(x))aee)) PL.s (xr)))
Byg PL (px eepp( Fo
The step-wise probabilities Eq. are an instance of importance sampling (Horvitz & Thompson 1952). The basic PLa distribution is adjusted by the probability assigned to token by the energy function (numerator is clamped at 24 while denominator sums over all the possible values of the token at position 0) with the additional marginalization over all subsequent tokens up to the horizon T. Since the summation involves exponentially many terms, unless / = T, this is approximated by samples drawn by PLM- Since both the numerator and the denominator take the same form as the partition function, we also use Eq. 4to to estimate the upper and lower bounds. E.g., the lower bound of log P(rr,/rrcc can be obtained by using the lower bound of the numerator and the upper bound of the denominator.
For t T, we can calculate the log probability by exhaustive enumeration. This gives idea of the true performance of our model at the last step, and it also provides sanity-check of the tightness of our estimators.
3.3 GENERATION
Generating from the joint model is a non-trivial task. A naive way is to generate from the joint model auto-regressivelly by marginalizing the future in Eq. 5 which term Top-k auto-regressive sampling. However, doing so is computationally expensive and impractical, and we only use this method for a qualitative analysis of the joint model in Appendix A
In order to generate efficiently, we use self self-normalizing importance sampling (Owen, 2013 Grover al., 2019). Under the assumptions that the model from which we wish to draw samples is the joint model, which is the product of the auto-regressive model and the energy function, and that the proposal distribution is the auto-regressive model itself, sampling proceeds simply by: a) sam- pling from the auto-regressive language model, followed by b) resampling according to the energy function. The algorithm is shown in Algorithm [ where we introduce an optional top-k constraint on the pretrained language model to improve the quality of samples in the se Without the top-k constraint, as the number of samples goes to infinity, we would recover exact samples from the joint model distribution.
4 EXPERIMENTS
In this section, we describe the experimental set up and the results we obtained by using the residual EBM for text generation, both in terms of perplexity and generation quality.
4.1 EXPERIMENTAL SETUP
Datasets
We consider two datasets: the Toronto Book Corpus (Zhu et al. 2015 Kiros et al. 2015)
and CC-News (Bakhtin et al. 2019). The former dataset consists of fiction books in 16 different
Adapting other types of local constraints such nucleus sampling (Hotzman et al. 2019 is straight- forward.
5

---

Published as a conference paper at ICLR 2020
genres, totaling about half billion words. The latter is a de-duplicated subset of the English portion of the CommonCrawl news dataset (Nagel) 2016), which totals around 16 Billion words. The book corpus is more challenging because the range of style and topics is more diverse than CC-News. Also, the book corpus 30 times smaller than CC-News and may pose generalization challenges because of its smaller size.
In all our experiments we use prefix of size 120 tokens and we generate the following 40 tokens; with the notation of Eq. 2 p 120 and T 160. For training the joint models, for efficiency we generated 16/128 samples per prefix for CC-News/Book Corpus offline, and sample uniformly from those samples training time.
Baselines We consider as base language model (BASE LM) used to generate negatives for the residual EBM, a transformer language model with 12 layers, h = 16, dmode 1024, djf 4096 (we refer to Vaswani et al. (017)) for notations). This is also our first baseline model.
The joint model has as many parameters as the sum of the number of parameters in the base LM and the number of parameters in the energy network. To make fair comparison, we consider two additional baselines that have the same number of parameters as our joint model.
The first baseline is a Residual Auto-regressive Language Model baseline (RALM):
log PRALM (1gl (1)) = log PLM (x) + log (xy + const where Ps takes the form of another auto-regressive language model. The parameters of Po are trained by exact maximum likelihood training of PRALM-
(6)
The second baseline an auto-regressive language model of the same size of our joint model (sum of the base LM and energy function parameters), dub this model Big Auto-regressive Language Model (BALM). BALM has 12 layers, = 16, dmoodt = 1568, dزf = 6272, and is trained by standard token level cross-entropy loss.
Residual EBM Architecture We consider two architectures for our residual EBM, both of them are based on transformers (Vaswani et al.) 2017 Devlin et al. 2018). The first version uses causal self-attention and is derived from the base LM, a unidirectional transformer (UNIT). It is of the same architecture BASE LM, except that in the final layer we project the mean-pooled hidden states to a scalar energy value. We initialize its parameters with a language model trained on the same dataset.
The second version is instead bi-directional (BIT), and the energy function is computed by project- ing the mean-pooled top hidden states down to a single scalar value. We consider three variants, a BIT-BASE following the architecture of RoBERTa-Baan Base, and a BIT-LARGE+ following RoBERTa- Large (Liu et al. 2019), and a BIT-MED with the same number of parameters as UNIT (such that JOINT BIT-MED has roughly the same number of parameters as BALM We initialize the pa- rameters with a trained BERT, and we use * to mark usage of external data, otherwise it means that training set. Notice how model can be interpreted natural way to finetune large bidirectional pretrained models for the text generation task.
BERT was trained
While we expect BIT to yield better results because it can fully leverage context also for intermediate tokens, we also consider UNIT to compare to the RALM baseline, which uses the same architecture and only differs in the way parameters trained and in the presence of local normalization.
We train our models on DGX nodes, each with 8 Nvidia V100s. To improve training speed, we use mixed precision t We use the Adam optimizer, with cosine learning rate decay and learning rate warmup. To stabilize training we used gradient norm clipping (Pascanu et al.) 2013). Detailed hyper-parameter settings can be found in Appendix
For generation, we use top-k sampling with A = 10 for all human evaluations. We take 10,000 samples from BASE LM for our joint sampling.
4.2 RESULTS
*We use models from the HuggingFace repository at Cransformers
https://gi wwwp.w../...........ot.tmm

---

Published as a conference paper at ICLR 2020
CC-News
Toronto Book Corpus
Model (#parameters)
Test
Test
LM (203M)
18.41
17.57
16.16
18.29
RALM (LM+203M)
17.01
16.17
15.71
17.85
BALM (408M)
16,50
15.74
15.00
16.99
JOINT UNIT (LM++03M)
16.42-16.44
15.57-15.58
15.12-15.13
16.98-17.00
JOINT BIT-BASE (LM+125M)
15.32-15.35
14.61-14.64
JOINT BIT-BASE* (LM++225M
15.11-15.17
14.37-14.42
14.14-14.16
15.72-15.74
JOINT BIT-LARGE* (LM4355M)
14.59-14.61
13.97-14.00
13.80-13.83
15.33-15.38
BASE MM-24I. (203M)
15.71
14.89
15.61
18.14
RALM-24L. (LM-241+203M)
15.70
14.89
15.63
18.17
BALM-24L (408M)
14.58
13.92
15.20
18.24
JOINT UNIT (LM-24L+203M)
14.59-14.61
13.81-13.82
15.12
15.16
17.46-17.48
JOINT BIT-BASE (LM-24L+125M)
13.68-13.69
13.01-13.03
JOINT BIT-BASES (LM-24L+125M)
13.60-13.62
12.98-12.95
14.11-14.12
16.17-16.18
JOINT BIT-MED (LM-24L+203M)
12.97-13.01
12.38-12.42
JOINT BIT-LARGE* (LM-24L+355M)
12.71-12.77
12.10-12.16
13.30-13.34
15.17-15.22
Table I: Validation and test perplexity on CC-News and Toronto Book Corpus. denotes models initialized with RoBERTa trained on additional data. The joint model perplexity ranges are estimated using 100,000 samples, see Eq. The number of parameters of each model is shown in parentheses.
1.1
PPL BASE LM-24L PPL
1.0
0.9
0.8
Joint BiT BiT-Med
Joint BiT Med lower
Joint BiT-Large" upper
0.6
Joint BiT-Large* lower
Joint BiT Large" exact
BALM-24L
0.5
10
15
20
25
30
35
position
Figure 1: Perplexity gain of JOINT BIT-MED and JOINT BIT-LARGE+ (using BASE LM-24L. 24L) each position CC-News. each position the lower and upper bounds (Eq,
relative to BASE LM-241 on the test estimated using the method in Eq. [ see for2 more details) are estimated using 20,000 samples. The shorter the horizon (moving to the right), the tighter the estimation is but also the more limited the gains compared to base LM un-normalized models are most useful on longer generations.
Automatic Evaluation Our main result is reported in where we compare models in terms of their perplexity. We can see that on both datasets, residual EBMs with causal attention JOINT UNIT outperforms the baseline RALM with approximately the same number of parameters. The non-residual baseline BALM performs similarly to JOINT UNIT, which might be due to the limita- tion that PLM is not trained jointly with the residual model in both JOINT UNIT and RALM. How- ever, by using our EBM approach, we can remove the causal attention mask and use bi-directional models, which achieves better performance than baselines and JOINT UNIT: without external data, JOINT BIT- BASE reaches a higher performance than JOINT UNIT with fewer parameters. By initial- izing from the state-of- the-art pretrained bi bi-directional transformers RoBERTa Base and RoBERTa- Large, JOINT BIT- B and JOINT BIT-LARGE* reach even better performance than JOINT BIT- BASE.

---

Published as a conference paper at ICLR 2020
Modell (baseline)
Model2 (compared model)
Rate
p-value
BASE LM
JOINT UNIT
52.85%
0.16
BASE LM
JOINT BIT-BASE
56.25%
0.015
BASE LM
JOINT BIT-LARGE*
58.93%
0.00084
BASE LM
BALM
46.77%
0.88
BALM
JOINT UNIT
50.00%
0.52
BALM
JOINT BIT-BASE
57.89%
0.0027
BALM
JOINT BIT-LARGE*
59.89%
0.00020
BALM-244
JOINT BIT- MED (24L)
56.23%
0.015
JOINT BIT-LARGE* (241.)
HUMAN
55.21%
0.036
BASE LM
BALM
54.85%
0.050
Table 2: Human evaluation results on a subset of 333 sentences on the CC-News test set. The rate is computed as the percentage of sentences where the number of turkers preferring Modell is strictly less than (denoted with <) or greater than (denoted with <) those preferring Model2. Attention check is used to drop some votes, so there might exist ties. p-value is based on single-sided binomial test.
In the lower part of the table, we show that if we make the big language model baseline BALM deeper (BALM- 24LL) (24 layers instead of 12, for the same number of parameters) we attain lower perplexity. However, training the joint model JOINT BIT-BASE on the residual of a deeper language model BASE LM-24L yields even lower perplexity, despite having fewer parameters. By using the same number of parameters by initializing from RoBERTa-Large, JOINT BIT- BASE obtains the best results.
BALM-244, JOINT BIT- BIT-MED further decreases perplexity. Finally,
One caveat of our evaluation protocol is that the perplexity bounds are only estimates, which might not reflect the true value, particularly since the number possible sequences grows exponentially with the number of words that generated. We therefore break down perplexity per position in the generated sequences as in Eq. 5 and compare the estimated PPLs to the true enumerated PPLs at the last position, shown in Figure | We find that the final generation step, the estimated bounds agree remarkably well with the exact values, proving that our method at least gets a reasonable PPL estimate the last generation step, and that JOINT BIT-MED outperforms baselines at the last generation step for sure.
Human Evaluation Better perplexity results do not necessarily imply better generations. Besides, since generation from the residual EBM requires approximations in Algorithm the limited sam- ple size might induce approximation errors compared to truly sampling from the joint distribution. Therefore, conducted human evaluations to compare generations from the residual EBM model to generations from the baseline language models.
For each prefix, we present completion from each model, and ask humans to select the one that is a better continuation. More details about human evaluation can be found in the Appendix A.4 The preference rates reported in Table confirm that indeed the generation quality of JOINT BIT-BASS and JOINT BIT-LARGE* is better than both language model baselines. Depending on the model variant, our joint model (with bidirectional EBM) preferred between 56% and almost 60% of the times; interestingly, the preference rate does not change much as we compare against base LM as opposed BALM. In fact, humans do not seem to have a strong preference for BALM over base LM, despite the former scores two perplexity points lower. Similarly, JOINT UNIT is not strongly preferred over BASE LM despite its lower perplexity score. We surmise that unidirectional scor- ing functions and auto-regressive models exhibit generation artifacts which are easily detected by humans, and these may overshadow the improvements brought by perplexity gains.
4.3 ANALYSES
In this section, we analyze some of the results we obtained. First, we check whether we used a sufficient number of samples in perplexity estimates. Second, assess whether the joint model produces fewer repetitions compared to the base language model, and finally we check how well some statistics of the model and data distributions match.
8

---

Published as a conference paper at ICLR 2020
1.00
15.3
unique percent
0.95
15.2
PPL
00.00
Real
15.1
Upper
BASE LM
0.85
Lower
Joint
15.0
20000
40000
60000
80000
10000
number samples
gram
Figure Left: PPL estimation for joint BIT-BASE on CC-News validation set vary the number of samples. Right: Percentage of Unique n- n-grams found in real data, samples from the joint model BrT-BASE and samples from the base language model. The joint sampling is done with 10,000 samples.
0.015
0.015
Real
Real
BASE LM
Joint
density
density
0.005
0.005
0.000
0.000
250
-000
-150
100
50
250
2000
150
log PLar(x)
log PL.((a) Eo(x)
Figure 3: Density plot of log-lrobbe log-probability scores using the base language model (left) or the joint model (right). The red curve corresponds to real samples, the black curve to samples from BASE LM and the green curve to samples from BIT-BASE. The joint model provides much better fit than the hase language model.
Number of samples. In Figure 2 we vary the number of samples we take in order to estimate PPL upper and lower bounds. Beyond 20,000 samples the upper estimate becomes very stable, although we have to emphasize that these estimates might be biased even though the gap between lower and upper bound closes take more samples.
Repetitions. A typical artifact of auto-regressive language models is their tendency to repeat phrases. It is then interesting check whether the joint model is able to alleviate this artifact. Fig. 2 shows that indeed the joint model has slightly higher percentage of unique n n-grams com- pared to the baseline language model with 2,3, 4, although still as high as the original human generated text.
necessary condition for the model match the data distribution. the joint model po matches the data distribution Pa. then statistics computed large population of samples from the two distributions should also match. In particular, Fig. 3 show the density plots of log likelloodood scores of the baseline language model (left) and joint model (right) when fed with their versus samples from the test set. We observe that the histogram of samples from the joint model matches the real data distribution more closely: The difference of means in the LM BASE is 21.6 1.64 whereas the difference is 6.20 in the joint approach.
samples
LIMITATIONS
In the previous sections highlighted the strengths of residual EBMs, namely their simplicity, efficiency both at training and test time, and their improved perplexity scores against strong auto- regressive language model baselines. In this section, comment their limitations to caution the

---

Published as a conference paper at ICLR 2020
reader about when these methods are more likely succeed and inform other researchers about what future avenues of research may naturally derive from this work.
In order to make training efficient and side step costly negative mining using the energy function itself, the current approach uses negatives generated from a pretrained auto-regressive language model. Therefore, is strong enough, and as long as the ground truth and other plausible sequences are reachable by the baseline language model.
model works as long the base language model from which we draw samples
If the base language model has poor quality, then generation from our joint model is going to be poor as well, as the joint model merely resamples generations from the original language model. Moreover, training is going to be trivial if the base language model is poor, because the residual energy function merely needs detect trivial generation artifacts from the base language model. In fact, observe that the role of positive and negative samples is symmetric in the loss of Eq. B This means that the energy function can choose to minimize the loss by either modeling the true data or the negative samples; since the latter have much simpler structure, it is going to model the negative samples. Therefore, importance sampling amounts to mostly down-weighing the worst samples from the base language model. The consequence of this is that search with poor base language model is going to be catastrophically inefficient, as we would need to sample an impractically large number of negatives in order to find samples that are reasonably close to the true data manifold.
To summarize, this work makes a rather strong implicit assumption on the quality of the base lan- guage model, and it is expected to work well only when this is rather strong. In our application, this assumption is met quite well in practice as large auto-regressive language models trained on large datasets have improved significantly in recent years (Radford al. 2019). In general however, residual learning always carries liability to its base model.
CONCLUSIONS AND FUTURE WORK
We investigated an EBM trained on the residual of pretrained autoregressive language model (Wang & Ou 201 Parshakova et al. 2019). The resulting joint model scores sequences holis- tically, thanks to the energy function. Training very efficient and consists of binary classification task between positives from the training set and pregenerated negatives from the fixed language model. Generation is also very efficient as it amounts to resampling from the large set of negatives produced by the base language model. Our estimates show that the resulting model has lower per- plexity than the base language model. Finally, this approach may be interpreted as natural way to finetune large bidirectional transformer like BERT for text generation applications.
In the future, we plan to investigate other ways to generate negatives that may strike a better trade- off between the amount of compute each negative requires and their closeness to the joint model distribution. It would also be interesting to explore other loss functions and the generation of longer pieces of text by using this model auto-regressively at the chunk level, as opposed to the token level.
REFERENCES
Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrim- inator rejection sampling. arXiv preprint arXiv:/810.06755 2018
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning discriminate machine from human generated text. arXiv preprint 1rXiv:1906.03311, 2019.
Léon Bottou. Une approche théorique de I' l'apprentissage connexionniste: Applications la recon- naissance de la parole, 1991. Ph.D. thesis, Université de Paris XI, Orsay, France.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In SIGNLL Conference Computational Natural Language Learning, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 110.0400,, 2018. URL
http:://axii>> wwwpariww..ogg//boooooo www.ooggg//b///ee.e 1800 04805
10

---

Published as a conference paper at ICLR 2020
Yilum Du and Igor Mordatch. Implicit generation and generalization in energy-basee models. CoRR, abs/1903.08689, 2019. BRLhttp:: http://xxxvv.o// org/ abs/1903. http////g/bbbb//b/bbb///// 08689
Sergey Edunov, Myle Oit, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In North American Chapter the Association for Computational Linguistics, 2018.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative con- vnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9155-9164, 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Aditya Grover, Jiaming Song, Alckh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-free importance weighting. arXiv preprint arXiv: 09531, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirtennth International Conference Artificial Intelligence and Statistics, pp. 297-304, 2010.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771-1800, 2002.
Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural networks: Tricks of the trade, pp. 599-619. Springer, 2012.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. arXiv preprint arXivv/904.0000
2019.
John Hopfield. Neural networks and physical systems with emergent collective computational abil- ities. In National Academy of Sciences of the USA, volume 79, pp. 2554 2558, 1982.
Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 1952.
Ryan Kiros, Yukun Zhu Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, and Sanja Urtasun, Raquel an d Fidler. Skip-thought vectors. arXiv preprint arXiv:1506.0677 2015.
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu-Jie Huang. A tutorial on eneegy-based learning. Predicting Structured Outpuls, 2006. MIT Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X7:1907.116999 2019.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for condi- tional models: Consistency and statistical efficiency. In Empirical Methods for Natural Language Processing, 2018
Sebastian Nagel. Ce-news.
aateeesssddaattttalallllloo. 2016.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent persistent short-run meme toward energy-based model. In Advances in Neural Information Pro- cessing Systems, pp. 5233-5243, 2019.
Sebastian Nowozin. Debiasing evidence approximations: On importance-weigheee autoencoders and jackknife variational inference. In International Conference on Learning Representations, 2018.
Art B. Owen. Monte Carlo theory, methods and examples. 2013. URL https://staatet stanford edu/ /tww///ww/////////// chapter
11

---

Published as a conference paper at ICLR 2020
Tetiana Parshakova, Jean-Marc Andreoli, and Mare Dymetman. Global autoregressive models for data-efficient sequence learning. In Conference on Computational Natural Language Learning, 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy based framework for unsupervised learning. In 11-th International Workshop on Artificial Intelligence and Statistics (AISTATS), 2007.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. In International Conference on Learning Representation, 2016.
Ronald Rosenfeld. Stanley F Chen, and Xiaojin Zhu. Whole-sentence exponential language models: vehicle for linguistic-statssicaa integration. Computer Speech & Language, 15(1):55-73, 2001.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1508.
1508:.0000079999 07909, 2015.
Libin Shen, Anoop Sarkar, and Franz Josef Och. Discriminative reranking for machine translation. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 177-184, 2004.
Y. W. Tch, M. Welling, S. Osindero, and Hinton G. E. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 1:1235-12000 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-599800 2017.
Bin Wang and Zhijian Ou. Language modeling with neural trans-dimensional random fields. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 294-300. IEEE, 2017.
Bin Wang and Zhijian Ou. Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 70-76. IEEE, 20184 8a.
Bin Wang and Zhijian Ou. Learning neural trans-dimensions random field language models with noise-contrastive estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6134-61 6138. IEEE, 2018b.
Bin Wang, Zhijian Ou, and Zhiqiang Tan. Trans-dimensionas random fields for language modeling. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 785-794, 2015.
Bin Wang. Zhijian Ou, and Zhiqiang Tan. Learning dimenssoal random fields with applica- tions to language modeling. IEEE transactions on pattern analysis and machine intelligence, 40 (4):876-890, 2017.
Max Welling, Michal Rosen-Zvi, and Geoffrey E, Hinton. Exponential family harmoniums with an application to information retrieval. In Neural Information Processing Systems, 2005.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2645-2644, 2016.
Jianwen Xie, Song-Chus Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 7093-7101, 2017.
12

---

Published as a conference paper at ICLR 2020
Jianwen Xie. Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for shape synthesis and analysis. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8629-8638, 2018.
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based spatial-temporal gener- ative convnets for dynamic patterns, IEEE transactions on pattern analysis and machine intelli- gence, 2019.
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversarially regular- ized autoencoders. In International Conference in Machine Learning, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference Computer Vision (ICCV), December 2015.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 TOP-K AUTO-REGRESSIVE SAMPLING
In this subsection, we factorize the joint model BIT-BASE auto-regressively. and compare its dif- ferences with BASE LM. Since even estimating the per step probabilities according to Eq. 5 too computationally expensive, we further approximate by only considering the top 128 words pre- dicted by BASE LM, where we sample 10,000 completions for each of them to estimate P( [ Then we take the top entries and re-normalize, and compare the top 10 probabilities of BASE LM.
Our initial explorations suggested that the joint model tends to generate fewer repetitions. Therefore we picked few LM samples where there are repetitions at 2t,, and use the same context Iect to estimate P(x =x) for the joint model. Some examples of ((xr(<<e) of BASE LM and BIT-BASE presented in Table 3 Indeed BIT-BASE usually assigns lower probabilities to repetitions even though the top k words remain the same, which is not surprising given that the existence of repetition a strong indicator of coming from the LM, which would lead to higher energy value hence lower joint probability.
Context Tec
Model
Rank
It.
P(wf(ree))
0
binding
0.39
+. is aimed at setting common benchmarks for orderly migration practices, thereby reducing irregular flows. The Global Compact contains ten guiding principles, including that migrants cannot be settled by countries with better integration policies and a fair and sustainable development. "For the first time in our history, legally binding and
1
legally
0.33
BASE LM
2
internationally
0.06
3
comprehensive
0.05
4
transparent
0.04
0
binding
0.18
1
legally
0.17
BATE
2
internationally
0.12
3
comprchensive
0.09
4
transparent
0.08
i companies that land their first-choice candidates 90-190 100% of the time, 24% of them have "thoroughly defined" their high performer attitudes. By contrast, only 1% of companies that struggle to land their first-choice candidates "thoroughly defined" their high
0
able
0.66
1
willing
0.09
BASE LM
2
eager
0.07
3
ready
0.05
4
well
0.04
0
ahle
0.75
willing
0.05
performer attitudes. So it seems pretty clear that companies that land their top-choice candidates are not always as willing and
BIT-BASE
2
eager
0.05
3
ready
0.04
4
well
0.03
E.. it reveals a key skill needed to lead the Fed. "You need to know what you don't know. And
0
banking
0.64
1
financial
0.10
BASE LM
2
insurance
0.09
you need to be willing listen when you don't know something." said Karen Dynan, who an assistant Treasury Secretary in Barack
3
technology IT
0.05
4
0.04
0
banking
0.92
Obama's 's second administration would regularly meet Fed governors. iEOS; New Delhi Dec 5 The following mergers under review by India's 's financial services and
1
financial
0.06
BAT-BASE
2
insurance
0.01
3
technology
0.00
4
IT
0.00
Table 3: Comparison of ((xe(ze)) between BASE LM and B:T-BASE few examples. marked with red. Only the top probabilities are shown.
Repetitions
Excerpt
from
swiss- ort.:p/wwwoo..oeoooo..cco.aooaar
http:///www..ecrocooo/o//aaaaaaa 125917c765dan
Excerpt
from
this-
EEceerpt
from
fed
14

---

Published as a conference paper at ICLR 2020
A.2 PROOF OF THEOREM 2
Theorem 2. Denote Tn as the empirical estimate log Be- exp( - [x(x)) with n samples an ~
PiM(i 1, ,,), and let Tn log 1 Σ (ipp-EE(op (xxx))) then Ve > 0, 3N > such that
vn > N we have
7o - <TT Zo [2m 1)T, - -2(n 1yT < Za +
Proof. From Nowozin (2018) Eq. 35, we can write B[Tn]]
112 B[T]] Zg
A3
22
+
1
(ug 3/22)
2p² n
3rd² n²
4p
n
n
10
n M3H22 +
n (us 10HzHH2)) + o(n (8)
5pc
Where μ == [[[T,, Mi [T, p))1. Equivalently.
P2 + o(n
ETT,l Ze
(9)
2μ²2 n
Therefore, lim [[Tn] = Zg. So > 11 such that when n > N1, [[T]] > Zp- - c. On
the other hand, limm-roo n(2, Ze ]] lim 2h o(1) / > 0, so iN₂ > 0 such that
when n N2 we have Ze E[T,]. Up to this point, we have proved that Zg T Zg.
For the other half part of the proof, using Eq. 8 we have
T = Zo
n2
+ n²
+ o
(10)
2p²
where c is a constant. Therefore, B[(2n - 1)T, - 2(n - 1)Tn (2n - 1)E - 2(n
1) Zg + n22
2μd o(n (). Therefore lim Be 1)T,
2(n 1)Tn-1] Zo.
hence Vc > 0, NN3 > 0 such that Vn > N₂ B[(2n - T)T, 2( 1)T < Zg +c. Furthermore,
lim n (2n 1)T, 2(n 1)T Zo) 1om 2n²
#2 + o(1)
> so EN₂ 0 such
that when n > N4 we have E[(2n - 1)7, 2(( 1)T > Zg.
Putting the above together, Vc > 0, let N =mmax -max N2, N₂, Na then Vn > N,
Ze [TT,] Ze (2n 1)7h 2(n -) Zg +
A.3
OPTIMIZATION SETTINGS
Model
fp16
batch size
warmup steps
max steps
max
max grad norm
LM
32
2,000
180,000
0.0001
10
RALM
2,000
180,000
0.0001
10
BALM
32
2,000
180,000
0.0001
10
JOINT UNIT
64
2,000
180,000
0.0003
10
JOINT BIT-BASE
60
2,000
90,000
0.00005
0.25
JOINT
60
2,000
90,000
0.00005
0.25
JOINT BIT-LARGE*
64
2,000
90,000
0.0003
10
BASE LM-24L
50
2,000
90,000
0.0003
0.25
RALM-24L
28
1,000
90,000
0.00015
0.25
BALM-24L
28
2,000
90,000
0.0003
0.25
JOINT UNIT (LM-24L)
64
2,000
180,000
0.0003
10
JOINT BIT-BASE (LMM-24L)
60
2,000
90,000
0.00005
0.25
JOINT BIT-BASE* (L.M-24L)
60
2,000
90,000
0.00005
0.25
JOINT BIT-MED (LM-24L)
32
2,000
90,000
0.00005
0.25
JOINT BIT-LARGE* (LM-24L)
20
2,000
90,000
0.00005
0.25
Table Optimization settings. We the setting for CC-News and Toronto Book Corpus.
The optimization settings are presented in Table
15

---

Published as a conference paper at ICLR 2020
pairs below decide which more reasonable extension initial words. not worry if one both extensions incomplete.
'If you try to tinker with this without the tools that only Congress has, you are as likely to break the cloud as you are to fix it,' he said. Google, which has waged similar battles with the government, and an array of other leading tech companies are supporting Microsoft Ginsburg suggested the wait-for-Congres approach had some appeal. 'Wouldn't it be wiser just to say, 'Let's leave things as they are-if Congress wants to regulate in this brave new world, it should just give up," Ginsburg said, according summary companies have history of fighting government regulations court, and have...
case. Justices Sonia Sotomayor
Bader
opinion
high court's concurrence.
you try to tinker with this without tools that only Congress you you to fix it,' said. Google, which has waged similar battles with the government, and array of other leading tech companies are supporting Microsoft in the case. Justices Sonia Sotomayor and Ruth Bader Ginsburg suggested the wait-for-Congress approach had some appeal. 'Wouldn't it wiser just to say, 'Let's leave things they are-if Congress wants regulate this brave world, it should regulating this brave new world?'; wrote Sotomayor and Bader Ginsburg. A ruling is due by the end of June. it's approved by Congress, court could...
likely to break the cloud
Figure 4: Screenshot of the human evaluation.
A.4 HUMAN EVALUATION
screenshot of the human evaluation experiments can be found Fig 4 Every page asks for comparisons, one of which we know what the ground truth answer is. We subsampled 333 sen- tences from the test set of CC-News, and asked 3 Amazon Mechanical turkers to vote. We consider continuation better if it gets formed a qualification task beforehand, where one of the continuations is real text, and we kept the top half performing turkers for further evaluation (corresponding to higher than 66.67% accuracy for discriminating real from LM samples - for a total of 26 qualified turkers). Then in the actual experiment, we one out of every four comparisons the turker did not pass the check.
votes. To check the quality of the received ratings, per-
attention check and drop responses
16

---

Published as a conference paper at ICLR 2020
A.5 QUALITATIVE ANALYSIS
We present generation examples when our approach BASE LM outperforms baseline BALM in Table 5 and when our approach underperforms in Table 6 Here the judgment is based on human evaluation when all three turkers unanimously voted in favor of one model over the other.
Prefix
Model
Generation
And the fact that the money is corning from the govern-
The timing of the donation couldn't it be beller, and even relevant than Chance
BASE LM
ment, it makes big diffcrence," he said. "We're the
to say that right now: Thank you to Dong and @jewelosco for donating $1,000,000 The New Chance Arts Literature Fund, which will ensure more students have access arts enrichment education! tsapportips pic
biggest donor of arts education, so to
and doesn't t oven have the kind of resources or funding
BALM
for the arts literary programs that we need do the things
that peuple want to supponi" said Chunce.
sad Cpl. Chris Bissormette of the Stony Brook Police
BIT-BASE- WORST
SocialWorks
Department, "So the reason is, as think about the impact of
(@) SocialWork Chi) November 20, 2017 "In the pasi.
these grants and these grants schools
CPS hasn't 't really gotten lot money from non-profits
to lund programiming. So, I Uhink that it a great time for us to do something to support that." said Chance. "We want to make that programs
BIT-BASH
and charitable arms and organization because there's 's been a past of coruptt activity in the city of Chicago," Chance said, "And so, when it comes down to it, it
GOLD
get his hands on the hall this week. He threw two touchdown passes on Thursday. He should get another touchdown as the goes on. He's got good chance be one of the Chiefs and the Chiefs defense has made them pay for job with ball. and they's been a factur. It's nice to have
To
quarter. The penalties are still somewhal of a problem but tight ends Travis Kelce and Demetrius Harris made some impressive grown-man foothall plays, -It was nice to see romning back Karcem Hunt get in the end for the first time since Week 3. He must leel good to end the drought. Kelce was visibly frustrated on the sidelines and rightfully so. The officials seem to leaning loward Oakland with calls today. Still, Kelce should've ve caught that easy pass that he dropped. Quarterback Alex Smith has delivered couple of nice deop balls to
BASB LM
BALM
The otfense has done
right end Martavis Bryant to get ball back, hut throw from Smith to Davis Webb is incomplete, which could lead to an incomplction. -The Chiefs are now 5-2 after winning help the Chiefs win this gume. His completion ball to wide receiver Sammy Watkins is a nice sign. -Jalen Ramsey should be back the lineup today. The Jaguars are going
BIT-BASE-
WORST
deep
BIT-BASE
receivers Albert Wilson Tyreek Hill. The footwork by Wilson on the doop ball was amazing and Hill's ball-tracking skills were on point as usual. Good things happen when Smith trusts
GOLD
can't say "no I've never seen something of that value, I'll try and find again and see what happons." So don't 't be afraid to find a way to detox or
amazing ability detoxify itself namely by
BASE LM
the liver, kidocys, and skin." she says, Detox programs and diets do sometimes make people better, that happens more often you'te re eating and/or drinking things that make
want get the full picture and not just have to look at the whole picture," she says. "It can be a great idea to talk to your doctor deternine what's causing your symptoms." " can get rid of toxins that are found in foods like whole wheat bread, nuts
BALM
sound like Kundashian cmpp (something doesn't doing in the first place).
BIT-BASE-
And if, somehow, you discover that you have elevated levels of mereury or lead your body like Kardashian said she bad, it's impostant to figure out why. Dr. Wider says. "Finding source extremely important For any heavy metal, so you
WORST
walnuts, walnuts and walnuts," she says. "If sourec of the metals has elevated levels of can't say, "Oh my God, it's lead, but I'm tine." But it's definitely problem problem in
BIT BASE
bodies," she says. "And it's
can lower your exposure over time." she says. Meaning, a detox diet probably isn't geing to help you as much as, say, reducing your inlake foods that are high mercury (like
GOLD
Table 5: Example generations when BASE outperforms BALM according to human evaluation. BIT- WORST shows the LM sample with the highest energy score.
9 from http://wwec...ooo...../ee//eeeer raiderssddddeeee week-swe
10 "Exceppt frem paw.gng......g.r...o..ooooooooee
Excerp https://www... amour, https///.....g.ooo.ooooo/oo/ooos
17

---

Published as a conference paper at ICLR 2020
Prefix
Model
Generation
later, she was dend, according the lawsnit. "It's not uncom-
the audio only came on hullway through the incident, did not record his initial words to the girl or her response. It also dropped in the hallway during their interaction, so it did not record any of what happened once the two left librury. Jones brought her outside to the library's 's front steps, where he waited for responding officers to arrive. They did not call for umbulance, uccording the lawsuit. Ruther, when the officers arrived, they told her to spic blond on the grass instend the concrete, and move from the steps to a bench. Some 20 minutes
BASE LM
mon for an oflicer lo sgrit blood on an unarmed, vulnerable adult. The use spit is illegal in public place."
later, Jones was arrested for diserderly conduct and released after posting bond, the suit said. She was arrested again on May 2. the suit, Jones is seeking more than $1 million. later, he walked up on the ground, where they began to yell and yell again. When Jones refused, the officers ordered him not to leave or leave any more. Joncs then pulled his gun on after officers' arival, Jones and the girl the library and headed to the parking lot of nearby apartment complex. "This the type of behavior that is most likely to cause harm
BALM
BIT BIT-BASE- WORST
BIT-BASE
later, JKGss morteer arrived and took her daughter to Cleveland Clinic Lakewood, where she treated for dislocated jase, a bead injury, and an injury to her knoe.
GOLD
the Vanily Track & Field Invile. AnThe U.S. Army Corps of Engineers has upproved 52 billion in construction work along the U.S.Mexico boundary as way
Bronson said, "The initiative provides variety of supports early childhood programs' children, familics and staff. The resourees provided through this portnership increase the quality of the participating p imgucts
BASE LM
Check back froquently for updates. The Seatincl also has a
leature buys basketball and wrestling teans. Boys goll' The Sentinel boys goll and wrestling teams have been one of
BALM
prograns, which benefits the community our future in such positive way." Scott said. Visit PNCGrow UpGreat. com/donorschoosee \nllere are Varsity sports headlines for April 13, 2018. Refresh to get the lntest as we add news throughout collect scores: Best of the best track and field Our Sentinel coverage area top performers lists for girls track and field
those teams
had some
ABLONDON, April 13 (IFR) The Following are some of the
BIT- BASE-
main factors expected to affect Spunish corporate hand prices Wednesday. BAML: Spanish sovercign wealth fund PPV AnA few weeks back, it's been rumored that the HTC Desire was going to the compuny's 's last flagship phone, and now, a new leak has confirmed that it
WORST
night we
BIT-BASE
and boys track and hield are updated going into the Saturday district meets. The season is heating up with more district and region races coming up next week. Click these links for girls top performers and boys top
UAA), General Motors (NYSE:GM) on November 4; and Procter & Gamble (N) for October. On the retail froat, Lowe's 's Companies (NYSEL
I snders meeting November 17-18 Papua New Guinca as potential Xi-Trump meet dates. If all clse fails, Trump Xi are expected meet the G20 meeting at the end of November. On the ecomomic calendar nexl week, the update on jobs and the U.S. trade deficit are the healliners on November Notable carnings reports: Akamai Tochnologies (NASDAQ Mondelez Intemationna (NASDAQ;MDLLZ and Olin Corp. (NYSE:OLN) on October 29; Under Arnour (NYSE:
BASE LM
bit
UA) October 30; CVS Tealth (NASDAQ:CVS)
BALM
November 27; Intel Corporation (NASDAQ:INTC) on Oc- tober 28; Verizon Communications (NYSE:V and ADDYXFF OLYMP), on November 30; and Qualcomm Incorporated (NASDAQ:
BIT-BASE-
[AA]
Adidas
(OTCF::ADDYE:
WORST
UAAI, Johnson Controls (NYSEICI) and Cisco Systems (NSSDA November 6. 6.A woman who had have her nose and mouth taped as punishment
BIT BASE
UAAL. cBay (NASDAQEBAY), Gencral Elsetie
GOLD
(NYSE:GF). Coca-Cola (NYSE:KO), Pfizer (NYSE:FF)
and Electronic Arts (NAS
Table 6: Example generations when BIT- BASE underperforms BALM according to human evaluation. BIT- AORS- shows the I.M sample with the highest energy score.
from ht htap://wwwsoeooo..oaaaaaaaoaaeao oh mom sues library over Leens rough LzeaLment
Excerpt from ht http:///wwww.ooooeeeee////////// sun/I1 def village academy steam 0118 20180410 story.
mtml Excemt from httpe//eecccoooocaeeooeoeoaaaaae
18


Reviews:
[{"review_id": "Syg4BwvptB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nContributions:\n\nThe main contribution of this paper lies in the proposed Residual Energy-based Model (EBM) for text generation. Traditional normalized models operate at the token level with MLE training, while the proposed EBM operates at the sentence level. Therefore, BERT, or RoBERTa, can be leveraged for EBM design. The residual energy function is trained via conditional NCE, which reduces to training a binary classifier to discriminate between real text and text generated by an auto-regressive language model. After model training, text can be generated by top-k joint sampling. Experiments are conducted on two large language modeling datasets. The proposed model achieves lower perplexity, and preferred by human evaluation.  \n\nStrengths:\n\n(1) Writing & Clarity: This paper is well written, easy to follow, and clearly presented. I enjoyed reading this paper. \n\n(2) Novelty: The proposed model contains some novelty inside. It is framed in a residual EBM framework, though by the end, the residual energy function reduces to training a binary classifier to discriminate real and fake text. Though simple, this idea is wrapped up in a nice framework. It is also interesting to observe that this sequence-level EBM regularization can be considered as a way to fine-tune BERT for the text generation task.\n\n(3) Experiments: Generally, the experiments are comprehensive. Detailed analysis, and human evaluation is also provided.\n\nWeaknesses:\n\n(1) Clarity: I have some concerns regarding the selection of baselines, with details shown below.\n\nThis paper is basically about using BERT as a binary classifier, which serves as a residual energy function to regularize a pre-trained language model and provides sequence-level supervision. The experiments are comprehensive, but on the other hand, it is also quite expected that the proposed model should work better than an MLE baseline, since sequence-level supervision is provided. \n\nI think if the authors want to make a stronger paper, they should also compare with other possible ways to inject sequence-level supervision. For example, a simple solution is to use GAN, like in a SeqGAN setup. And the discriminator in the GAN will be the same BERT-based binary classifier. In this GAN setup, sequence-level supervision is also provided. \n\nThen the difference is that in the GAN setup, the BERT-based binary classifier is a discriminator, but in this paper's setup, it is a residual energy function. It would be interesting to discuss and conduct experiments to see which way is better. \n\n(2) Experiments: I have some concerns regarding the experimental setup. \n\na) One of the main results is Table 1, which reports all the PPL numbers. However, reporting PPL results is less interesting, because we also care about the diversity of generated samples. Lower PPL does not necessarily mean higher-quality text. Though Figure 2 provides some analysis on the diversity, a more comprehensive evaluation on this will be appreciated. \n\nb) It will be good if the authors can also provide some generated samples for qualitative analysis. \n\nOverall, I think this paper is well executed. The paper is well written, and experiments are carefully conducted. However, on the other hand,  I also think the conclusion in this paper is expected, it only shows that the proposed model is better than an MLE baseline. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyl9u-SX5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors make good points, starting from the exposure bias and label bias suffered by the mainstream neural auto-regressive models.\nResidual EBMs are defined and trained using NCE. Experiments on two large language modeling datasets show that residual EBMs yield lower perplexity and generation via importance sampling is of higher quality, compared to locally normalized baselines.\n\nIn generally, the paper is well motivated and interesting. But I have some concerns.\n\n1. Missing important relevant references.\n\nEBMs (a.k.a. un-normalized models, random fields) have been successfully developed in language modeling in recent years. A large body of this paper has been studied in [5,6], including the model and the NCE estimation method. The model Eq.(2) is exactly the model in [5], defining the model in the form of exponential tilting of a reference distribution.\nConnecting and comparing to these previous works are needed.\n\n[1] R. Rosenfeld, S. F. Chen, and X. Zhu, “Whole-sentence exponential language models: a vehicle for linguistic-statistical integration,” Computer Speech & Language,  2001.\n[2] B. Wang, Z. Ou, and Z. Tan, “Trans-dimensional random fields for language modeling,” ACL, 2015.\n[3] B. Wang, Z. Ou, and Z. Tan, “Learning transdimensional random fields with applications to language modeling,” IEEE transactions on pattern analysis and machine intelligence, 2018.\n[4] B. Wang and Z. Ou, “Language modeling with neural trans-dimensional random fields,” IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017.\n[5] B. Wang and Z. Ou, “Learning neural trans-dimensional random field language models with noise-contrastive estimation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.\n[6] B. Wang and Z. Ou, “Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation,” IEEE Spoken Language Technology Workshop (SLT), 2018.\n\n2. I am a little bit concerned that the theoretical contribution seems weak. \nThough Eq. (4) and (5) seem to be novel, I am not sure whether such a contribution is substantial enough to motivate acceptance.\n\nI'm happy to adjust the score if the paper can be better placed in the literature and the authors take efforts to improve the paper.\n\n--------update after reading the response-----------\nBeing well-placed in the literature and properly claiming contribution with respect to prior work is one of the key questions in reviewing a paper. The first version of the paper clearly lacks in this respect. That's the main concern when I gave a 1.\n\nI appreciate the authors' response. The updated paper has been improved to address my main concern, although the added discussions presented in the updated paper is not as clear as the authors' clarifications in the response. I suggest to polish the main text incorporating these clarifications.\n\nGenerally, it is nice to see the successful application of energy-based/random-field-based models in text generation, besides in speech recognition. I update the score to 6 (Weak Accept).\n\nIt would have been better that the following can be further clarified.\n\n\"the partition function estimated via importance sampling would lead to bias favoring the random field language model\" --- this comment is not clear to me. \n\nBoth Eq.4 and Eq.5 give estimates for perplexity. It would be better to clarify different uses of the two equations. If the perplexities are estimated using Eq.4 (as in Table 1), then what is the purpose of developing Eq.5?\n\nHow to calculate the lower and upper bounds of the step-wise perplexity gain at each position in Figure 1?\n\nUnder Figure 1, \"At each position the lower and upper bounds (see Eq. 4) are estimated using 20,000 samples.\" But in the main text, it is said that \"We therefore break down perplexity per position in the generated sequences as in Eq. 5\" at page 8. It is confusing.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJggab7ccS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is an interesting extension of Gutmann and Hyvarinen (2010), where the parametric model is the combination of a noise model (language model) and an energy function (residual energy), so the difference of parametric model and the noise model cancels out the noise model. Therefore optimizing (3) under some conditions converges to a set of parameters of the parametric model (P\\theta(x) here) that best describes the data.\n\nOne important assumption of Gutmann and Hyvarinen (2010) is that there exists a set of optimum parameters for the parametric model such that the probability of data and the parametric models match for these optimum parameters. This should be mentioned in Theorem-1.  \n\nDoes Theorem-1 need extra parameters to act as a normalization constant in order for the theorem to hold at the optimum?\nlog P_lm(x) - E(x) + const = log p_data\n\nTo sample from the model, the authors first sample from the language model and re-sample it with respect to the energy values of the residual model.\n\n\nTo compute the perplexity, they have given an upperbound and lowerboud for the partition function based on number samples in Theorem 2, but I haven't checked the correction of the bounds.  They also factorize the joint model in auto-regressive factorization to compute the perplexity by approximate marginalizing. \n\n\nAs mentioned in Section 5, this approach heavily depends on a strong pretrained language model. \n\nHave you considered improving the language model during training?\n\nThe described idea is simple and effective and I really liked it.\n\n\n--- Based on other reviews and the authors' response (especially review #3), I reduced my rating to 'Weak accept'. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Residual Energy-based Model for text generation.

After rebuttal and discussion, the reviewers all converged on a vote to accept, citing novelty and interestingness of the approach.

Authors are encouraged to revise to address reviewer comments.
Decision: Accept (Poster)