Calibration, Entropy Rates, and Memory in Language Models
Authors: Mark Braverman
Xinyi Chen
Sham Kakade
Karthik Narasimhan
Cyril Zhang
Yi Zhang
Venue:  (2020)
Keywords: information theory
natural language processing
calibration

Abstract: Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are \emph{miscalibrated}: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibration-based approach can also be used to measure the amount of memory that language models use for prediction.

Under review as a conference paper at ICLR 2020
CALIBRATION, ENTROPY RATES, AND MEMORY IN LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Building accurate language models that capture meaningful long-term dependen- cies is core challenge in natural language processing. Towards this end, we present a calibration- a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable meth- ods to mitigate this phenomenon. Furthermore, we show how this calibration- based approach can also be used to measure the amount of memory that language models use for prediction.
approach to measure long-term discrepancies between
INTRODUCTION
Recent advances language modeling have resulted in significant improvements on a wide vari- cty of benchmarks (Dai et al., 2018; Gong al., 2018; Takase et al., 2018). Capturing long-tern dependencies has especially been a major focus, with approaches ranging from explicit memory- based neural networks (Grave al., 2016; Ke al., 2018) optimization improvements to stabilize learning (Le et al., 2015; Trinh et al., 2018). However, while these techniques seem to improve on standard metrics like perplexity and even produce remarkably coherent text (Radford et al., 2019), we still do not have appropriate measures to assess long-term properties in language models, making it difficult choose between different model options for downstream tasks.
Starting from Shannon's seminal work that essentially introduced statistical language modeling (Shannon, 1951), the most classical and widely studied long-term property of language model is its entropy rate - the average amount of information contained per word, conditioned on the preceding words. A learned model provides an upper bound for the entropy rate of language, via its cross-entropy loss. The exponential of the entropy rate can be interpreted as the effective support size of the distribution of the next word (intuitively, the average number of "plausible" word choices to continue document), and the perplexity score of a model (the exponential of the cross entropy loss) is an upper bound for this quantity. In state-of-the ff-theear art models trained on billion-scale corpora, this number ranges between 10 and 30 (Melis et al., 2017; Radford al., 2019). A natural diagnos- tie question, with which we begin our work, is whether the long-term generations of these models exhibit the same entropy rates as the underlying languages they modeling predictively.
Model
Corpus
Test ppl.
EaiRale
AWD-LSTM (Merity et al., 2017)
PTB
58.3
93.1
CNN-LSTM (Jazetowicz et al., 2016)
GBW
29.8
49.4
Transformer (Vaswani al., 2017b)
GBW
28.1
34.7
GPT-2 (Radford al., 2019)
WehText
23.7
61.2
Table 1: Perplexity degradations for generations from popular language models. State-of-the-art performance is usually reported via perplexity with respect to the test corpus (one-step prediction loss), but there is striking blowup in the perplexity (i.e. exponential of the entropy) of these models' long-term generations. Test ppl. is the exponential of the cross-entropy the model with respect to the test corpus.

---

Under review as a conference paper at ICLR 2020
Empirically, and perhaps surprisingly, it turns out that the entropy rate of generated text is substan- tially higher than the estimate for true text derived from the model's one-step predictions. As seen in Table 1 (see also Figure 1), this is true for both state-of-thee ofrt LSTMs and Transformers trained variety of datasets. As a timely example, the GPT-2 model (Radford et al., 2019), the object of much recent attention for its seemingly coherent and on-topic generations, suffers dramatic degradation in its entropy rate, from 23.7 to 61.2.
This empirical finding notable since the neural attention- and memory-based techniques have been steadily improving on standard metrics like perplexity and, in some cases, even produce remarkably coherent text (often with some heuristics to reject poor generations). That the perplexity of generated text is so much higher than it is under the true distribution suggests that there are significant gaps in our current methodologies in accurately learning language models, particularly if we are interested in generating text that globally resembles the modeled language itself.
The focus of this work is twofold: to improve generations hased on any mea- surement mismatch long-term property of the model (e.g. the entropy rate), and quantify the way a model's predictions depend on the distant past. Central to both of these is a calibration- based approach, which utilized in statistics and other areas of machine learning (Dawid, 1982; 1985; Foster, 1991; Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017; Niculescu-Mizil and Caruana, 2005).
Our contributions.
First, we show that, from worst-case perspective, even an extremely accurate model (with E average KL divergence from the true distribution) may have generated text with a substantially different entropy rate as compared to the true distribution. Indeed, we show that this worst-case amplification may occur for a variety of long-term properties of a probabilistic language model; this is because the one-step KL divergence does not in general provide tight control over the expectation of bounded function. The observed entropy rate amplification (as seen in Table 1) demonstrates that this is not only of theoretical concern. We then describe calibration procedure to fix this mismatch while simultaneously improving the perplexity of the language model. From a statistical perspective, the procedure is simple, and discuss approaches to make it computationally efficient.
Second, we provide a definition for long-term memory in language models as the mutual information between the models predictions and the distant past in the input. We then provide upper bound on the amount of this mutual information using calibrated distributions (with a exponent). This allows us estimate the amount of context used by language model a function of the distance of past tokens from the current prediction time step.
We perform empirical studies accompany our theoretical results. We first the entropy rate calibration algorithm to fix an LSTM language model, resulting in a drop of around 20 perplexity points the generated text (so that the entropy rate of the model more accurately matches that of the language itself). Then, we empirically estimate and compare the long-term memory of state-of- tae-art language models. Our insights point towards new ways of assessing (and fixing) language models, especially in terms of their long-term properties, in a manner complementary to existing metrics like perplexity.
2 RELATED WORK
Improving language modeling with long-term dependencies. Recent approaches to improving language modeling have focused on several ways to better capture long-term dependencies, from using manually-defined context representations (Mikolov and Zweig. 2012; Ji et al., 2015; Wang and Cho, 2016) or document- topies (Wang et al., 2017) to using L.STM recurrent neural networks with careful initialization (Le et al.. 2015), auxiliary loss signals (Trinh et al.. 2018) or augmented memory structures (Grave et al., 2016; Ke et al., 2018). Wiseman and Rush (2016) use scoring functions over sequences and search-based optimization to improve generation in seq2seq models.
More recent work has demonstrated the upplicability of Transformer networks (Vaswani et al., 2017a) to the task, potentially side-stepping issues in training recurrent networks (e.g. vanish- ing/exploding gradients) and scaling to longer contexts (Dai et al., 2018; Radford et al., 2018). All these papers propose either architectural or optimization innovations to improve language model
2

---

Under review as a conference paper at ICLR 2020
training. In contrast, we define and measure explicit long-term properties of language models and show that calibrating them correctly can provide improvements to any black-bor language model.
Recent empirical breakthroughs have stemmed from language models which do not specify unique autoregressive factorization (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019), and thus do not specify unique Pr. It remains an interesting problem to identify and sample from distributions induced by these models (Wang and Cho, 2019); thus, our end-to-end theoretical guarantees do not hold in this setting.
Information- approaches. While most language models aim predict distribution over the next token conditioned on the context, there have been alternative approaches relying on anformation-theorede measures, Jost and Atwell (1994) propose model which makes use of mutual information between word pairs to generate word sequences that retain longer-term dependencies. McAllester (2018) propose a training objective based on mutual information for predictive modeling, and demonstrate its application for phoneme prediction. Clarkson and Robinson (1999) develop a hybrid metric using both perplexity and entropy rate, and show that it correlates better with a downstream metric like word error rate. Such works propose alternative optimization objectives; in contrast, we show how to use ceeemation-thhooede measures to improve models with respect to existing objectives like cross-entropy.
Measuring long-term statistics. Khandelwal al. (2018) analyze LSTMM-aased language models and empirically show that such models make use of a finite context for prediction. Lin and Tegmark (2017) measure mutual information between any two symbols in human languages, and show that it decays with distance, roughly following power law distribution. Takahashi and Tanaka-Ishii (2018) provide an upper bound for the entropy (character-level) of human languages by training neural language models with various context and data sizes and extrapolating to infinity. While also make use of measures like entropy and mutual information across longer contexts, our goal is use these better calibrate the language model and provably improve its perplexity.
Calibration and integral probability metrics. The idea of matching properties of the models' predictions to the empirical outcomes, in online setting, goes back (at least) to the "prequential principle" of Dawid (1982; 1985), with subsequent work in online and game-theoreti, settings (Fos- ter, 1991; Vovk, 2001; Kalai et al., 1999). The idea of improving probability scores is also common in machine learning (Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017: Niculescu- and Caruana, 2005). Recently, Ott et al. (2018) assessed model calibration for machine translation systems using word-level probabilities. The notion of examining the expectation of functions as a metric for the distance between two distributions sometimes goes under the name of integral prob- ability metrics (Mller, 1997; Sriperumbudur et al., 2009), and this notion is becoming increasingly relevant again in unsupervised learning through the connections to GANs (Mrouch and Sercu, 2017). In this work, we directly focus on the KI. divergence, where our use of calibration is largely based on basic facts about exponential families (Brown, 1986).
Relation to inneration-improve heuristics. If the sole objective is to improve the qualitative coherency of sampled generations. a wide variety of heuristics exist the literature. The simplest of these is a constant multiplicative adjustment to the model's logits (known as softmax temperature Xie (2017)). This is a specific version of our method (Algorithm 2) with constant logistic regression feature instead of the next-token conditional entropy. Relatedly, greedy and top-k sampling (used in state-of-the-an works such as (Radford et al., 2019)) are heuristics which make local modifications to the model's conditional probabilities to decrease diversity and eliminate nonsensical generations.
Efforts to push the empirical state of the art in generation quality have given rise to more complex heuristics. Bengio et al. (2015) propose retraining the network its own generations with care- fully scheduled probability for each token. Some works regularize a model's generations with an auxiliary reverse language model Zhang al. (2019); Liu et al. (2016). Yet others promote realism using adversarial training protocols (Bahdanau et al., 2016; Lin et al., 2017; 7; Fedus et al., 2018).
We stress that calibration methods result in provable improvement in the original training objective (i.e. lower perplexity). As far as we know, none of the aforementioned heuristic meth- ods can hope provide such strong guarantee, since they are fundamentally designed to bias
3

---

Under review as a conference paper at ICLR 2020
models towards different objective. Our work mitigates model hallucinations for (almost) free', in the sense that the global objective (entropy rate drift) is improved without worsening the local objective (perplexity). Furthermore, calibration preserves the computational efficiency of density estimation: conditional probability vector from Algorithm 2 can be computed using Ovvocabulary size) inferences the original model. In the more advanced heuristics, the implied distribution over sequences is lost, and is only accessible by black-box sampling.
PRELIMINARIES
We first define some useful quantities for our analyses. Let Pr(Wi, W2...., WT) represent the true underlying distribution over T length sequences of words, where the vocabulary is of size M. Let Wi. denote random sequence length T, with distribution Pr(Wi:r). For clarity of exposition, assume that all sequences (i.e. sentences documents books) are of equal length T.
For any distributions D and over length-T sequences, recall that the entropy H(.), KL- divergence, and entropy rate are, respectively, defined by: H(D) = Ew loog D(Wir=wwr)
KT,(D D') == E w log D=WT)= and EntRate(D) H()). Let Pr(Wir)) (((W)===i:r)
denote learned distribution over sequences. In the typical sequential prediction setting, the proba- bilistic model is implicitly defined by the conditional distributions Pr(W,(W<))) which are typically efficiently computable. It is standard for such language model to be trained to minimize the cross entropy objective:
CE(Pr
Pr)
1 T
E BITTTPP
Σ log Pr(w,lwe.)
log Pr(wir)
Note that for an accurate language model, we would hope that: CE(Pr Pr) se EntRate(Pr) ie. the entropy rate of the sequences generated under the learned model is nearly that of the cross entropy of the model (with respect to the true distribution Pr).
Throughout, we assume that
KL(Pr Pr) CE(Pr || Pr) EntRatc(Pr) S
holds for some E. other words, the (unknown) measures the degree of sub-optimality of the learned model, this is often referred to as the Bayes regret.
4 CALIBRATION AND ENTROPY RATES
In this section, we assess the long-term properties of language models when generating text. Specifically, we quantify the amplification the entropy rate of generations under an e-accurate model (Eq. 1). We then provide procedure to fix this amplification, without increasing the perplex- ity of the model. Proofs for all statements are provided in the supplementary material.
For generality, consider a function f: [M'] + R, defined on T' length sequences. Let the mean and variance of ff under distribution 'D be denoted by (D(f) and (f)
pD(f):=
Wi winDD [(WL)),,
E
obl))=
E
4.1 ERROR AMPLIFICATION UNDER MODEL
If our learned model Pr is accurate, may hope that (f) 1p; ie. that the expected value of f under the true distribution Pr is close to its expected value under our model. We can quantify this gap follows:
Technically, the statistical of fitting one more parameter.
4

---

Under review as a conference paper at ICLR 2020
100
Transfomen
60
et (LSTM)
90
50
80
40
70
60
20
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Generation length
Generation length
Figure 1: Perplexity (exponential of conditional entropy, given the past) of the t-th generated word, for two popular language models, averaged over more than 500 generation runs with different con- texts. At f 1, this is the model's upper bound for the language's perplexity. As t co, this is the exponential of the entropy rate of the model's own generations. For perfectly calibrated model, this curve would be flat (gray dotted lines). Left: LSTM trained Penn Treebank. Right: GPT-2 Transformer.
Lemma 4.1. (Pinsker's Inequality (Csiszar and Körner, 2011)) Suppose that for all Wirr f(w1r) S B. Then:
- -F s By 2PL Pr).
Since this holds for any bounded function, we can obtain the error amplification of the entropy rate of Pr simply by choosing f log Pr.
Before we proceed, order to rule out amplification of this entropy rate due to arbitrarily small probabilities (which can blow up - log Pr), it is helpful to define the 7-mixture distribution as: D(n) = (1 >)D + Uni, where the Uni is the uniform distribution over all M sequences.
We will then consider the model Pr (<)) which has only minor degradation in the cross entropy compared Pr, and, yet, may have large amplification in the entropy rate.
Corollary 4.2. (Entropy rate amplisication under generations) Suppose the bound in equation holds. The e-mixture distribution has KL bounded
T
KLL(PT
Pr (5)
1+ T
2 7
2
We have that:
CCEPP Pr (0)
EattRat((Pr)
1+
2
E and
T
CCE(Pr || Pr (c)) - EntRate(Prr (c) S 22(TT +1)
log M + log(1/E)
T
This bound shows that, in the worst case, even small cross entropy may provide little control over the generations under our model (in terms entropy rate). In fact, for e O(7) (which we may hope is accurate model), the bound is vacuous; later remark shows this worst case bound is unimprovable, see the supplementary material.
The above theorems suggest that entropy rate amplification is theoretical possibility in the worst case, which our experiments show is in fact prevalent in pratice. These entropy rate amplifications evident from the plots in Figure 1. Regardless of the text corpus the language model, observe that the entropy rate under the model's generations quickly increases with time, indicating that this is persistent problem for state-of-the-art language models while generating text.
4.2 MODEL CALIBRATION
We now describe a procedure to fix this error amplification. First, let us define distribution Pro such that:
(w exp(a[(wi.T)) Pr(wir)) where 7..-
Σ exp(af(wn:r)) Pr(wwrr)
Za
#ITT
We then recover calibrated model that does not suffer from error amplification in f::
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1 (Inefficient) Entropy Rate Calibration
1: Input: Model Pr (c)
2: Define a model class:
(wr(w))) (Pi(w))))(o))
IZa.
3: Fit a": a² = argmin CB(Pr || Pra)
4: Return Pra-
Lemma 4.3. (Calibration to f with model improvement) Suppose the variance of f is uniformly bounded in that there exists o such that the following holds for all a, opr, (f) S of Let argmin, CE(Pr || Pra.). . We have
1 (w(J) (F)
(fr(f) - PPr.. (f) 0, and CF(Pr Pra.) S CE(Pr PT)
T
20
Entropy rate calibration. We can now apply the previous result to fix the entropy rate amplifica- tion seen in Table 1. Note that it is trivial to avoid the entropy rate amplification if we were allowed degrade the quality of our model, in terms of perplexity (e-g. unigram model does not have this amplification. However, we show that it is possible to match the entropy rate without having to sacrifice the quality of our model. fact, we can both improve our model and more accurately match the entropy rate, by fitting a family of onepparameter models.
Theorem 4.4. (Entropy rate calibration) Suppose equation holds. Algorithm I returns a Pra- such that: the following calibration property is satisfied:
CE(Pr PTa-)) (Ptaate(Praa).
Furthermore, Pr.- has entropy close to the true entropy rate as specified by:
1 EntRate(Pr) EntRato (Pr. ) s 1 T E,
and Pro- an improvement over the original model as characterized by:
CE(Pr || Pr..)) S CE(Pr || (r)
1
CE(Pr Pr EatRate Pr)
log M + log(1/s)
T
This result shows that simply need single parameter a define new model class that is powered up version of our original model. Then, we can fit this a to minimize the cross-entropy of the new model with respect the true distribution Pr, in order to eliminate the entropy rate amplification.
Even though this algorithm fits only single parameter, not easily implementable since it re- quires an integration over sequences, at least in its exact form. One future direction would be to sample based approach. This may be an interesting alternative ideas like beam search (Steinbiss et al., 1994; Ortmanns and Ney, 2000; Antoniol et al., 1995), which also aims to minimize global cost function on sequences that is inconsistent with the token-level perplexity loss used train the underlying generative model.
Lookahead algorithms. In order to sidestep the computational issues of Algorithm 1, we provide another simple approach based what can be viewed as a "one-step" lookahead correction (Al- gorithm 2). Let W, be random variable with conditional distribution Pr)) H(W++al ( denotes the entropy of this conditional distribution, i.c.
(W++1lee ((ws)) =
log (1(e))alge)
~ Pr(-Iwee)

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Local Entropy Rate Calibration
1: Input: Model Pr (c)
where We Pr ( (-IWz)).
2: Define a model class:
Pro (wr) (11))aP (a
where
Pra(wslued = Pr(wslwer) exp (-a (.Q.H(WW)) HW aZa
3: Fit a": a² = argmin, CE(Pr
|| Pra)
4: Return Pra-
Note that H when predicting W2 using learned model.
includes the word W/, so we require computing the entropy at time t +
For a conditional distribution, D(W1:)). let us define:
T
E PDD = T MeeePP (.(w)))))))
1
Σ
E
Thus, jrD is the average of H(We+1lush) with respect distribution which uses D for sampling the last word W; (at every timestep). Intuitively, the resulting model Pra with a positive a would suppress sampling words leading to larger entropy but rather encourage words that stablizes the entropy 1-step ahead in the future. Therefore, our learned language model Pr was accurate, we would hope that: MPP z Pp, The following corollary shows that this is achievable, along with improving the model's perplexity.
Corollary 4.5. Suppose Equation 1 holds. Then, Algorithm 2 returns Pro- such that:
Pr (())
P pptt log M + log(1/e]
ПP: Ar..- = 0, and CE(Pr PIa-) S CE(Pr
This result provides with Algorithm 2, which is computationally quite tractable. We first use the learned model Pr to define new model class Pr.. which scales Pr by an exponential distribution over the weighted 1-step lookahead entropy И (We+1
(LSTM4)
e+1 [w<)). Then, similar Algo-
70
rithm 1, we simply fit the single parameter a to minimize the cross-entropy of new model with respect to Pr, which fixes the entropy am- plification in the resulting model Pr.. We ob- serve this empirically in Figure 2 - our cali- bration results in a perplexity drop of almost 20 points over long-term generations under an LSTM model. Model and implementation de- tails are in the supplementary material.
original
calibrated
40
60
80
100
length
Figure 2: Effect of calibrating LSTM genera- tive model with 1-step lookahead. Blue: perplex- ity curve (i.e. exponential of conditional entropy H) from the setting of Figure 1. Green: the same perplexity measurements after applying local cal- ibration.
Generations from calibrated model. Ta- ble 2 provides sample generations from a cali- brated Transformer model trained on the GBW dataset, compared to original version. Qualita- tively, the calibrated generations: (I) are shorter and more concise, and (2) display a better grasp of discourse structure across sentences. More generations are provided in the supplementary material.
CALIBRATION AND MEMORY
Defining notion of memory in language models is challenging, and multiple equally sensible no- tions may co-exist. Here we present our choice from first principles. Let us say that W, is sample

---

Under review as a conference paper at ICLR 2020
Original model
Calibrated model
Actual results could differ materially from those indicated by these forward-looking statements as result of various important factors, including without limitation changes in general economic and business conditions including more diffi- cult real estate environments: [...174 tokens...] risks related to investigations by other companies inadequate information systems the impact of reduced availability of assumptions upon such companies using such as ours to gauge CNET financial condition and other factors.
Actual results could difeer materially from those dicated by these forward-looking statements as result of a variety of factors, including but not lim- ited (i) the risk that the tender offer could close one or more manner or at all ; ii risks asso- ciated with conducting business in foreign jurisdic- tions iii difficulties in combining some or all of the businesses under one roof;, ; (iv) decreased de- mand for electricity natural gas and other energy products, including adverse effects on the pricing of oil and natural gas: and ( v the risks associated with doing business internationally Bluepoint Games, Inc. highly experienced li- censing gaming and entertainment firm focused on developing the next generation of casual games based on the PlayStation R BRAVIA family of video game machines for the North American mar- ket Bluepoint a wholly owned subsidiary of Bluchill ID Holdings L.P.
Bluepoint Games, Inc. is highly experienced and multi-faceted publisher of licensed virtual worlds for gamers, developers and technology profession- als l...114 tokens...J James Upon CEO of MyNetSShetetWeb and the three previous Develop- ers of MySQL.. Based in Redwood City Califor- nia, BlueMountain is the leader franchise and game development for the massively multiplayer online game.
Table 2:
Sample generations from a calibrated, state-of-the-art Transformer model trained on the GBW corpus, seeded with prefixes of sentences (in italics) from the holdout validation set. from model at time t, i.e. W Pr(W,IWer <t). Let us also assume that W<t Pr(Wes). We will define the memory at gap T as the mutual information between W, and the distant past (those words greater than steps ago) conditioned on the subsequence W.-ra-1- Precisely,
(W;; W <c-T We-r------)- H
H (W, (et)
where we are not explicitly denoting the dependence in this definition
Intuitively, I can be viewed as how much uncertainty (entropy) in the prediction W2 the model is able to reduce by utilizing the deep past Wer- in addition to the recent past Wt-rit-1-
The difficulty in estimating this mutual information is due to estimating H HWW.WW, (Wrr.a)). which requires the marginalized model Pr(W.[W To (even approximately) marginalize model distribution Pr(W. W<x) over the deep past W W<r-r is statistically difficult, since it requires the access to pool of samples of Wer that share an common recent past Wi-ri-1- Nevertheless, we now show that it is possible to obtain an upper bound (which is computationally efficient to estimate).
Upper bounding mutual information using calibrated models. In the above, we were consider- ing the mutual information between W. and Wet-r conditioned on W2-ræ-11 Let us now consider a more general setting, where we have distribution Pr(Z, X) where Z, Y, and X are random variables. We wil eventually consider Z, Y,Y,
X to be W., Wi W respectively.
For distributions D(-IY.X) and D- and for a E R. define
(2 = X) ( /Za.
We say that DI-I,, X) is calibrated to D0-1Y, X), if Da=o is unimprovable in that for all a CE(Pr || (D) s CE(Pr D.).
Note this condition is achievable due to that calibrating model D(-1Y. involves dimen- sional (convex) estimation problem (over a).
While we may attempt to estimate 1- for a given t, we can remove the dependence by either defining this quantity by with an average over f by using appropriate stationarity assumptions. In our experiments, average over t.
8

---

Under review as a conference paper at ICLR 2020
bound (nats)
CR(Pr] Pr))
1. upper bound
0.6
4.8144
0.6180
0.003515
10
4.5258
0,4226
-0.01041
0.4
15
4,4166
0.2678
-0.00447
20
4.3347
0.2485
0002268
0.2
25
4.2777
0,2274
-0.08844
30
4.2408
0.2143
-0.02323
0.0
10
15
20
25
30
history length
Figure 3: Left: Plot of the upper bound on 1- derived from calibrated models. Right: The measure- ments of the upper bound mutual information, the cross entropy of the limited memory model Pr as well as the optimal calibration coefficient a* for various time lengths т. Details the model used here can be found in the supplementary material.
Theorem 5.1. Suppose we have a model Pr(Z(X), and suppose 7 ~ Pr(-IX), where is depen- dent only on X. Suppose that Pr is calibrated to Pr. Then we have that:
< CE(Pr Pr) H(ēY,,X), where:
CE(Pr
Pr)
E
E (-P((-1
log Pr(Z
Memory estimation.
We first learn another We ~ Pr(-IW.-.a-)) and then calibrate Pr Pr.
Corollary 5.2. Suppose -.. (-IWe)) model calibrated Pr(-IW.- For a random
variable, Wa Pr (-1Wz(), we have that:
I(Wall WWW-r/WW-rree CE(Pr Pr) H(W Wer), where:
CE(Pr Pr)
E We Pa
log
Pr(W, W Wi-ra-1)
This corollary gives us means to efficiently provide upper bounds on the mutual information. The directly estimate [E(Pr||Prr through Monte
key is that since Pr is efficiently computable, Carlo estimation. We measure the upper bounds on 1, of a LSTM model with trained limited- memory models Pr (see details in the supplementary material) and report them in Figure 3. As expected, the memory estimate gradually decays with longer т, indicating that the models make more use of the recent past to generate text.
6 CONCLUSION
We have introduced a calibration- based approach detect and provably correct the discrepancies between the long-term generations of language models and the true distributions they estimate se- quentially. particular, for state-of-the-art neural language models, we have observed large degra- dations of the entropy rate under iterative generation, and proposed first-order correction which is both computationally tractable and effective. Using the same calibration approach, we have derived estimators for the amount of information extracted by these models from the deep past.
Aside from the empirical findings and improvements, we hope that this work will inspire a more principled line of discourse the quality of long-term generations in language models. It remains an interesting open problem to relate the plethora of "future-ane aware"" generation- iimprtiiiniiiooiiiinngiii heuristics our calibration framework.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Giuliano Antoniol, Fabio Brugnara, Mauro Cettolo, and Marcello Federico. Language model rep- resentations for beam-search decoding. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 588-591. IEEE, 1995.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.0706. 07086, 2016.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- tems, pages 1171-1179, 2015.
L. D. Brown. Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory. Institute of Mathematical Statistics, Hayworth, CA, USA, 1986. ISBN 0-940- 60010-2.
Philip Clarkson and Tony Robinson. Towards improved language model evaluation measures. In Sixth European Conference on Speech Communication and Technology, 1999.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Interscience, New York, NY, USA, 2006. ISBN 0471241954.
Imre Csiszar and János Körner. Information theory: coding theorems for discrete memoryless sys- tems. Cambridge University Press, 2011.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Language modeling with longer-term dependency. 2018.
A. P. Dawid. The well-calibrated bayesian. Journal of the Am. Stat. Assoc, 77, 1982.
A. Dawid. The impossibility of inductive inference. Journal of the Am. Stat. Assoc, 80, 1985.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_. arXiv preprint arXiv: 100l::/80/077770 2018.
D. P. Foster. Prediction in the worst case. Annals of Statistics, 19, 1991.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tin Liu. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pages 1341-1352, 2018.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with continuous cache. arXiv preprint arXiv:1612.04442 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, pages 1321-1330. JMLR. org, 2017.
Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. Document context language models. arXiv preprint arXiv: 15/ 1.03962 2015.
Uwe Jost and ES Atwell. Proposal for a nuutual-informatie based language model. In Proceedings of the 1994 AISB Workshop on Computational Linguistics for Speech and Handwriting Recogni- fion. AISB, 1994.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the 2016.
limits of language modeling. arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
E. Kalai, E. Lehrer, and R. Smorodinsky. Calibrated forecasting and merging. Games and Economic Behavior, 29, 1999.
Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through remind- ing. In Advances in Neural Information Processing Systems, pages 7651-7662, 2018.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), volume 1, pages 284-294, 2018.
Quoe V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1544.000 2015.
Henry Lin and Max Tegmark. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299, 2017.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming- Ting Sun. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, pages 3155-3165, 2017.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. Agreement on target-bidirectional neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 411- 416, 2016.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X7::997.1119999 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus english: The penn treebank. 1993.
David McAllester. Information theoretic co-training. arXiv preprint arXiv:100..0757., 2018.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and Optimizing LSTM Language Models. arXiv preprint arXiv:1708.02188 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An Analysis of Neural Language Mod- eling at Multiple Scales. arXiv preprint arXiv: 000118003.0444 2018
Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234 239. IEEE, 2012.
Youssef Mrouch and Tom Sercu. Fisher gan. In Guyon, V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 2513 2523. Curran Associates, Inc., 2017. URL http: http:://pppppppppppeeee cc/paper/ 6845 feeefffsshhraaaaa pdf.
Alfred Mller. Integral probability metries and their generating classes of functions. Advances in Applied Probability, 29:429 443, 06 1997. doi: 10.2307/1428011.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd international conference on Machine learning, pages 625-632.
ACM, 2005.
Stefan Ortmanns and Hermann Ney. Look-ahead techniques for fast beam search. Computer Speech & Language, 14(1):15-32, 2000.
Myle Ott, Michael Auli, David Grangier, et al. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3953-3962, 2018.

---

Under review as a conference paper at ICLR 2020
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61-74. MIT Press, 1999.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3--s-wertt amazonaws. com/openai- understanding paper. pdf, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1): 50-64, 1951.
Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schlkopf, and Gert Lanckriet. On integral probability metries, phi-divergences and binary classification. 01 2009.
Volker Steinbiss, Bach-Hiep Tran, and Hermann Ney, Improvements in beam search. In Third International Conference on Spoken Language Processing, 1994.
Shuntaro Takahashi and Kumiko Tanaka-Ishii. Cross entropy of neural language models at infinitya new bound of the entropy rate. Entropy, 20( 1):899, 2018.
Sho Takase, Jun Suzuki, and Masaaki Nagata. Direct output connection for high-rank language model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pages 4599-4609, 2018.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoe V Le. Learning longer-term dependencies in rans with auxiliary losses. arXiv preprint 1803
2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pages 5998 5998-6008, 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017b.
V. Vovk. Competitive on-line statistics. International Statistical Review, 69, 2001.
Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert as a markov random field language model. arXiv preprini arXiv: 040404444, 2019.
Tian Wang and Kyunghyun Cho. Larger-context language modelling with recurrent neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), volume 1, pages 1319 1329,,12,000 2016.
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. arXiv preprint 17Xiv:17/2.09783, 2017.
Sam Wiseman and Alexander M Rush. Sequence-to-sequeee learning as beam-search optimization. arXiv preprint arXiv:160M) 1606.02960 2016
Ziang Xie. Neural text generation: A practical guide. arXiv preprint urXiv: 2017.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates, 2002.
Zhirui Zhang, Shuangzhi Wu. Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. Regularizing neural machine translation by farget-bidirectina. agreement. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 443-450, 2019.
12

---

Under review as a conference paper at ICLR 2020
A
PROOFS FOR SECTION 4
Proof. (of Lemma 4.1) We have
E W1:T~Pr
[f(wir)]-
E
[f(wir)]
Σ
Pr(wi:r) PPr(wxr)) f(11x)
"1
Pr Pr
B
(Pr (|Pr))
where we have used Holder's and Pinsker's inequalities Cover and Thomas (2006).
Proof. (of Corollary 4.2) First observe:
1
log 1 Pr(wi:T)
log (c) < log
log(1 - E) S log
2
8) P)
Pi(wer)
Pr (w1:))
and that:
MT
log
I log
(2)
Pr (WITT)
For the first claim, we have
KL Pr (0))
log Pr(WEr)
I S T )
T WLTTPr
Pr (WIT)
using assumption in Equation 1. For the second claim, taking f = log pp
with Lemma 4.1, we have:
pp (wir)
CE(Pr | Pr ) (at (
T V (KL Pr (e)
1
1
s
log
(e)
Pr
80
MT
T 22 + 1) log
which completes the proof.
Proof: (of Lemma 4.3) By definition,
1
CF(Pr Pra) CR(Pr Pr)
E [s(wir))
Log(Za))
T wirppr
we have:
aCE Praa)
(-wr-(f) + PFco (5)
da The first claim now follows from optimality of a*. For the second claim,
T
@CE((r([PPa)
1 0 log(Za)
pa
T Pa
0 Ewy,
Pr(wir)
T da
Em exp(af((ir))) Pr(wir)
(f)
T Pra
T
By Taylor's theorem, we have:
a² of
CE(Pr][PE.) s CE(Pr - a 7 (
(F;
2 T
Taking the the which minimizes the upper bound, leads the second claim.
13

---

Under review as a conference paper at ICLR 2020
Remark A.1. (Sharpness) If 2 T, then there exists problem where the bound is sharp and EntRate(Pr) takes on the maximal value of Oolo M). As an example, consider model Pr, that starts by generating words under the true distribution Pr and has a + probability of transitioning into a mode in which it generates words uniformly at random thereafter.
Proof. (of Theorem 4.4) We can apply the previous lemma using
log
Pr(wIr))
and so our calibration condition implies:
0 = (Pe(f) MPr... (f) 1---(1 (prr(logPr) Pr) Mpr.. (log Pr))
Now observe that:
T [B(Pr||Pro>) upr( (pr(-(1++ lo) Pr log Za.) (1 x²) (Prr)p Pr) log Za.
and, similarly,
T EntRate(Prar) - a11 MPr.. (log Pr) + log Zo-
These imply:
CF(Pr|[Pi-)) 1 FFacRate(PPP.)) -1 a²) (ur:(log Pr) PP (log Pr))
which completes the proof of the first claim.
The proof of the second claim uses
m(J) (F, = (CE(Pr||Pr) (CCE(PP - EntRate(Pr)) (Pr)
and, by Equation 2,
o Tl + log(1/s),
which completes the proof.
Now move the proof of Corollary 4.5.
Suppose ((Wst) be a function of Wst For a conditional distribution, D(W1.)), let us now define:
AD(s) T
E
E
[f(w))]
Pr i D(.wee)
Define:
(ra(we(We)) = exp(af(wgr)) Pr(welwer)
Za,
and
Pra(war) = (w
Lemma A.I. Suppose
< 0². Let
a = argmin CE(Pr [Pra).
We have that:
pp.(f) - Pp. (f) =0
and that
CF(Pr|[Pia.) < CE(Pr Pr) (F(f) (f))
a²
Proof. (sketch) The proof is identical to that of Lemma 4.3, with the addition of using linearity of expectation.
[
14

---

Under review as a conference paper at ICLR 2020
B PROOFS FOR SECTION
Proof. (of Theorem 5.1) It is convenient to define the distribution:
( Pr(Z)x,)) Pr(Y,
We then have:
-(()) ((
by the defintion of the mutual information.
The proof consists of showing that:
H (( = EVIZDD log D(Z S CE(Pr|[Pr)
Let us take (za(Z = Pr(ZIx,)) (Pr(zix)) /Za. The zero gradient condition for the optimality at a = implies:
OCE(Pr
Pr..)
da
a=0
Ex,Y~Pe
-EZNPECI,Y) log Pr(Z)Y) E (..F(((1.))) log Pr(Z))))
-Eywi-lEZ-Piiin log Ex,y E (E2- log Pr Pr(Z[Y)]
CE(Pr + Ex log Pr(Z
This implies:
CE(Pr|[Pr)
Exy-Pi((22-((ixx) log
Pr(Z())
Ex,Y log
Pr(Z(r)
Ey,z~Dlog Pr(Z|Y)
Ey, log D(Z
H(ZY),
where the last step uses the definition of Ž and Jensen's inequality.
C EXPERIMENTAL DETAILS
In this section, outline the experimental setups used obtain the empirical results throughout the paper. For the calibration and memory experiments (Table row 1, Figure (left), Figures 2, 3), base model is 3-layer L.TT with with 400 embedding dimension and 1 1150 hidden nodes. We train it on the Penn Treebank (PTB) corpus Marcus et al. (1993), following the setup of Merity et al. (2017) and Merity al. (2018) for 500 epochs using SGD with batch size 20 and BPTT length 70. The trained base model achieves 64.3 validation perplexity and 58.3 test perplexity.
The limited-mentt memory models Pr(- ([ used for the memory estimation in Section 5 share the same architecture as our base model while, during training, the hidden states is re-initialized after reading every tokens (r takes value from (5, .0))).
Finally, for the entropy rate measurements of larger-scale state-of-the-art language models (Ta- ble rows 2-4, Figure 1 (right)), icz et al. (2016): Radford et al. (2019) for rows and 4, while we trained the model using the tensor2tensor framework. The model for row 2 is an LSTM with CNN-embedded inputs,
used the pretrained weights published alongside Jozefow-
15

---

Under review as a conference paper at ICLR 2020
trained on the Google Billion Words (GBW) corpus. The other two Transformer Vaswani et al. (2017a) models trained on GBW (row 3), and an proprietary corpus derived from a web crawl (Web- Text; row 4). For GPT-2, since the authors have not published training validation data, used the text of several New York Times articles as a stand-in validation set; the cross entropy loss is comparable that reported the validation set. The entropy rate amplification plot in Figure (bottom) corresponds to the setup from row 4.
To measure the conditional entropy after generations, we measured the empirical conditional en- tropy of the 1-th word over >> 500 independent generations, which were produced by the standard way of iteratively sampling from the next predicted conditional distribution, seeded with ground- truth text up to > 100 random points in the validation set. We used the entropy rate at t = 700 as proxy for the asymptotic limit in Table
D ADDITIONAL GENERATION SAMPLES
In this section, to provide a better sense of the qualitative effect of calibration, we provide below some additional generations, seeded by 10-token prefixes of the holdout (validation) sentences from the Google Billion Words dataset. Here, we used the model we trained for row 3 of Table To identify a failure mode for the uncalibrated model, we selected the seed prefixes which resulted in unusually long generations by the uncalibrated model.
Original model
Calibrated model
Actual results could differ materially from those indicated by these forward-looking state- ments result of numerous factors includ- ing the risks associated with the timely and ef- ficient completion and integration of the Tem- porary Liquidity Guarantee Department 's pervision into the commercial, open market solar energy energy efficiency electric util- ity transmission and water demands of res- idential and business customers. Comcast ability to successfully implement its business plan timing of completion of the acquisition and the effectiveness of the efforts and strate- gies involved in the integration of Rhapsody timing of regulatory and client approvals and availability of key enhancements.
Actual results could differ materially from those indicated by these forward-looking state- ments a result of a variety of factors in- cluding but not limited to (i) the risk that the tender offer could close one or more manner at all (i risks associated with conducting business in foreign jurisdictions ; ( (ii ) difficul- ties in combining some all of the businesses under one roof", ( iv ) decreased demand for electricity natural gas and other energy prod- ucts, including adverse effects on the pricing of oil and natural gas ; and the risks asso- ciated with doing business internationally.
16

---

Under review as a conference paper at ICLR 2020
Actual results could differ materially from those indicated by these forward-looking state- ments as a result of various important factors including without limitation changes in general economic and business conditions. in- cluding more difficult real estate environments declines in information technology spend- ing: continued availability of capital and gov- ernment regulations changes in general eco- nomic and business conditions the possibil- ity that extended unemployment and healthcare policies may change or may reduce access to quality care services failure to obtain ad- equate and affordable medications: changes in certain CME/ CE product mix disruption in CME credit markets: uncertainty of the out- comes of regulatory investigations of compa- nies which the Company has an interest de- pendence on suppliers for most of its products consolidation among financial institutions ability to attract and retain skilled personnel changes in rapidly changing technology and regulatory environments arrogance and com- placency among financial analysts the impact of competition inability retain and moti- vate senior management difficulties in the in- tegration of acquired businesses the effects of redundancy and loss of key employees litiga- tion, including claims and the challenge of in- surance practices uncertainties relating lit- igation: risks related to investigations by other companies inadequate information systems ; the impact of reduced availability of as- sumptions upon such companies using such as ours to gauge CNET 's financial condition and other factors.
Actual results could differ materially from those indicated by such forward-looking state- ments as a result of various important factors including those discussed in the company periodic reports that are filed with the Securi- ties and Exchange Commission and available on the SEC 's website at www.sec.gov.
17

---

Under review as a conference paper at ICLR 2020
Actual results could differ materially from those indicated by such forward-looking state- ments as a result of a variety of factors, in- cluding ability improve Among these factors are changes in the gen- eral economy changes in political and nomic conditions changes in interest rates, changes in technology and implementation of regulatory policies and legislation the direc- tion of interest rates and changes the banking industry changes in loan prepayment activ- ity changes consumer preferences and con- sumer and business lending markets. legisla- tion or public compliance with applicable laws and regulations and changes in the business regulatory environment. We caution you that there are many uncertainties that could cause actual results to differ materially from those indicated in the forward-looking statements Among them are the risk factors that could cause results to differ from those expressed in the forward-looking statements These fac- tors include but are not limited to gen- eral economic and business conditions. includ- ing the financial markets ; fluctuations terest rates government regulation of the fi- nancial services industry and possible failures planning assumptions and estimates poten- tial funding requirements: unexpected changes in increases (including goodwill impair- ment) competition the potentially lengthy. protracted U.S. recession and migratory con- sumer and business conditions.
Actual results could differ materially from those indicated by these forward-looking state- ments as a result of various important factors, including those discussed in the Risk Factors " section of the Company Annual Report on Form 10-K for the most recently ended fiscal year.
liquidity
in-
Bluepoint Games, Inc. is highly experienced and multi-faceted publisher of licensed virtual worlds for gamers, developers and technology professionals The company is based in Van- couver, , Canada BlueKai innovative games distributed by Devices EA, LLC. and Club Penguin. BlueKai owns and is the exclusive li- censor of Scrabulous. Bluetooth Interactive
Bluepoint Games Inc. is highly experienced gaming and entertainment company with sev- eral renowned blockbuster franchises including PC GameHouse ( ((R) GameHouse ( R
). Heavenly Sword ( (TM) ) EverQuest ( R Untold Story (TM) ) and EverQuest ( Through its wholly-owned subsidiary. Blue- hill ID R ), the Bluchill ID logo and tagline are registered trademarks of Bluchill ID Cor- poration and its subsidiaries in the U.S. and in other countries.
Inc. has acquired JoShear-S Swain Media LLC premier developer and publisher of commu- nity based games for the handheld game de- vice For further information please visit www.netgear.com. / ngcleveld Sprint 's fan- tasy game publisher and Web doing business within the Entertainment Group is James Upon CEO of MyNetShelter and the three pre- vious Developers of MySQL. Based in Red- wood City California Bluntann the leader in franchise and game development for the massively multiplayer online game
18

---

Under review as a conference paper at ICLR 2020
Bluepoint Games, Inc. is highly experienced gaming entertainment and mobile games company with a vertically integrated portfolio including: games (TM) , social network, mo- bile, casual games, MMORPG production, distribution and licensing including its flag- ship games, SUIT and TIMMERIX TM ) as well as its award-winning gaming, basketball and entertainment network. In order to create a highly integrated pure and socially respon- sible Game R family Bluepoint has col- laborated with Amplify Systems International Inc. various titles for PlayStation ( R 2 PLAYSTATION ( R ) 5 Wii ( TM) ) 3 PS3 Wii TM ( and PS3 titles as well as PC games for PC PSP, POOL, , Wii ( (TM) ( (and successor title and IP (
Bluepoint Games, Inc. is highly experienced licensing gaming and entertainment firm fo- cused on developing the next generation of casual games based the PlayStation ( R BRAVIA family of video game machines for the North American market Bluepoint is wholly owned subsidiary of Bluehill ID Hold- ings L.P.
(R) ) in addi- tion to its focused gaming entertainment and communication services. BlueBay 's exclu- sive licensee worldwide licensee of the Blue- point TM ) ZMFAO Gateway series it the world 's leading portable gaming, PC and mo- bile phone company For more information, see UNK, Inc. and Oakpoint: ZWC 's Com- munity Health Business Development Center Bluepoint Games Inc. is highly experi- enced, innovative entertainment sports gam- ing company whose products and services are used by some of the most recognized and re- spected names in the world of gaming includ- ing Pokemon Macau ( Valve ) Quat- tro Super Smash Bros. Good Neighbor Games IGN Games, Vail Resorts, Kania ( Ocean Spray Pemberton and Roatenham PURE Holdings. TeenNick, National Amuse- ments, SEGA Games, Cirrus (Aireraft) and
Bluepoint Games Inc. is highly experienced player in the growing genre of casual games for both casual and active gaming enthusiasts Bluepoint is an early stage Company with a sigmificant following among youth and adults in Europe and the United States with an im- pressive track record global on-line gaming opportunities.
www.netapool.com.
Nursing Homes Genworth 's 2009 Cost of Care Survey conducted by the Robert Wood Johnson Foundation and released today re- veals the extent which members of the U.S. population adheres to practices recommended since 1995. including Python for Life that fell asleep from 11 p.m. to a.m. sleep time from 11 p.m. to 3 a.m., spare time from 8 a.m. to 9 p.m. and use of of art non- in-innasive technologies. A remodeling and refurbishment of hospital fa- cilities is underway as the nation 's economy begins to gain momentum. Similar the pre- vious years Thinking About Health Hear how health plans are working to address var- ious congressional proposals to advance best practices in patient care and provide greater accountabilitty advocacy and transparency to consumers.
Nursing Homes Genworth 's 2009 Cost of Care Survey is based on interviews with 516 lamily friends and neighbors of insured and self-employed people conducted from Jan
rolling three-hour
19

---

Under review as a conference paper at ICLR 2020
Nursing Homes Genworth 2009 Cost of Care Survey based on a double-blind ran- domized double-blind placebo-controlled survey which involved an assessment of the cost-effectivennss of healthcare associated with adequate diet and regular physical tivity compared to its managed-care counter- parts The margin of eπrоr for this survey 3.3 percentage points at the 95 percent level of confidence..
Nursing Homes : Genworth 2009 Cost of Care Survey conducted by Harris Interactive performed significantly worse than control group of its peers who provided care but were not able to offer health care to their employees
Nursing Homes Genworth 's 2009 Cost of Care Survey conducted by CareScout ( R ) and published in the April 2009 issue
Nursing Homes Genworth 's 2009 Cost of Care Survey includes a series of health and medical cost reports more than 100 home medical equipment and related products in- cluding more than 3.9 million units of durable medical equipment. IBC 's cost of more than $ 100 billion is a significant portion of Medicare spending on home health care
evaluated findings from the 10-year nearly 900,000-member Specialty Health Manage- ment Association is more than 6,000 profes- sionals living the United States
Table 3: More generations from a state-of-the-art Transformer model trained on GBW, seeded with prefixes of sentences from the holdout validation set.
20


Reviews:
[{"review_id": "r1lY9byg9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a calibration-based approach to measure long range discrepancies between a model distribution and the true distribution in terms of the difference between entropy rate and cross entropy, which is exactly the forward KL divergence. It propose a simple one parameter estimation to improve the model and provides experiments to show the effectiveness.\n\nThis paper in fact provides theoretical justification to the so-called temperature sweep method that is hotly debated in the area of text generation. Several issued should be clearly addressed before the acceptance for publication.\n\n1. The authors should read the Language GANs falling short paper, and conduct the experiments in the same way in that paper and compare their approach with temperature sweep method. The temperature sweep method is only used at inference stage. The proposed approach, Algorithm 2, is used at training stage.\n\n2. The paper provides theoretical results in terms of population distribution, the true but known distribution, however in practice, empirical distribution is used instead, for example the cross entropy in section 3 used in training, are these theoretical results still valid for empirical distribution? If yes, please state in the paper, if not please state why? \n\n3. On page 6, first line, it is stated \"Since this holds for any bounded function, we can obtain the error amplification of the entropy rateof \\hat{Pr} simply by choosing f = − log \\hat{Pr}.\" The log function is unbounded, so please be careful. Fortunately \\hat{Pr}^{\\epsilon} is bounded, so Corollary 4.2 is correct. For the proof of Corollary 4.2, I don't know how to get the inequality in (2) and the first claim, so please provides more steps or explanations.\n\n4. How the entropy rate of each language model in Table 1 is obtained? \n\n5. In section 3, capital letter is used for random variable, but to define H, CR, EntRate, KL, small letter is used, which is not consistent. Also some is used as subscript, some is under E\n\n6. Many unconditional language model papers are not cited, for example, ELMO, BERT, XLNet, Albert et la. and many language GANs paper. On the other hand, many papers for conditional language models are cited,  these papers are not appropriate to cite since the paper targets on unconditional language model.\n\n7. In the first paragraph of page 1, there is a statement of \"Capturing long-termdependencies has especially been a major focus, with approaches ranging from explicit memorybased neural networks (Grave et al., 2016; Ke et al., 2018) to optimization improvements to stabilizelearning (Le et al., 2015; Trinh et al., 2018).\"\n\nIn the second paragraph of page 1, there is a statement of \"Capturing long-term dependencies has especially been a major focus, with approaches ranging from explicit memory-based neural networks (Grave et al., 2016; Ke et al., 2018) to optimizationimprovements aimed at stabilizing training (Le et al., 2015; Trinh et al., 2018).\" \n\nThis is redundant.\n\n8. This paper focuses  on the forward KL divergence, which is related to the quality of language model, but doesn't anything about diversity of language model, which is related to the reverse KL divergence? Can it be extended to the reverse KL divergence?\n\nMissing references:\n\nM. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. Language GANs falling short. In Neural Information Processing Systems Workshop on Critiquing and Correcting Trends in Machine Learning, 2018. \n\nWilliam Fedus, Ian J. Goodfellow, and Andrew M. Dai. MaskGAN: Better text generation via filling in the . ICLR, 2018.\n\nJiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. AAAI, 2018.\n\nFerenc Huszar. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101, 2015.\n\nLantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. AAAI, 2017.\n\nZhongliang Li, Tian Xia, Xingyu Lou, Kaihe Xu, Shaojun Wang, and Jing Xiao. Adversarial discrete sequence generation without explicit neural networks as discriminators. AISTATS, 2019.\n\nKevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. NIPS, 2017.\n\nEhsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diversity and quality in text generation models. NAACL-HLT, 2019.\n\nA Quality-Diversity Controllable GAN for Text Generation\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1ekmgJRqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(emergency review)\n\nThis paper demonstrates that a left-to-right language model suffers a high entropy rate when generating a long-term sequence of words. Then the authors claim that this is because of entropy rate amplification, which could be mitigated by 'calibration'. With local entropy rate calibration, a language model could achieve lower perplexity generating shorter and concise sequences of words.\n\nThe proposed technique (local entropy rate calibration) is straightforward to implement, and empirically shown to be effective. This would be easily applied to the decoder in many seq2seq models, expected to improve various language generation tasks. However, other language models that use bi-directional connections (BERT, RoBERTa, ALBERT) or GAN based language generation models are omitted, and I think these models should be considered to make this work have more impact. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HJlgKVq2YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper highlights and studies the problem of \"entropy rate drift\" for language models: the entropy rate of the language generated by a trained model is much higher than the entropy of ground truth sequences and this discrepancy worsen with the length of generation. The authors interestingly claim that the well-known lack of coherence in long-term model generations is due to this entropy rate drift. Valuably, the entropy rate drift is characterized mathematically. The authors propose a calibration method and prove that their method can interestingly reduce *both* the entropy rate drift and the perplexity of a miscalibrated model, *even though* they assume a rather simplistic model of miscalibration (one that leaks a small amount of mass to all the sequences of a given length). The author quantitatively show that their calibration method reduces the entropy rate drift and qualitatively show that their generations \"make sense\" in the long-term. As an auxiliary result, they show that a similar calibration method can be used to quantify the past information used by the model by upper-bounding the mutual information between the current prediction and the long-term past, given the short-term past, e.g. I(W_t | W<{t-\\tau} | W_{t-\\tau:t-1}).\n\nI really enjoyed reading this paper and was really intrigued by the author's solution. However, in the current form, this paper is below the bar of acceptance due to some weaknesses: (i) rather strong assumption for the main derivation; (ii) lack of clarity and computational complexity of the proposed algorithms; (iii) weak experimental results and missing hint to current models / applications / concurring models. I would be more than happy to increase my score if the authors could kindly respond to the points below.\n\n1) About the assumptions: in your theory, you show that the proposed solution is guaranteed to improve a particular constrained form of miscalibrated model P^\\epsilon, in which a epsilon uniform distribution over all possible sentences of length K is added to the model distribution. This seems a rather strong assumption to me and bumps up to increasing the probability of each word by a small amount for each per-step conditional distribution estimated by the model.\n1.1) Could you elaborate on why intuitively this assumption is something that happens in current models like GPT-2 ? Do you have a way of quantifying whether this assumption reasonably holds ?\n\n2) About complexity:\n2.1) What’s roughly the computational complexity of Algorithm 2 ?\n2.2) Do you need to compute H(W_{t+1}|w_{<= t}) for all possible 1-step continuations w_t ? That seems quite expensive to me in the case of a large vocab. (e.g. sample a word, run forward one step and compute future entropy).\n2.3) If this is correct, how to address this issue ? If I am missing something, it would be good to add to the paper some explicit considerations about the technical feasibility of the algorithm.\n\n3) The amazing thing to me is that you reduce entropy-rate drift (although wrt a particular miscalibrated model, cf. 1) by re-minimizing cross-entropy using the ground-truth corpus. If you could give intuition for this , it would be great and make a much stronger paper !\n3.1) Could you explain why intuitively this works ?\n3.2) If \\alpha is positive, Algorithm 2 is basically penalizing the model for producing words that lead to higher entropy in the future. Why this correlates to less cross-entropy with the data distribution ?\n3.3) Why would calibrating a model that may not conform to the assumption 1) in general prefer to have an \\alpha > 0 ?\n\n4) About the experiments:\n4.1) Where are the perplexities of the calibrated models ?\n4.2) Are the perplexity improving as the theory would suggest ?\n4.3) Is there any small , systematic human study that you could perform apart from just showing few examples of generations ?\n4.4) Could you report some quantitative metrics of the generations from your models and the baselines, for example as computed in previous papers (https://arxiv.org/abs/1801.07736, https://arxiv.org/abs/1908.04319)\n4.5) How do your generations change if you use beam search or arg-max, top-k sampling ? Does your method also help in preventing repetitions ?\n\nMinor:\n\n- I think you are missing a minus in Algorithm 2 inside the exponential.\n\n====\n\nUpdated review:\n\nI thank the authors for their answer. I think that overall this paper provides interesting insight on a hard problem.\nI am raising my score and strongly hope that the authors add the considerations made in their response in the main paper: warning about complexity, optimal \\alphas for each model and details.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows empirically that the state-of-the-art language models have a problem of increasing entropy when generating long sequences. The paper then proposes a method to mitigate this problem. As the authors re-iterated through their rebuttal, this paper approaches this problem theoretically, rather than through a comprehensive set of empirical comparisons.

After discussions among the reviewers, this paper is not recommended to be accepted. Some skepticism and concerns remain as to whether the paper makes sufficiently clear and proven theoretical contributions.

We all appreciate the approach and potential of this paper and encourage the authors to re-submit a revision to a future related venue.
Decision: Reject