Year: 2019

Abstract:
Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibrationbased approach can also be used to measure the amount of memory that language models use for prediction.

heading: 1 INTRODUCTION
text: Recent advances in language modeling have resulted in significant improvements on a wide variety of benchmarks (Dai et al., 2018; Gong et al., 2018; Takase et al., 2018). Capturing long-term dependencies has especially been a major focus, with approaches ranging from explicit memorybased neural networks (Grave et al., 2016; Ke et al., 2018) to optimization improvements to stabilize learning (Le et al., 2015; Trinh et al., 2018). However, while these techniques seem to improve on standard metrics like perplexity and even produce remarkably coherent text (Radford et al., 2019), we still do not have appropriate measures to assess long-term properties in language models, making it difficult to choose between different model options for downstream tasks.
Starting from Shannon’s seminal work that essentially introduced statistical language modeling (Shannon, 1951), the most classical and widely studied long-term property of a language model is its entropy rate — the average amount of information contained per word, conditioned on the preceding words. A learned model provides an upper bound for the entropy rate of a language, via its cross-entropy loss. The exponential of the entropy rate can be interpreted as the effective support size of the distribution of the next word (intuitively, the average number of “plausible” word choices to continue a document), and the perplexity score of a model (the exponential of the cross entropy loss) is an upper bound for this quantity. In state-of-the-art models trained on billion-scale corpora, this number ranges between 10 and 30 (Melis et al., 2017; Radford et al., 2019). A natural diagnostic question, with which we begin our work, is whether the long-term generations of these models exhibit the same entropy rates as the underlying languages they are modeling predictively.
Empirically, and perhaps surprisingly, it turns out that the entropy rate of generated text is substantially higher than the estimate for true text derived from the model’s one-step predictions. As seen in Table 1 (see also Figure 1), this is true for both state-of-the-art LSTMs and Transformers trained on a variety of datasets. As a timely example, the GPT-2 model (Radford et al., 2019), the object of much recent attention for its seemingly coherent and on-topic generations, suffers a dramatic degradation in its entropy rate, from 23.7 to 61.2.
This empirical finding is notable since the neural attention- and memory-based techniques have been steadily improving on standard metrics like perplexity and, in some cases, even produce remarkably coherent text (often with some heuristics to reject poor generations). That the perplexity of generated text is so much higher than it is under the true distribution suggests that there are significant gaps in our current methodologies in accurately learning language models, particularly if we are interested in generating text that globally resembles the modeled language itself.
Our contributions. The focus of this work is twofold: to improve generations based on any measurement mismatch on a long-term property of the model (e.g. the entropy rate), and to quantify the way a model’s predictions depend on the distant past. Central to both of these is a calibrationbased approach, which is utilized in statistics and other areas of machine learning (Dawid, 1982; 1985; Foster, 1991; Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017; Niculescu-Mizil and Caruana, 2005).
First, we show that, from a worst-case perspective, even an extremely accurate model (with ε average KL divergence from the true distribution) may have generated text with a substantially different entropy rate as compared to the true distribution. Indeed, we show that this worst-case amplification may occur for a variety of long-term properties of a probabilistic language model; this is because the one-step KL divergence does not in general provide tight control over the expectation of a bounded function. The observed entropy rate amplification (as seen in Table 1) demonstrates that this is not only of theoretical concern. We then describe a calibration procedure to fix this mismatch while simultaneously improving the perplexity of the language model. From a statistical perspective, the procedure is simple, and we discuss approaches to make it computationally efficient.
Second, we provide a definition for long-term memory in language models as the mutual information between the models predictions and the distant past in the input. We then provide an upper bound on the amount of this mutual information using calibrated distributions (with a single-parameter exponent). This allows us to estimate the amount of context used by a language model as a function of the distance of past tokens from the current prediction time step.
We perform empirical studies to accompany our theoretical results. We first use the entropy rate calibration algorithm to fix an LSTM language model, resulting in a drop of around 20 perplexity points in the generated text (so that the entropy rate of the model more accurately matches that of the language itself). Then, we empirically estimate and compare the long-term memory of state-ofthe-art language models. Our insights point towards new ways of assessing (and fixing) language models, especially in terms of their long-term properties, in a manner complementary to existing metrics like perplexity.

heading: 2 RELATED WORK
text: Improving language modeling with long-term dependencies. Recent approaches to improving language modeling have focused on several ways to better capture long-term dependencies, from using manually-defined context representations (Mikolov and Zweig, 2012; Ji et al., 2015; Wang and Cho, 2016) or document-level topics (Wang et al., 2017) to using LSTM recurrent neural networks with careful initialization (Le et al., 2015), auxiliary loss signals (Trinh et al., 2018) or augmented memory structures (Grave et al., 2016; Ke et al., 2018). Wiseman and Rush (2016) use scoring functions over sequences and search-based optimization to improve generation in seq2seq models.
More recent work has demonstrated the applicability of Transformer networks (Vaswani et al., 2017a) to the task, potentially side-stepping issues in training recurrent networks (e.g. vanishing/exploding gradients) and scaling to longer contexts (Dai et al., 2018; Radford et al., 2018). All these papers propose either architectural or optimization innovations to improve language model
training. In contrast, we define and measure explicit long-term properties of language models and show that calibrating them correctly can provide improvements to any black-box language model.
Recent empirical breakthroughs have stemmed from language models which do not specify a unique autoregressive factorization (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019), and thus do not specify a unique P̂ r. It remains an interesting problem to identify and sample from distributions induced by these models (Wang and Cho, 2019); thus, our end-to-end theoretical guarantees do not hold in this setting.
Information-theoretic approaches. While most language models aim to predict a distribution over the next token conditioned on the context, there have been alternative approaches relying on information-theoretic measures. Jost and Atwell (1994) propose a model which makes use of mutual information between word pairs to generate word sequences that retain longer-term dependencies. McAllester (2018) propose a training objective based on mutual information for predictive modeling, and demonstrate its application for phoneme prediction. Clarkson and Robinson (1999) develop a hybrid metric using both perplexity and entropy rate, and show that it correlates better with a downstream metric like word error rate. Such works propose alternative optimization objectives; in contrast, we show how to use information-theoretic measures to improve models with respect to existing objectives like cross-entropy.
Measuring long-term statistics. Khandelwal et al. (2018) analyze LSTM-based language models and empirically show that such models make use of a finite context for prediction. Lin and Tegmark (2017) measure mutual information between any two symbols in human languages, and show that it decays with distance, roughly following a power law distribution. Takahashi and Tanaka-Ishii (2018) provide an upper bound for the entropy (character-level) of human languages by training neural language models with various context and data sizes and extrapolating to infinity. While we also make use of measures like entropy and mutual information across longer contexts, our goal is to use these to better calibrate the language model and provably improve its perplexity.
Calibration and integral probability metrics. The idea of matching properties of the models’ predictions to the empirical outcomes, in an online setting, goes back (at least) to the “prequential principle” of Dawid (1982; 1985), with subsequent work in online and game-theoretic settings (Foster, 1991; Vovk, 2001; Kalai et al., 1999). The idea of improving probability scores is also common in machine learning (Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017; Niculescu-Mizil and Caruana, 2005). Recently, Ott et al. (2018) assessed model calibration for machine translation systems using word-level probabilities. The notion of examining the expectation of functions as a metric for the distance between two distributions sometimes goes under the name of integral probability metrics (Mller, 1997; Sriperumbudur et al., 2009), and this notion is becoming increasingly relevant again in unsupervised learning through the connections to GANs (Mroueh and Sercu, 2017). In this work, we directly focus on the KL divergence, where our use of calibration is largely based on basic facts about exponential families (Brown, 1986).
Relation to generation-improving heuristics. If the sole objective is to improve the qualitative coherency of sampled generations, a wide variety of heuristics exist in the literature. The simplest of these is a constant multiplicative adjustment to the model’s logits (known as softmax temperature Xie (2017)). This is a specific version of our method (Algorithm 2) with a constant logistic regression feature instead of the next-token conditional entropy. Relatedly, greedy and top-k sampling (used in state-of-the-art works such as (Radford et al., 2019)) are heuristics which make local modifications to the model’s conditional probabilities to decrease diversity and eliminate nonsensical generations.
Efforts to push the empirical state of the art in generation quality have given rise to more complex heuristics. Bengio et al. (2015) propose retraining the network on its own generations with a carefully scheduled probability for each token. Some works regularize a model’s generations with an auxiliary reverse language model Zhang et al. (2019); Liu et al. (2016). Yet others promote realism using adversarial training protocols (Bahdanau et al., 2016; Lin et al., 2017; Fedus et al., 2018).
We stress that our calibration methods result in a provable improvement in the original training objective (i.e. lower perplexity). As far as we know, none of the aforementioned heuristic methods can hope to provide such a strong guarantee, since they are fundamentally designed to bias
models towards a different objective. Our work mitigates model hallucinations for (almost) free1, in the sense that the global objective (entropy rate drift) is improved without worsening the local objective (perplexity). Furthermore, calibration preserves the computational efficiency of density estimation: a conditional probability vector from Algorithm 2 can be computed using O(vocabulary size) inferences on the original model. In the more advanced heuristics, the implied distribution over sequences is lost, and is only accessible by black-box sampling.

heading: 3 PRELIMINARIES
text: We first define some useful quantities for our analyses. Let Pr(W1,W2, . . . ,WT ) represent the true underlying distribution over T length sequences of words, where the vocabulary is of size M . Let W1:T denote a random sequence of length T , with distribution Pr(W1:T ). For clarity of exposition, we assume that all sequences (i.e. sentences or documents or books) are of equal length T .
For any distributions D and D′ over length-T sequences, recall that the entropy H(·), KLdivergence, and entropy rate are, respectively, defined by: H(D) := Ew1:T∼D [ log 1D(W1:T=w1:T ) ] ,
KL(D ‖ D′) := Ew1:T∼D [ log D(W1:T=w1:T )D′(W1:T=w1:T ) ] , and EntRate(D) := 1TH(D). Let P̂r(W1:T ) denote a learned distribution over sequences. In the typical sequential prediction setting, the probabilistic model is implicitly defined by the conditional distributions Pr(Wt|W<t), which are typically efficiently computable. It is standard for such a language model to be trained to minimize the cross entropy objective:
CE(Pr ‖ P̂r) := 1 T E w1:T∼Pr [ T∑ t=1 log 1 P̂r(wt|w<t) ] = 1 T E w1:T∼Pr [ log
1
P̂r(w1:T )
] .
Note that for an accurate language model, we would hope that: CE(Pr ‖ P̂r) ≈ EntRate(P̂r), i.e. the entropy rate of the sequences generated under the learned model is nearly that of the cross entropy of the model (with respect to the true distribution Pr).
Throughout, we assume that
1 T KL(Pr ‖ P̂r) = CE(Pr ‖ P̂r)− EntRate(Pr) ≤ ε (1)
holds for some ε. In other words, the (unknown) ε measures the degree of sub-optimality of the learned model, this ε is often referred to as the Bayes regret.

heading: 4 CALIBRATION AND ENTROPY RATES
text: In this section, we assess the long-term properties of language models when generating text. Specifically, we quantify the amplification in the entropy rate of generations under an ε-accurate model (Eq. 1). We then provide a procedure to fix this amplification, without increasing the perplexity of the model. Proofs for all statements are provided in the supplementary material.
For generality, consider a function f : [M ]T → R, defined on T length sequences. Let the mean and variance of f under distribution D be denoted by µD(f) and σ2D(f)
µD(f) := E w1:T∼D [f(w1:T )], σ 2 D(f) := E w1:T∼D [(f(w1:T )− µD(f))2] .

heading: 4.1 ERROR AMPLIFICATION UNDER OUR MODEL
text: If our learned model P̂r is accurate, we may hope that µPr(f) ≈ µP̂r(f) i.e. that the expected value of f under the true distribution Pr is close to its expected value under our model. We can quantify this gap as follows:
1Technically, at the statistical cost of fitting one more parameter.
Lemma 4.1. (Pinsker’s Inequality (Csiszar and Körner, 2011)) Suppose that for all w1:T , f(w1:T ) ≤ B. Then: ∣∣µPr(f)− µP̂r(f)∣∣ ≤ B√2KL(Pr ‖ P̂r) . Since this holds for any bounded function, we can obtain the error amplification of the entropy rate of P̂r simply by choosing f = − log P̂r. Before we proceed, in order to rule out amplification of this entropy rate due to arbitrarily small probabilities (which can blow up − log P̂r), it is helpful to define the γ-mixture distribution as: D(γ) := (1 − γ)D + γUni, where the Uni is the uniform distribution over all MT sequences. We will then consider the model P̂r (ε) , which has only a minor degradation in the cross entropy
compared to P̂r, and, yet, may have a large amplification in the entropy rate. Corollary 4.2. (Entropy rate amplification under generations) Suppose the bound in equation 1 holds. The ε-mixture distribution has KL bounded as:
1 T KL(Pr ‖ P̂r (ε) ) ≤
( 1 + 2
T
) ε .
We have that:
|CE(Pr ‖ P̂r (ε) )− EntRate(Pr)| ≤ ( 1 + 2
T
) ε , and
|CE(Pr ‖ P̂r (ε) )− EntRate(P̂r (ε) )| ≤ √ 2ε(T + 1) ( logM + log(1/ε)
T
) .
This bound shows that, in the worst case, even a small cross entropy may provide little control over the generations under our model (in terms of entropy rate). In fact, for ε = O( 1T ) (which we may hope is an accurate model), the bound is vacuous; a later remark shows this worst case bound is unimprovable, see the supplementary material.
The above theorems suggest that entropy rate amplification is a theoretical possibility in the worst case, which our experiments show is in fact prevalent in pratice. These entropy rate amplifications are evident from the plots in Figure 1. Regardless of the text corpus or the language model, we observe that the entropy rate under the model’s generations quickly increases with time, indicating that this is a persistent problem even for state-of-the-art language models while generating text.

heading: 4.2 MODEL CALIBRATION
text: We now describe a procedure to fix this error amplification. First, let us define a distribution P̂rα such that:
P̂rα(w1:T ) = exp(αf(w1:T )) · P̂r(w1:T )
Zα where Zα = ∑ w1:T exp(αf(w1:T )) · P̂r(w1:T ) .
We can then recover a calibrated model that does not suffer from error amplification in f :
Algorithm 1 (Inefficient) Entropy Rate Calibration
1: Input: Model P̂r (ε)
. 2: Define a model class:
P̂rα(w1:T ) = ( P̂r(w1:T ) (ε) )1+α /Zα .
3: Fit α∗: α∗ = argminα CE(Pr ‖ P̂rα) 4: Return P̂rα∗
Lemma 4.3. (Calibration to f with model improvement) Suppose the variance of f is uniformly bounded in that there exists σ2+ such that the following holds for all α, σ 2 Prα
(f) ≤ σ2+ . Let α∗ = argminα CE(Pr ‖ P̂rα) . We have
µPr(f)− µP̂rα∗ (f) = 0, and CE(Pr ‖ P̂rα∗) ≤ CE(Pr ‖ P̂r)− 1
T
(µ(f)− µ P̂r (f))2
2σ2+ .
Entropy rate calibration. We can now apply the previous result to fix the entropy rate amplification seen in Table 1. Note that it is trivial to avoid the entropy rate amplification if we were allowed to degrade the quality of our model, in terms of perplexity (e.g. a unigram model does not have this amplification. However, we show that it is possible to match the entropy rate without having to sacrifice the quality of our model. In fact, we can both improve our model and more accurately match the entropy rate, by fitting a family of one-parameter models.
Theorem 4.4. (Entropy rate calibration) Suppose equation 1 holds. Algorithm 1 returns a P̂rα∗ such that: the following calibration property is satisfied:
CE(Pr ‖ P̂rα∗) = EntRate(P̂rα∗).
Furthermore, P̂rα∗ has entropy close to the true entropy rate as specified by: |EntRate(Pr)− EntRate(P̂rα∗)| ≤ ( 1 + 1
T
) ε,
and P̂rα∗ is an improvement over the original model as characterized by:
CE(Pr ‖ P̂rα∗) ≤ CE(Pr ‖ P̂r (ε) )− 1 2 CE(Pr ‖ P̂r(ε))− EntRate(P̂r) logM + log(1/ε)T 2 . This result shows that we simply need a single parameter α to define a new model class that is a powered up version of our original model. Then, we can fit this α to minimize the cross-entropy of the new model with respect to the true distribution Pr, in order to eliminate the entropy rate amplification.
Even though this algorithm fits only a single parameter, it is not easily implementable since it requires an integration over sequences, at least in its exact form. One future direction would be to a sample based approach. This may be an interesting alternative to ideas like beam search (Steinbiss et al., 1994; Ortmanns and Ney, 2000; Antoniol et al., 1995), which also aims to minimize a global cost function on sequences that is inconsistent with the token-level perplexity loss used to train the underlying generative model.
Lookahead algorithms. In order to sidestep the computational issues of Algorithm 1, we provide another simple approach based on what can be viewed as a “one-step” lookahead correction (Algorithm 2). Let Ŵt be a random variable with conditional distribution P̂r(·|W<t). H(Ŵt+1|w≤t) denotes the entropy of this conditional distribution, i.e.
H(Ŵt+1|w≤t) = E wt+1∼P̂r(·|w≤t)
[ log
1
P̂r(wt+1|w≤t)
] .
Algorithm 2 Local Entropy Rate Calibration
1: Input: Model P̂r (ε) , where Ŵt ∼ P̂r (ε)
(·|W<t). 2: Define a model class:
P̂rα(w1:T ) = P̂α(w1)P̂α(w2|w1) . . . where
P̂rα(wt|w<t) = P̂r(wt|w<t) · exp ( −α ·H(Ŵt+1|w≤t) ) /Zα.
3: Fit α∗: α∗ = argminα CE(Pr ‖ P̂rα) 4: Return P̂rα∗
Note that H(Ŵt+1|w≤t) includes the word wt, so we require computing the entropy at time t + 1 when predicting Wt using a learned model.
For a conditional distribution, D(W1:T ), let us define:
µ̄D = 1
T T∑ t=1 E w<t∼Pr E wt∼D(·|w<t) [H(Ŵt+1|w≤t)]
Thus, µ̄D is the average of H(Ŵt+1|w≤t) with respect to a distribution which uses D for sampling the last word Wt (at every timestep). Intuitively, the resulting model P̂rα with a positive α would suppress sampling words leading to larger entropy but rather encourage words that stablizes the entropy 1-step ahead in the future. Therefore, if our learned language model P̂r was accurate, we would hope that: µ̄Pr ≈ µ̄P̂r . The following corollary shows that this is achievable, along with improving the model’s perplexity.
Corollary 4.5. Suppose Equation 1 holds. Then, Algorithm 2 returns a P̂rα∗ such that:
µ̄Pr − µ̄P̂rα∗ = 0, and CE(Pr ‖ P̂rα∗) ≤ CE(Pr ‖ P̂r (ε) )− 1 2
( µ̄− µ̄
P̂r (ε)
logM + log(1/ε)T
)2 .
This result provides us with Algorithm 2, which is computationally quite tractable. We first use the learned model P̂r to define a new model class P̂rα, which scales P̂r by an exponential distribution over the weighted 1-step lookahead entropy H(Ŵt+1|w≤t). Then, similar to Algorithm 1, we simply fit the single parameter α to minimize the cross-entropy of the new model with respect to Pr, which fixes the entropy amplification in the resulting model P̂rα. We observe this empirically in Figure 2 – our calibration results in a perplexity drop of almost 20 points over long-term generations under an LSTM model. Model and implementation details are in the supplementary material.
Generations from a calibrated model. Table 2 provides sample generations from a cali-
brated Transformer model trained on the GBW dataset, compared to its original version. Qualitatively, the calibrated generations: (1) are shorter and more concise, and (2) display a better grasp of discourse structure across sentences. More generations are provided in the supplementary material.

heading: 5 CALIBRATION AND MEMORY
text: Defining a notion of memory in language models is challenging, and multiple equally sensible notions may co-exist. Here we present our choice from first principles. Let us say that Ŵt is a sample
from a model at time t, i.e. Ŵt ∼ P̂r(Wt|W<t). Let us also assume that W<t ∼ Pr(W<t). We will define the memory at gap τ as the mutual information between Ŵt and the distant past (those words greater than τ steps ago) conditioned on the subsequence Wt−τ :t−1. Precisely,
Iτ := I(Ŵt;W<t−τ |Wt−τ :t−1) = H(Ŵt|Wt−τ :t−1)−H(Ŵt|W<t) ,
where we are not explicitly denoting the t dependence in this definition2.
Intuitively, It can be viewed as how much uncertainty (entropy) in the prediction Wt the model is able to reduce by utilizing the deep past W<t−τ in addition to the recent past Wt−τ :t−1.
The difficulty in estimating this mutual information is due to estimating H(Ŵt|Wt−τ :t−1), which requires the marginalized model P̂r(Wt|Wt−τ :t−1). To (even approximately) marginalize a model distribution P̂r(Wt|W<t) over the deep past W<t−τ is statistically difficult, since it requires the access to a pool of samples of W<t that share an common recent past Wt−τ :t−1. Nevertheless, we now show that it is possible to obtain an upper bound (which is computationally efficient to estimate).
Upper bounding mutual information using calibrated models. In the above, we were considering the mutual information between Ŵt and W<t−τ conditioned on Wt−τ :t−1. Let us now consider a more general setting, where we have a distribution Pr(Z, Y,X) where Z, Y , and X are random variables. We wil eventually consider Z, Y,X to be Ŵt,Wt−τ :t−1W<t−τ , respectively.
For distributions D(·|Y,X) and D̃(·|Y,X) and for α ∈ R, define Dα(Z|Y,X) := D(Z|Y,X) · ( D̃(Z|Y,X) )α /Zα .
We say that D(·|Y,X) is calibrated to D̃(·|Y,X), if D = Dα=0 is unimprovable in that for all α
CE(Pr ‖ D) ≤ CE(Pr ‖ Dα) .
Note this condition is achievable due to that calibrating a model to D̃(·|Y,X) involves a one dimensional (convex) estimation problem (over α).
2While we may attempt to estimate Iτ for a given t, we can remove the t dependence by either defining this quantity by with an average over t or by using appropriate stationarity assumptions. In our experiments, we average over t.
Theorem 5.1. Suppose we have a model P̂r(Z|X), and suppose Z̃ ∼ P̃r(·|X), where Z̃ is dependent only on X . Suppose that P̂r is calibrated to P̃r. Then we have that:
I(Ẑ;X|Y ) ≤ CE(Pr ‖ P̃r)−H(Ẑ|Y,X) , where:
CE(Pr ‖ P̃r) = E Y∼Pr E Z∼Pr(·|Y )
[ log
1
P̃r(Z|Y )
] .
Memory estimation. We first learn another W̃t ∼ P̃r(·|Wt−τ :t−1), and then calibrate P̂r to P̃r.
Corollary 5.2. Suppose P̂r cal
(·|W<t) is a model calibrated to P̃r(·|Wt−τ :t−1). For a random variable, Ŵ calt ∼ P̂r cal (·|W<t), we have that:
I(Ŵ calt ;W<t−τ |Wt−τ :t−1) ≤ CE(Pr ‖ P̃r)−H(Ŵ calt |W<t), where:
CE(Pr ‖ P̃r) = E Wt−τ:t∼Pr
[ log
1
P̃r(Wt|Wt−τ :t−1)
] .
This corollary gives us a means to efficiently provide upper bounds on the mutual information. The key is that since P̃r is efficiently computable, we can directly estimate CE(Pr ||P̃r) through Monte Carlo estimation. We measure the upper bounds on Iτ of a LSTM model with trained limitedmemory models P̃r (see details in the supplementary material) and report them in Figure 3. As expected, the memory estimate gradually decays with longer τ , indicating that the models make more use of the recent past to generate text.

heading: 6 CONCLUSION
text: We have introduced a calibration-based approach to detect and provably correct the discrepancies between the long-term generations of language models and the true distributions they estimate sequentially. In particular, for state-of-the-art neural language models, we have observed large degradations of the entropy rate under iterative generation, and a proposed first-order correction which is both computationally tractable and effective. Using the same calibration approach, we have derived estimators for the amount of information extracted by these models from the deep past.
Aside from the empirical findings and improvements, we hope that this work will inspire a more principled line of discourse on the quality of long-term generations in language models. It remains an interesting open problem to relate the plethora of “future-aware” generation-improving heuristics to our calibration framework.

heading: A PROOFS FOR SECTION 4
text: Proof. (of Lemma 4.1) We have∣∣∣∣ Ew1:T∼Pr[f(w1:T )]− Ew1:T∼P̂r[f(w1:T )] ∣∣∣∣ = ∣∣∣∣∣∑ w1:T ( Pr(w1:T )− P̂r(w1:T ) ) f(w1:T ) ∣∣∣∣∣ ≤ ∣∣∣Pr−P̂r∣∣∣ 1 B
≤ √
2KL(Pr ||P̂r))B where we have used Holder’s and Pinsker’s inequalities Cover and Thomas (2006).
Proof. (of Corollary 4.2) First observe:
log 1
P̂r (ε) (w1:T ) ≤ log 1 (1− ε)P̂r(w1:T ) = log
1 P̂r(w1:T ) − log(1− ε) ≤ log 1 P̂r(w1:T ) + 2ε
and that:
log 1
P̂r (ε) (w1:T ) ≤ log M
T
ε . (2)
For the first claim, we have
1 T KL(Pr ||P̂r (ε) ) = 1 T E w1:T∼Pr log Pr(w1:T ) P̂r (ε) (w1:T )  ≤ (1 + 2 T )ε .
using our assumption in Equation 1.
For the second claim, taking f = log 1 P̂r (ε) (w1:T ) with Lemma 4.1, we have:∣∣∣∣CE(Pr ||P̂r(ε))− EntRate(P̂r(ε))∣∣∣∣ ≤ 1T √ 2KL(Pr ||P̂r (ε) ) ∣∣∣∣∣log 1 P̂r (ε) ∣∣∣∣∣ ∞
= 1
T
√ 2ε(T + 1) log MT
ε ,
which completes the proof.
Proof. (of Lemma 4.3) By definition,
CE(Pr ||P̂rα) := CE(Pr ||P̂r)− α
T E w1:T∼Pr [f(w1:T )] +
1 T log(Zα) ,
we have: ∂CE(Pr ||P̂rα)
∂α =
1
T
( −µPr(f) + µP̂rα(f) ) The first claim now follows from optimality of α∗.
For the second claim,
∂2CE(Pr ||P̂rα) ∂2α = 1 T ∂2 log(Zα) ∂2α
= 1
T
∂
∂α
∑ w1:T
f(w1:T ) exp(αf(w1:T )) · P̂r(w1:T )∑ w1:T exp(αf(w1:T )) · P̂r(w1:T )
= 1
T σ2 P̂rα (f) ≤ σ2+ T .
By Taylor’s theorem, we have:
CE(Pr ||P̂rα) ≤ CE(Pr ||P̂r)− α · 1
T
( µPr(f)− µP̂r(f) ) + α2 2 · σ2+ T .
Taking the the α which minimizes the upper bound, leads to the second claim.
Remark A.1. (Sharpness) If ε ≥ 1T , then there exists a problem where the bound is sharp and EntRate(P̂r) takes on the maximal value of O(logM). As an example, consider a model P̂r, that starts by generating words under the true distribution Pr and has a 1T probability of transitioning into a mode in which it generates words uniformly at random thereafter.
Proof. (of Theorem 4.4) We can apply the previous lemma using
f = log 1
P̂r(w1:T ) ,
and so our calibration condition implies: 0 = µPr(f)− µP̂rα∗ (f) = − ( µPr(log P̂r)− µP̂rα∗ (log P̂r) ) .
Now observe that:
T · CE(Pr ||P̂rα∗) = µPr(−(1 + α∗) log P̂r + logZα∗) = −(1 + α∗)µPr(log P̂r) + logZα∗
and, similarly, T · EntRate(P̂rα∗) = −(1 + α∗)µP̂rα∗ (log P̂r) + logZα∗ .
These imply:
CE(Pr ||P̂rα∗)− EntRate(P̂rα∗) = − 1
T (1 + α∗) ( µPr(log P̂r)− µP̂rα∗ (log P̂r) ) = 0 ,
which completes the proof of the first claim.
The proof of the second claim uses
µ(f)− µ P̂r
(f) = T ( CE(Pr ||P̂r)− EntRate(P̂r) ) ,
and, by Equation 2, σ2+ ≤ T logM + log(1/ε) ,
which completes the proof.
Now we move on to the proof of Corollary 4.5.
Suppose f(W≤t) be a function of W≤t. For a conditional distribution, D(W1:T ), let us now define:
µ̄D(f) = 1
T T∑ t=1 E w<t∼Pr E wt∼D̂(·|w<t) [f(w≤t)] .
Define: P̂t,α(wt|w<t) := 1
Zα,t exp(αf(w≤t)) · P̂r(wt|w<t)
and P̂rα(w1:T ) := P̂1,α(w1)P̂2,α(w2|w1) . . . .
Lemma A.1. Suppose f ≤ σ2+. Let
α∗ = argmin α CE(Pr ||P̂rα) .
We have that: µ̄Pr(f)− µ̄P̂rα∗ (f) = 0 and that
CE(Pr ||P̂rα∗) ≤ CE(Pr ||P̂r)− (µ̄(f)− µ̄ P̂r (f))2
σ2∗ .
Proof. (sketch) The proof is identical to that of Lemma 4.3, with the addition of using linearity of expectation.

heading: B PROOFS FOR SECTION 5
text: Proof. (of Theorem 5.1) It is convenient to define the distribution:
D(Z, Y,X) = P̂r(Z|X,Y ) · Pr(Y,X) .
We then have: I(Ẑ;X|Y ) = H(Ẑ|Y )−H(Ẑ|Y,X)
by the defintion of the mutual information.
The proof consists of showing that:
H(Ẑ|Y ) = EY,Z∼D log 1
D(Z|Y ) ≤ CE(Pr ||P̃r) .
Let us take P̂rα(Z|X,Y ) = P̂r(Z|X,Y ) · ( P̃r(Z|X) )α
/Zα. The zero gradient condition for the optimality at α = 0 implies:
0 = ∂CE(Pr ||P̂rα)
∂α
∣∣∣ α=0
= EX,Y∼Pr [ −EZ∼Pr(·|X,Y ) log P̃r(Z|Y ) + EZ∼P̂r(·|X,Y ) log P̃r(Z|Y ) ] = −EY∼Pr[EZ∼Pr(·|Y ) log P̃r(Z|Y ) + EX,Y∼Pr[EZ∼P̂r(·|X,Y ) log P̃r(Z|Y )]
= CE(Pr ||P̃r) + EX,Y∼Pr[EZ∼P̂r(·|X,Y ) log P̃r(Z|Y )] .
This implies:
CE(Pr ||P̃r) = EX,Y∼Pr[EZ∼P̂r(·|X,Y ) log 1
P̃r(Z|Y ) ]
= EX,Y,Z∼D log 1
P̃r(Z|Y )
= EY,Z∼D log 1
P̃r(Z|Y )
≥ EY,Z∼D log 1 D(Z|Y ) = H(Ẑ|Y ) ,
where the last step uses the definition of Ẑ and Jensen’s inequality.

heading: C EXPERIMENTAL DETAILS
text: In this section, we outline the experimental setups used to obtain the empirical results throughout the paper. For the calibration and memory experiments (Table 1 row 1, Figure 1 (left), Figures 2, 3), our base model is a 3-layer LSTM with with 400 embedding dimension and 1150 hidden nodes. We train it on the Penn Treebank (PTB) corpus Marcus et al. (1993), following the setup of Merity et al. (2017) and Merity et al. (2018) for 500 epochs using SGD with batch size 20 and BPTT length 70. The trained base model achieves 64.3 validation perplexity and 58.3 test perplexity.
The limited-memory models P̃r(·|Wt−τ :t−1) used for the memory estimation in Section 5 share the same architecture as our base model while, during training, the hidden states is re-initialized after reading every τ tokens (τ takes value from {5, 15, . . . , 30}). Finally, for the entropy rate measurements of larger-scale state-of-the-art language models (Table 1 rows 2-4, Figure 1 (right)), we used the pretrained weights published alongside Jozefowicz et al. (2016); Radford et al. (2019) for rows 2 and 4, while we trained the model using the tensor2tensor framework. The model for row 2 is an LSTM with CNN-embedded inputs,
trained on the Google Billion Words (GBW) corpus. The other two are Transformer Vaswani et al. (2017a) models trained on GBW (row 3), and an proprietary corpus derived from a web crawl (WebText; row 4). For GPT-2, since the authors have not published training or validation data, we used the text of several New York Times articles as a stand-in validation set; the cross entropy loss is comparable to that reported on the validation set. The entropy rate amplification plot in Figure 1 (bottom) corresponds to the setup from row 4.
To measure the conditional entropy after t generations, we measured the empirical conditional entropy of the t-th word over > 500 independent generations, which were produced by the standard way of iteratively sampling from the next predicted conditional distribution, seeded with groundtruth text up to > 100 random points in the validation set. We used the entropy rate at t = 700 as a proxy for the asymptotic limit in Table 1.

heading: D ADDITIONAL GENERATION SAMPLES
text: In this section, to provide a better sense of the qualitative effect of calibration, we provide below some additional generations, seeded by 10-token prefixes of the holdout (validation) sentences from the Google Billion Words dataset. Here, we used the model we trained for row 3 of Table 1. To identify a failure mode for the uncalibrated model, we selected the seed prefixes which resulted in unusually long generations by the uncalibrated model.
Original model Calibrated model Actual results could differ materially from those indicated by these forward-looking statements as a result of numerous factors including the risks associated with the timely and efficient completion and integration of the Temporary Liquidity Guarantee Department ’s supervision into the commercial , open market , solar energy , energy efficiency , electric utility transmission , and water demands of residential and business customers , Comcast ’s ability to successfully implement its business plan , timing of completion of the acquisition and the effectiveness of the efforts and strategies involved in the integration of Rhapsody , timing of regulatory and client approvals and availability of key enhancements . Actual results could differ materially from those indicated by these forward-looking statements as a result of a variety of factors , including but not limited to ( i ) the risk that the tender offer could close in one or more manner or at all ; ( ii ) risks associated with conducting business in foreign jurisdictions ; ( iii ) difficulties in combining some or all of the businesses under one roof ; ( iv ) decreased demand for electricity , natural gas and other energy products , including adverse effects on the pricing of oil and natural gas ; and ( v ) the risks associated with doing business internationally .
Actual results could differ materially from those indicated by these forward-looking statements as a result of various important factors , including , without limitation : changes in general economic and business conditions , including more difficult real estate environments ; declines in information technology spending ; continued availability of capital and government regulations ; changes in general economic and business conditions ; the possibility that extended unemployment and healthcare policies may change , or may reduce access to quality care services ; failure to obtain adequate and affordable medications ; changes in certain CME / CE product mix ; disruption in CME credit markets ; uncertainty of the outcomes of regulatory investigations of companies in which the Company has an interest ; dependence on suppliers for most of its products ; consolidation among financial institutions ; ability to attract and retain skilled personnel ; changes in rapidly changing technology and regulatory environments ; arrogance and complacency among financial analysts ; the impact of competition ; inability to retain and motivate senior management ; difficulties in the integration of acquired businesses ; the effects of redundancy and loss of key employees ; litigation , including claims and the challenge of insurance practices ; uncertainties relating to litigation ; risks related to investigations by other companies ; inadequate information systems ; the impact of reduced availability of ; * assumptions upon such companies using such as ours to gauge CNET ’s financial condition ; and other factors . Actual results could differ materially from those indicated by such forward-looking statements as a result of various important factors , including those discussed in the company ’s periodic reports that are filed with the Securities and Exchange Commission and available on the SEC ’s website at www.sec.gov.
Actual results could differ materially from those indicated by such forward-looking statements as a result of a variety of factors , including our ability to improve our liquidity . Among these factors are changes in the general economy , changes in political and economic conditions , changes in interest rates , changes in technology and implementation of regulatory policies and legislation , the direction of interest rates and changes in the banking industry , changes in loan prepayment activity , changes in consumer preferences and consumer and business lending markets , legislation or public compliance with applicable laws and regulations and changes in the business or regulatory environment . We caution you that there are many uncertainties that could cause actual results to differ materially from those indicated in the forward-looking statements . Among them are the risk factors that could cause results to differ from those expressed in the forward-looking statements . These factors include , but are not limited to : general economic and business conditions , including the financial markets ; fluctuations in interest rates ; government regulation of the financial services industry and possible failures ; planning assumptions and estimates ; potential funding requirements ; unexpected changes in cost increases ( including goodwill impairment ) ; competition ; the potentially lengthy , protracted U.S. recession ; and migratory consumer and business conditions . Actual results could differ materially from those indicated by these forward-looking statements as a result of various important factors , including those discussed in the ” Risk Factors ” section of the Company ’s Annual Report on Form 10-K for the most recently ended fiscal year . Bluepoint Games , Inc. is a highly experienced and multi-faceted publisher of licensed virtual worlds for gamers , developers and technology professionals . The company is based in Vancouver , Canada . BlueKai ’s innovative games are distributed by Devices EA , LLC , and Club Penguin . BlueKai owns and is the exclusive licensor of Scrabulous . BluetoothQ Interactive Inc. has acquired JoShear-Swain Media , LLC , a premier developer and publisher of community based games for the handheld game device . For further information , please visit : www.netgear.com / ngcleveld . Sprint ’s fantasy game publisher and Web doing business within the Entertainment Group is James Upon , CEO of MyNetSheltetWeb and the three previous Developers of MySQL . Based in Redwood City , California , BlueMountain is the leader in franchise and game development for the massively multiplayer online game . Bluepoint Games , Inc. is a highly experienced gaming and entertainment company with several renowned blockbuster franchises including PC , GameHouse ( ( R ) ) GameHouse ( ( R ) ) , Heavenly Sword ( ( TM ) ) , EverQuest ( R ) , Untold Story ( TM ) and EverQuest ( R ) II . Through its wholly-owned subsidiary , Bluehill ID ( R ) , the Bluehill ID logo and tagline are registered trademarks of Bluehill ID Corporation and its subsidiaries in the U.S. and in other countries .
Bluepoint Games , Inc. is a highly experienced gaming , entertainment and mobile games company with a vertically integrated portfolio including : games ( TM ) , social network , mobile , casual games , MMORPG , production , distribution , and licensing including its flagship games , SUIT and TIMMERIX ( TM ) , as well as its award-winning gaming , basketball and entertainment network . In order to create a highly integrated , pure and socially responsible Game ( R ) family , Bluepoint has collaborated with Amplify Systems International , Inc. on various titles for PlayStation ( R ) 2 , PLAYSTATION 3 ( R ) 5 , Wii ( TM ) 3 , PS3 , Wii ( TM ) ( and PS3 titles ) as well as PC games for PC , PSP , POOL , Wii ( TM ) ( and successor title ) and IP ( R ) , in addition to its focused gaming , entertainment and communication services . BlueBay ’s exclusive licensee worldwide licensee of the Bluepoint ( TM ) ZMFAO Gateway series , it is the world ’s leading portable gaming , PC and mobile phone company . For more information , see UNK , Inc. and ” Oakpoint : ZWC ’s Community Health Business Development Center . Bluepoint Games , Inc. is a highly experienced licensing , gaming and entertainment firm focused on developing the next generation of casual games based on the PlayStation ( R ) BRAVIA family of video game machines for the North American market . Bluepoint is a wholly owned subsidiary of Bluehill ID Holdings L.P. Bluepoint Games , Inc. is a highly experienced , innovative entertainment sports gaming company whose products and services are used by some of the most recognized and respected names in the world of gaming including : Pokemon , Macau ( Valve ) , Quattro , Super Smash Bros. , Good Neighbor Games , IGN Games , Vail Resorts , Kania ( Ocean Spray , Pemberton and Roatenham ) , PURE Holdings , TeenNick , National Amusements , SEGA Games , Cirrus ( Aircraft ) and www.netapool.com. Bluepoint Games , Inc. is a highly experienced player in the growing genre of casual games for both casual and active gaming enthusiasts . Bluepoint is an early stage Company with a significant following among youth and adults in Europe and the United States with an impressive track record in global on-line gaming opportunities . Nursing Homes : Genworth ’s 2009 Cost of Care Survey , conducted by the Robert Wood Johnson Foundation and released today , reveals the extent to which members of the U.S. population adheres to practices recommended since 1995 , including : a rolling three-hour ” Python for Life ” that fell asleep from 11 p.m. to 2 a.m. , sleep time from 11 p.m. to 3 a.m. , spare time from 8 a.m. to 9 p.m. , and use of state-of-the art non-invasive technologies . A remodeling and refurbishment of hospital facilities is underway as the nation ’s economy begins to gain momentum . Similar to the previous years , Thinking About Health - Hear how health plans are working to address various congressional proposals to advance best practices in patient care and provide greater accountability , advocacy and transparency to consumers . Nursing Homes : Genworth ’s 2009 Cost of Care Survey is based on interviews with 516 family , friends and neighbors of insured and self-employed people conducted from Jan .

Review summaries:
This paper highlights and studies the problem of "entropy rate drift" for language models: the entropy rate of the language generated by a trained model is much higher than the entropy of ground truth sequences and this discrepancy worsen with the length of generation. The authors interestingly claim that the well-known lack of coherence in long-term model generations is due to this entropy rate drift. Valuably, the entropy rate drift is characterized mathematically. The authors propose a calibration method and prove that their method can interestingly reduce *both* the entropy rate drift and the perplexity of a miscalibrated model, *even though* they assume a rather simplistic model of miscalibration (one that leaks a small amount of mass to all the sequences of a given length). The author quantitatively show that their calibration method reduces the entropy rate drift and qualitatively show that their generations "make sense" in the long-term. As an auxiliary result, they show that a similar calibration method can be used to quantify the past information used by the model by upper-bounding the mutual information between the current prediction and the long-term past, given the short-term past, e.g. I(W_t | W<{t-\tau} | W_{t-\tau:t-1}).

This paper demonstrates that a left-to-right language model suffers a high entropy rate when generating a long-term sequence of words. Then the authors claim that this is because of entropy rate amplification, which could be mitigated by 'calibration'. With local entropy rate calibration, a language model could achieve lower perplexity generating shorter and concise sequences of words.