Calibration, Entropy Rates, and Memory in Language Models

1 INTRODUCTION . Recent advances in language modeling have resulted in significant improvements on a wide variety of benchmarks ( Dai et al. , 2018 ; Gong et al. , 2018 ; Takase et al. , 2018 ) . Capturing long-term dependencies has especially been a major focus , with approaches ranging from explicit memorybased neural networks ( Grave et al. , 2016 ; Ke et al. , 2018 ) to optimization improvements to stabilize learning ( Le et al. , 2015 ; Trinh et al. , 2018 ) . However , while these techniques seem to improve on standard metrics like perplexity and even produce remarkably coherent text ( Radford et al. , 2019 ) , we still do not have appropriate measures to assess long-term properties in language models , making it difficult to choose between different model options for downstream tasks . Starting from Shannon ’ s seminal work that essentially introduced statistical language modeling ( Shannon , 1951 ) , the most classical and widely studied long-term property of a language model is its entropy rate — the average amount of information contained per word , conditioned on the preceding words . A learned model provides an upper bound for the entropy rate of a language , via its cross-entropy loss . The exponential of the entropy rate can be interpreted as the effective support size of the distribution of the next word ( intuitively , the average number of “ plausible ” word choices to continue a document ) , and the perplexity score of a model ( the exponential of the cross entropy loss ) is an upper bound for this quantity . In state-of-the-art models trained on billion-scale corpora , this number ranges between 10 and 30 ( Melis et al. , 2017 ; Radford et al. , 2019 ) . A natural diagnostic question , with which we begin our work , is whether the long-term generations of these models exhibit the same entropy rates as the underlying languages they are modeling predictively . Empirically , and perhaps surprisingly , it turns out that the entropy rate of generated text is substantially higher than the estimate for true text derived from the model ’ s one-step predictions . As seen in Table 1 ( see also Figure 1 ) , this is true for both state-of-the-art LSTMs and Transformers trained on a variety of datasets . As a timely example , the GPT-2 model ( Radford et al. , 2019 ) , the object of much recent attention for its seemingly coherent and on-topic generations , suffers a dramatic degradation in its entropy rate , from 23.7 to 61.2 . This empirical finding is notable since the neural attention- and memory-based techniques have been steadily improving on standard metrics like perplexity and , in some cases , even produce remarkably coherent text ( often with some heuristics to reject poor generations ) . That the perplexity of generated text is so much higher than it is under the true distribution suggests that there are significant gaps in our current methodologies in accurately learning language models , particularly if we are interested in generating text that globally resembles the modeled language itself . Our contributions . The focus of this work is twofold : to improve generations based on any measurement mismatch on a long-term property of the model ( e.g . the entropy rate ) , and to quantify the way a model ’ s predictions depend on the distant past . Central to both of these is a calibrationbased approach , which is utilized in statistics and other areas of machine learning ( Dawid , 1982 ; 1985 ; Foster , 1991 ; Zadrozny and Elkan , 2002 ; Platt , 1999 ; Guo et al. , 2017 ; Niculescu-Mizil and Caruana , 2005 ) . First , we show that , from a worst-case perspective , even an extremely accurate model ( with ε average KL divergence from the true distribution ) may have generated text with a substantially different entropy rate as compared to the true distribution . Indeed , we show that this worst-case amplification may occur for a variety of long-term properties of a probabilistic language model ; this is because the one-step KL divergence does not in general provide tight control over the expectation of a bounded function . The observed entropy rate amplification ( as seen in Table 1 ) demonstrates that this is not only of theoretical concern . We then describe a calibration procedure to fix this mismatch while simultaneously improving the perplexity of the language model . From a statistical perspective , the procedure is simple , and we discuss approaches to make it computationally efficient . Second , we provide a definition for long-term memory in language models as the mutual information between the models predictions and the distant past in the input . We then provide an upper bound on the amount of this mutual information using calibrated distributions ( with a single-parameter exponent ) . This allows us to estimate the amount of context used by a language model as a function of the distance of past tokens from the current prediction time step . We perform empirical studies to accompany our theoretical results . We first use the entropy rate calibration algorithm to fix an LSTM language model , resulting in a drop of around 20 perplexity points in the generated text ( so that the entropy rate of the model more accurately matches that of the language itself ) . Then , we empirically estimate and compare the long-term memory of state-ofthe-art language models . Our insights point towards new ways of assessing ( and fixing ) language models , especially in terms of their long-term properties , in a manner complementary to existing metrics like perplexity . 2 RELATED WORK . Improving language modeling with long-term dependencies . Recent approaches to improving language modeling have focused on several ways to better capture long-term dependencies , from using manually-defined context representations ( Mikolov and Zweig , 2012 ; Ji et al. , 2015 ; Wang and Cho , 2016 ) or document-level topics ( Wang et al. , 2017 ) to using LSTM recurrent neural networks with careful initialization ( Le et al. , 2015 ) , auxiliary loss signals ( Trinh et al. , 2018 ) or augmented memory structures ( Grave et al. , 2016 ; Ke et al. , 2018 ) . Wiseman and Rush ( 2016 ) use scoring functions over sequences and search-based optimization to improve generation in seq2seq models . More recent work has demonstrated the applicability of Transformer networks ( Vaswani et al. , 2017a ) to the task , potentially side-stepping issues in training recurrent networks ( e.g . vanishing/exploding gradients ) and scaling to longer contexts ( Dai et al. , 2018 ; Radford et al. , 2018 ) . All these papers propose either architectural or optimization innovations to improve language model training . In contrast , we define and measure explicit long-term properties of language models and show that calibrating them correctly can provide improvements to any black-box language model . Recent empirical breakthroughs have stemmed from language models which do not specify a unique autoregressive factorization ( Devlin et al. , 2018 ; Yang et al. , 2019 ; Liu et al. , 2019 ) , and thus do not specify a unique P̂ r. It remains an interesting problem to identify and sample from distributions induced by these models ( Wang and Cho , 2019 ) ; thus , our end-to-end theoretical guarantees do not hold in this setting . Information-theoretic approaches . While most language models aim to predict a distribution over the next token conditioned on the context , there have been alternative approaches relying on information-theoretic measures . Jost and Atwell ( 1994 ) propose a model which makes use of mutual information between word pairs to generate word sequences that retain longer-term dependencies . McAllester ( 2018 ) propose a training objective based on mutual information for predictive modeling , and demonstrate its application for phoneme prediction . Clarkson and Robinson ( 1999 ) develop a hybrid metric using both perplexity and entropy rate , and show that it correlates better with a downstream metric like word error rate . Such works propose alternative optimization objectives ; in contrast , we show how to use information-theoretic measures to improve models with respect to existing objectives like cross-entropy . Measuring long-term statistics . Khandelwal et al . ( 2018 ) analyze LSTM-based language models and empirically show that such models make use of a finite context for prediction . Lin and Tegmark ( 2017 ) measure mutual information between any two symbols in human languages , and show that it decays with distance , roughly following a power law distribution . Takahashi and Tanaka-Ishii ( 2018 ) provide an upper bound for the entropy ( character-level ) of human languages by training neural language models with various context and data sizes and extrapolating to infinity . While we also make use of measures like entropy and mutual information across longer contexts , our goal is to use these to better calibrate the language model and provably improve its perplexity . Calibration and integral probability metrics . The idea of matching properties of the models ’ predictions to the empirical outcomes , in an online setting , goes back ( at least ) to the “ prequential principle ” of Dawid ( 1982 ; 1985 ) , with subsequent work in online and game-theoretic settings ( Foster , 1991 ; Vovk , 2001 ; Kalai et al. , 1999 ) . The idea of improving probability scores is also common in machine learning ( Zadrozny and Elkan , 2002 ; Platt , 1999 ; Guo et al. , 2017 ; Niculescu-Mizil and Caruana , 2005 ) . Recently , Ott et al . ( 2018 ) assessed model calibration for machine translation systems using word-level probabilities . The notion of examining the expectation of functions as a metric for the distance between two distributions sometimes goes under the name of integral probability metrics ( Mller , 1997 ; Sriperumbudur et al. , 2009 ) , and this notion is becoming increasingly relevant again in unsupervised learning through the connections to GANs ( Mroueh and Sercu , 2017 ) . In this work , we directly focus on the KL divergence , where our use of calibration is largely based on basic facts about exponential families ( Brown , 1986 ) . Relation to generation-improving heuristics . If the sole objective is to improve the qualitative coherency of sampled generations , a wide variety of heuristics exist in the literature . The simplest of these is a constant multiplicative adjustment to the model ’ s logits ( known as softmax temperature Xie ( 2017 ) ) . This is a specific version of our method ( Algorithm 2 ) with a constant logistic regression feature instead of the next-token conditional entropy . Relatedly , greedy and top-k sampling ( used in state-of-the-art works such as ( Radford et al. , 2019 ) ) are heuristics which make local modifications to the model ’ s conditional probabilities to decrease diversity and eliminate nonsensical generations . Efforts to push the empirical state of the art in generation quality have given rise to more complex heuristics . Bengio et al . ( 2015 ) propose retraining the network on its own generations with a carefully scheduled probability for each token . Some works regularize a model ’ s generations with an auxiliary reverse language model Zhang et al . ( 2019 ) ; Liu et al . ( 2016 ) . Yet others promote realism using adversarial training protocols ( Bahdanau et al. , 2016 ; Lin et al. , 2017 ; Fedus et al. , 2018 ) . We stress that our calibration methods result in a provable improvement in the original training objective ( i.e . lower perplexity ) . As far as we know , none of the aforementioned heuristic methods can hope to provide such a strong guarantee , since they are fundamentally designed to bias models towards a different objective . Our work mitigates model hallucinations for ( almost ) free1 , in the sense that the global objective ( entropy rate drift ) is improved without worsening the local objective ( perplexity ) . Furthermore , calibration preserves the computational efficiency of density estimation : a conditional probability vector from Algorithm 2 can be computed using O ( vocabulary size ) inferences on the original model . In the more advanced heuristics , the implied distribution over sequences is lost , and is only accessible by black-box sampling .

Summary: This paper highlights and studies the problem of "entropy rate drift" for language models: the entropy rate of the language generated by a trained model is much higher than the entropy of ground truth sequences and this discrepancy worsen with the length of generation. The authors interestingly claim that the well-known lack of coherence in long-term model generations is due to this entropy rate drift. Valuably, the entropy rate drift is characterized mathematically. The authors propose a calibration method and prove that their method can interestingly reduce *both* the entropy rate drift and the perplexity of a miscalibrated model, *even though* they assume a rather simplistic model of miscalibration (one that leaks a small amount of mass to all the sequences of a given length). The author quantitatively show that their calibration method reduces the entropy rate drift and qualitatively show that their generations "make sense" in the long-term. As an auxiliary result, they show that a similar calibration method can be used to quantify the past information used by the model by upper-bounding the mutual information between the current prediction and the long-term past, given the short-term past, e.g. I(W_t | W<{t-\tau} | W_{t-\tau:t-1}).