Evaluating Distributional Distortion in Neural Language Modeling

A fundamental characteristic of natural language is the high rate at which speakers produce novel expressions . Because of this novelty , a heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language ( Baayen , 2001 ) . Standard language modeling metrics such as perplexity quantify performance of language models ( LM ) in aggregate . As a result , we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events . To address this gap , we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities . Training LMs on generations from these artificial languages , we compare the sequence-level probability estimates given by LMs to the true probabilities in the target language . Our experiments reveal that LSTM and Transformer language models ( i ) systematically underestimate the probability of sequences drawn from the target language , and ( ii ) do so more severely for lessprobable sequences . Investigating where this probability mass went , ( iii ) we find that LMs tend to overestimate the probability of ill-formed ( perturbed ) sequences . In addition , we find that this underestimation behaviour ( iv ) is weakened , but not eliminated by greater amounts of training data , and ( v ) is exacerbated for target distributions with lower entropy . 1 INTRODUCTION . Natural language is fundamentally creative—speakers and listeners frequently produce and comprehend sentences which have never been produced before ( Fodor , 1975 ; Fodor & Pylyshyn , 1988 ; Chomsky , 1975 , 1955 ) . As a side-effect of this property , distributions in natural language are characterized by a heavy-tail of individually improbable events which collectively account for a significant amount of the total probability mass of the distribution ( Khmaladze , 1988 ; Baayen , 2001 ) . Precisely approximating this large number of rare events is one of the foundational challenges for models of natural language ( Good , 1953 ; Jelinek , 1980 ; Katz , 1987 ; Kneser & Ney , 1995 ; Wood et al. , 2011 ; Goldwater et al. , 2011 ) . Autoregressive neural language models ( Bengio et al. , 2003 ; Mikolov et al. , 2013 ; Radford et al. , 2019 ) attempt to do so by decomposing the probability of an event ( a sequence ) into a series of conditional distributions , each parameterized by a shared neural network . Recently , a growing body work has sought to understand how these language models ( LM ) fit the distribution of a language beyond standard measures such as perplexity . Meister & Cotterell ( 2021 ) , for example , investigated the statistical tendencies of the distribution defined by neural LMs , whereas Kulikov et al . ( 2021 ) explored whether they adequately capture the modes of the distribution they attempt to model . At the same time , increased focus has been given to performance on rare or novel events in the data distribution , both for models of natural language ( Lent et al. , 2021 ; Dudy & Bedrick , 2020 ; Oren et al. , 2019 ) and neural models more generally ( see , for example Sagawa et al. , 2020 ; D ’ souza et al. , 2021 ; Chen et al. , 2021 ; Blevins & Zettlemoyer , 2020 ; Czarnowska et al. , 2019 ; Horn & Perona , 2017 ; Ouyang et al. , 2016 ; Bengio , 2015 ; Zhu et al. , 2014 ) . Neither of these branches of work , however , has explored instance-level LM performance on rare sequences in the distribution . As a result , we have relatively little understanding of how neural LMs approximate sequences in the heavy-tail characteristic of natural language . In this work , we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution . Our instance-level evaluation scheme explicitly compares the target probability distribution of the language to the distribution defined by the LM . Since the true distribution of any natural language is in practice unknown , we use a Transformer LM trained on natural data as a generative model to define target artificial languages for which we can exactly compute sequence probabilities . Training LSTM and Transformer LMs on sequences sampled from these target artificial languages , we compare the sequence-level probability estimates given by neural LMs to the target probabilities in the language . By controlling the entropy of the generative model ’ s conditional distributions , we create a set of artificial languages with varying distributional properties , and analyze how LM estimation behaviour is modulated by the properties of the target distribution . Our experiments uncover the extent to which neural LMs provide a distorted fit of the language they are trained to model . We find that LSTM and Transformer LMs ( i ) systematically underestimate the probability of sequences drawn from the target language and ( ii ) do so more when such sequences are rare . Where did this underestimated probability mass go ? We do not find that the underestimation is accompanied by overestimation in the head of distribution . Rather , we find that LMs tend to ( iii ) overestimate the probability of rare perturbed ( ill-formed ) sequences . Interpreted together , these findings indicate that on the one hand , neural LMs under-represent well-formed sequences in the tail of the language they attempt to model , and on the other hand , over-represent ill-formed sequences far away from high probability zones in sequence-space . In addition , we find that ( iv ) greater amounts of training data lessen underestimation but do not eliminate it and that ( v ) underestimation is exacerbated for target distributions with lower entropy . 2 BACKGROUND . We begin by briefly characterizing why distributions with a large number of rare events ( LNRE ) emerge in natural language , and why these events pose challenges for LMs . Furthermore , we motivate the need for instance-level evaluation when dealing with a large number of rare events . Productivity In the context of language production , a language user has the ability to produce , at any given point in their linguistic lifespan , an utterance which they have never produced before . This creativity is the result of the generative property of productivity , which states that on the basis of finite linguistic experience , a language user can produce and comprehend an unbounded number of grammatically acceptable utterances ( Chomsky , 1975 , 1955 ) . Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes . Because of this property , many of the distributions in natural language—particularly the distribution over the sequences of a language—are characterized by a heavy-tail of rare events . LNRE Zone To make explicit the connection between productivity and a heavy-tail of rare events , let PN denote the probability of sampling a novel ( previously unseen ) event from some distribution after having sampled N events . Then productivity as described above states that PN > 0 for all sample sizes N that occur in practice . The range of sample sizes N for which it is the case that PN > 0 is known as the LNRE zone ( Khmaladze , 1988 ; Baayen , 2001 ) . The LNRE zone for natural language appears to be very large , and it seems likely that PN will remain greater than 0 for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone , it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation ( MLE ) . Accounting for this enormous amount of novelty is thus a central challenge in language modeling . Language modeling A model M of the language L attempts to define a distribution pM over variable length sequences x = ( x1 , . . . , x|x| ) which closely resembles the true distribution of the language pL . That is , pM ( x ) ≈ pL ( x ) for all x ∈ Σ∗ , where Σ denotes the vocabulary of M , and Σ∗ is the set of all strings of finite length , known as the Kleene closure of Σ . In the LNRE zone , this means that a LM must define a distribution over a support containing a very large set of sequences 1For an empirical validation of this claim on a sample of practical size from the OpenWebText corpus , see the Appendix . which have never occurred in a training corpus ( or equivalently , have all occurred with frequency 0 ) , which take on a very wide array of differing probabilities . For example , while the sequences x1 , x2 ∈ Σ∗ have likely never occurred in any sample of English , most would agree that x1 is far more probable than x2 : x1 : The East pond in Parc Lafontaine was filled to the brim with diet Coke . x2 : Certain nak indicate liberationing among theorter codity voters vandalized . LM Evaluation For a perfect LM of English , we would expect the estimated probabilities of the sequences x1 and x2 to match their probabilities under the true distribution pEnglish . However , since pEnglish and it ’ s underlying generative process are unknown , it is not possible to explicitly evaluate how closely instance-level probability estimates align . As a proxy , the mean perplexity of the model on a holdout set of sequences D is typically used : PP ( pM , D ) = 1 |D| |D|∑ i=1 exp − 1|x ( i ) | |x ( i ) |∑ t=1 log pM ( x ( i ) t | x ( i ) < t ) ( 1 ) which measures whether the model , on average , assigns high likelihood to unseen instances . This measure does not , however , tell us whether instance-level estimates align with their true counterparts , nor is it necessarily indicative of performance on rare , idiosyncratic events in D. In this way , the lack of access to the ground-truth distribution severely complicates LM evaluation on the heavy-tail of rare sequences in language . In the following section , we introduce a methodology to overcome these limitations . 3 LANGUAGE MODEL EVALUATION IN THE LNRE ZONE . We propose evaluating language model performance on the heavy-tail of rare events by working with a known probability distribution over sequences . Specifically , we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model L. The distribution over sequences induced by a sampling scheme from L , denoted pL , is then our artificial language . We expect a model M of this artificial language to assign probabilities pM ( x ) to sequences x which match the target probabilities pL ( x ) of x under pL . Here we define pL using an ancestral sampling scheme with softmax T = 0.85 . One could , for example , define an artificial language pL′ by sampling from the top-k tokens of the distribution defined by L at each time step , or using an ancestral sampling scheme with softmax T = T ′ . To characterize neural LM behaviour on rare events , we train Transformer and LSTM LMs on data sampled from pL , and compare the instance-level probability estimates given by pM to target probabilities under pL . We summarize the components of this methodology in Table 1 , and overview it in greater detail in the following section .

Summary: This paper investigates how language models allocate their probability mass, with an emphasis on rare sequences that are part of the 'heavy tail' of the distribution of natural language sequences. The authors use a language model to define a target distribution with access to samples and ground-truth probabilities. A language model is trained on an empirical estimate of the target distribution, and the authors study the gap between the learned model's and target distribution's probability assignments.   Using this methodology, the authors uncover various interesting phenomena: the model systematically assigns lower probabilities than the target distribution, but assigns unusually high probabilities to unnatural, perturbed sequences (suggesting an explanation for where the probability mass moved to). The authors include several fine-grained analyses with additional interesting findings.