Evaluating Distributional Distortion in Neural Language Modeling

A fundamental characteristic of natural language is the high rate at which speakers produce novel expressions . Because of this novelty , a heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language ( Baayen , 2001 ) . Standard language modeling metrics such as perplexity quantify performance of language models ( LM ) in aggregate . As a result , we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events . To address this gap , we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities . Training LMs on generations from these artificial languages , we compare the sequence-level probability estimates given by LMs to the true probabilities in the target language . Our experiments reveal that LSTM and Transformer language models ( i ) systematically underestimate the probability of sequences drawn from the target language , and ( ii ) do so more severely for lessprobable sequences . Investigating where this probability mass went , ( iii ) we find that LMs tend to overestimate the probability of ill-formed ( perturbed ) sequences . In addition , we find that this underestimation behaviour ( iv ) is weakened , but not eliminated by greater amounts of training data , and ( v ) is exacerbated for target distributions with lower entropy . 1 INTRODUCTION . Natural language is fundamentally creative—speakers and listeners frequently produce and comprehend sentences which have never been produced before ( Fodor , 1975 ; Fodor & Pylyshyn , 1988 ; Chomsky , 1975 , 1955 ) . As a side-effect of this property , distributions in natural language are characterized by a heavy-tail of individually improbable events which collectively account for a significant amount of the total probability mass of the distribution ( Khmaladze , 1988 ; Baayen , 2001 ) . Precisely approximating this large number of rare events is one of the foundational challenges for models of natural language ( Good , 1953 ; Jelinek , 1980 ; Katz , 1987 ; Kneser & Ney , 1995 ; Wood et al. , 2011 ; Goldwater et al. , 2011 ) . Autoregressive neural language models ( Bengio et al. , 2003 ; Mikolov et al. , 2013 ; Radford et al. , 2019 ) attempt to do so by decomposing the probability of an event ( a sequence ) into a series of conditional distributions , each parameterized by a shared neural network . Recently , a growing body work has sought to understand how these language models ( LM ) fit the distribution of a language beyond standard measures such as perplexity . Meister & Cotterell ( 2021 ) , for example , investigated the statistical tendencies of the distribution defined by neural LMs , whereas Kulikov et al . ( 2021 ) explored whether they adequately capture the modes of the distribution they attempt to model . At the same time , increased focus has been given to performance on rare or novel events in the data distribution , both for models of natural language ( Lent et al. , 2021 ; Dudy & Bedrick , 2020 ; Oren et al. , 2019 ) and neural models more generally ( see , for example Sagawa et al. , 2020 ; D ’ souza et al. , 2021 ; Chen et al. , 2021 ; Blevins & Zettlemoyer , 2020 ; Czarnowska et al. , 2019 ; Horn & Perona , 2017 ; Ouyang et al. , 2016 ; Bengio , 2015 ; Zhu et al. , 2014 ) . Neither of these branches of work , however , has explored instance-level LM performance on rare sequences in the distribution . As a result , we have relatively little understanding of how neural LMs approximate sequences in the heavy-tail characteristic of natural language . In this work , we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution . Our instance-level evaluation scheme explicitly compares the target probability distribution of the language to the distribution defined by the LM . Since the true distribution of any natural language is in practice unknown , we use a Transformer LM trained on natural data as a generative model to define target artificial languages for which we can exactly compute sequence probabilities . Training LSTM and Transformer LMs on sequences sampled from these target artificial languages , we compare the sequence-level probability estimates given by neural LMs to the target probabilities in the language . By controlling the entropy of the generative model ’ s conditional distributions , we create a set of artificial languages with varying distributional properties , and analyze how LM estimation behaviour is modulated by the properties of the target distribution . Our experiments uncover the extent to which neural LMs provide a distorted fit of the language they are trained to model . We find that LSTM and Transformer LMs ( i ) systematically underestimate the probability of sequences drawn from the target language and ( ii ) do so more when such sequences are rare . Where did this underestimated probability mass go ? We do not find that the underestimation is accompanied by overestimation in the head of distribution . Rather , we find that LMs tend to ( iii ) overestimate the probability of rare perturbed ( ill-formed ) sequences . Interpreted together , these findings indicate that on the one hand , neural LMs under-represent well-formed sequences in the tail of the language they attempt to model , and on the other hand , over-represent ill-formed sequences far away from high probability zones in sequence-space . In addition , we find that ( iv ) greater amounts of training data lessen underestimation but do not eliminate it and that ( v ) underestimation is exacerbated for target distributions with lower entropy . 2 BACKGROUND . We begin by briefly characterizing why distributions with a large number of rare events ( LNRE ) emerge in natural language , and why these events pose challenges for LMs . Furthermore , we motivate the need for instance-level evaluation when dealing with a large number of rare events . Productivity In the context of language production , a language user has the ability to produce , at any given point in their linguistic lifespan , an utterance which they have never produced before . This creativity is the result of the generative property of productivity , which states that on the basis of finite linguistic experience , a language user can produce and comprehend an unbounded number of grammatically acceptable utterances ( Chomsky , 1975 , 1955 ) . Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes . Because of this property , many of the distributions in natural language—particularly the distribution over the sequences of a language—are characterized by a heavy-tail of rare events . LNRE Zone To make explicit the connection between productivity and a heavy-tail of rare events , let PN denote the probability of sampling a novel ( previously unseen ) event from some distribution after having sampled N events . Then productivity as described above states that PN > 0 for all sample sizes N that occur in practice . The range of sample sizes N for which it is the case that PN > 0 is known as the LNRE zone ( Khmaladze , 1988 ; Baayen , 2001 ) . The LNRE zone for natural language appears to be very large , and it seems likely that PN will remain greater than 0 for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone , it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation ( MLE ) . Accounting for this enormous amount of novelty is thus a central challenge in language modeling . Language modeling A model M of the language L attempts to define a distribution pM over variable length sequences x = ( x1 , . . . , x|x| ) which closely resembles the true distribution of the language pL . That is , pM ( x ) ≈ pL ( x ) for all x ∈ Σ∗ , where Σ denotes the vocabulary of M , and Σ∗ is the set of all strings of finite length , known as the Kleene closure of Σ . In the LNRE zone , this means that a LM must define a distribution over a support containing a very large set of sequences 1For an empirical validation of this claim on a sample of practical size from the OpenWebText corpus , see the Appendix . which have never occurred in a training corpus ( or equivalently , have all occurred with frequency 0 ) , which take on a very wide array of differing probabilities . For example , while the sequences x1 , x2 ∈ Σ∗ have likely never occurred in any sample of English , most would agree that x1 is far more probable than x2 : x1 : The East pond in Parc Lafontaine was filled to the brim with diet Coke . x2 : Certain nak indicate liberationing among theorter codity voters vandalized . LM Evaluation For a perfect LM of English , we would expect the estimated probabilities of the sequences x1 and x2 to match their probabilities under the true distribution pEnglish . However , since pEnglish and it ’ s underlying generative process are unknown , it is not possible to explicitly evaluate how closely instance-level probability estimates align . As a proxy , the mean perplexity of the model on a holdout set of sequences D is typically used : PP ( pM , D ) = 1 |D| |D|∑ i=1 exp − 1|x ( i ) | |x ( i ) |∑ t=1 log pM ( x ( i ) t | x ( i ) < t ) ( 1 ) which measures whether the model , on average , assigns high likelihood to unseen instances . This measure does not , however , tell us whether instance-level estimates align with their true counterparts , nor is it necessarily indicative of performance on rare , idiosyncratic events in D. In this way , the lack of access to the ground-truth distribution severely complicates LM evaluation on the heavy-tail of rare sequences in language . In the following section , we introduce a methodology to overcome these limitations . 3 LANGUAGE MODEL EVALUATION IN THE LNRE ZONE . We propose evaluating language model performance on the heavy-tail of rare events by working with a known probability distribution over sequences . Specifically , we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model L. The distribution over sequences induced by a sampling scheme from L , denoted pL , is then our artificial language . We expect a model M of this artificial language to assign probabilities pM ( x ) to sequences x which match the target probabilities pL ( x ) of x under pL . Here we define pL using an ancestral sampling scheme with softmax T = 0.85 . One could , for example , define an artificial language pL′ by sampling from the top-k tokens of the distribution defined by L at each time step , or using an ancestral sampling scheme with softmax T = T ′ . To characterize neural LM behaviour on rare events , we train Transformer and LSTM LMs on data sampled from pL , and compare the instance-level probability estimates given by pM to target probabilities under pL . We summarize the components of this methodology in Table 1 , and overview it in greater detail in the following section .

Summary: The paper conducts experiments to evaluate whether neural sequence models such as LSTMs and Transformers are able to correctly assess the probability of rare sentences, which collectively constitute a large probability mass in natural language productions (heavy-tail phenomenon). In order to do so, it performs experiments in a controlled synthetic environment where a first language model $P_L$ is trained on a corpus of natural sentences, and a second model $P_M$ is trained to emulate the first model. The authors observe that the second model systematically tends to underestimate the probability of rare $P_L$ sentences, the more so the rarer such sentences are, and show that some artificially corrupted sentences tend to receive higher probability from $P_M$ than from $P_L$, partly explaining where the missing probability mass over rare well-formed sentences went.