Evaluating Distributional Distortion in Neural Language Modeling
Authors: Benjamin LeBrun
Alessandro Sordoni
Timothy J. O'Donnell
Venue: ICLR 2022 Poster (2022)

Abstract: A fundamental characteristic of natural language is the high rate at which speakers produce novel expressions. Because of this novelty, a heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language (Baayen, 2001). Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate.  As a result, we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events. To address this gap, we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities. Training LMs on generations from these artificial languages, we compare the sequence-level probability estimates given by LMs to the true probabilities in the target language. Our experiments reveal that LSTM and Transformer language models (i) systematically underestimate the probability of sequences drawn from the target language, and (ii) do so more severely for less-probable sequences. Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill formed (perturbed) sequences. In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

Published as a conference paper at ICLR 2022
EVALUATING DISTRIBUTIONAL DISTORTION IN NEURAL LANGUAGE MODELING
Benjamin LeBrun 12.1 Alessandro Sordoni** & Timothy J. O'DonnelJ:224"
'ceGill University Mill Quebec Artificial Intelligence Institute 'Microsoft Research *Canada CIFAR Al Chair, Mila
ABSTRACT
A fundamental characteristic of natural language is the high rate at which speak- produce novel expressions. Because of this novelty, heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language (Baayen 200T). Standard language modeling metries such per- plexity quantify the performance of language models (LM) in aggregate. As a result, we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events. To address this gap, develop controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities. Training LMs on generations from these artificial lan- guages, compare the sequence-level probability estimates given by LMs to the true probabilities in the target language. Our experiments reveal that LSTT and Transformer language models (i) systematically underestimate the probability of sequences drawn from the target language, and (ii) do so more severely for less- probable sequences. Investigating where this probability mass went, (ii) we find that LMs tend to overestimate the probability of ill-formed (perturbed) sequences. In addition, find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.
INTRODUCTION
Natural language fundamentally cpeative-speakers and listeners frequently produce and comprehend sen- tences which have never been produced before (Fodor 1975 Fodor & Pylyshyn| 1988 Chomsky 1975, 1955). As a side-effect this property, distributions in natural language characterized by heavy-tail of individu- ally improbable events which collectively account for a significant amount of the total probability mass of the distribution (Khmaladze] 1988 Baayen 2001). Precisely approximating this large number of rare events is one of the foundational challenges for models of natural lan-
sequence log probability estimate
Perfect approximation
-30
-39
guage (Good 1953] Jelinek] 1980 Katz, 1987 Kneser & Ney [995] Wood et al.) 2011 Goldwater et al. (001)) Autoregressive neural language models (Bengio et al.
sequence log probability
Figure 1: GPT2 sequence probability es- timates plotted against the true sequence probabilities. Neural LMs underestimate the probability of sequences drawn from lan- guage they are trained to model.
2003 Mikolov et al. 2013; Radford et al. 2019) attempt to do so by decomposing the probability of an event (a sequence) into series of conditional distributions, each parameterized by a shared neural network.
Recently, growing body work has sought to understand how these language models (LM) fit the distribution of a language beyond standard measures such
CCoresponding author: benjamin. lebrun @ mail. megiil.ca maail. Co-senior autorship

---

Published as a conference paper at ICLR 2022
as perplexity. MMister & Cotterell (202I). for example, investigated the statistical tendencies of the distribution defined by neural LMs, whereas Kulikov et al. (202I) explored whether they adequately capture the modes of the distribution they attempt to model. At the same time, increased focus has been given to performance on rare or novel events in the data distribution, both for models of natural
Whereas Kulikov
language (McCoy al. 202T Lent et. [02T]] Dudy & Bedrick |000 Oren al. 2019) and neural
models more generally (see, for example Sagawa et al. [020 D souza et al. 2021 Chen ct al.
2021
Blevins & Zettlemoyer 2020; Czarnowska et al. 2019 Horn & Perona 2017 Ouyang et al.
2016
Bengio 2015 Zhu et al. (0144) Neither of these branches of work, however, has explored instance-
level LM performance on rare sequences in the distribution. As result, we have relatively little
understanding of how neural LMs approximate sequences in the heavy-tail characteristic of natural
language.
In this work, we introduce controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution. Our instance-level evaluation scheme explicitly compares the target probability distribution of the language to the distribution defined by the LM. Since the true distribution of any natural language is in practice unknown, we use a Transformer LM trained natural data generative model define target artificial languages for which we can exactly compute sequence probabilities. Training LSTM and Transformer LMs on sequences sampled from these target artificial languages, compare the sequence-level probability estimates given by neural LMs to the target probabilities in the language. By controlling the entropy of the generative model's conditional distributions, we create a set of artificial languages with varying distributional properties, and analyze how LM estimation behaviour is modulated by the properties of the target distribution.
Our experiments uncover the extent to which neural LMs provide a distorted fit of the language they are trained to model. We find that LSTM and Transformer LMs (i) systematically underesti- mate the probability of sequences drawn from the target language and (ii) do so more when such sequences are rare. Where did this underestimated probability mass go? We do not find that the underestimation is accompanied by overestimation in the head of distribution. Rather, we find that LMs tend to (iii) overestimate the probability of rare perturbed (ill-formed) sequences. Interpreted together, these findings indicate that on the one hand, neural LMs under-represent well-formed se- quences in the tail of the language they attempt to model, and on the other hand, over-represent ill-formed sequences far away from high probability zones in sequence-space In addition, we find that (iv) greater amounts of training data lessen underestimation but do not eliminate it and that (v) underestimation is exacerbated for target distributions with lower entropy.
BACKGROUND
We begin by briefly characterizing why distributions with large number of rare events (LNRE) emerge in natural language, and why these events pose challenges for LMs. Furthermore, we moti- vate the need for instance-level evaluation when dealing with large number of rare events.
Productivity In the context of language production, a language user has the ability to produce, at any given point in their linguistic lifespan, utterance which they have never produced before. This creativity is the result of the generative property of productivity, which states that on the basis of finite linguistic experience, language user can produce and comprehend an unbounded number of grammatically acceptable utterances (Chomsky/ 1975, 1955). Productive processes induce a dis- tribution which places non-zero probability on unseen events all practical sample sizes. Because of this property, many of the distributions in natural gegguage-patticlaatt the distribution over the sequences of language- are characterized by heavy-tail of rare events.
LNRE Zone To make explicit the connection between productivity and heavy-tail of rare events, let Pv denote the probability of sampling a novel (previously unseen) event from some distribution after having sampled N events. Then productivity as described above states that Px > 0 for all sample sizes N that occur in practice. The range of sample sizes for which it is the case that PN > 0 is known the LNRE zone (Khmaladze) 1988 Baayen 2001). The LNRE zone for natural language appears to be very large, and it seems likely that PN will remain greater than 0 for samples of natural language many orders of magnitude larger than all the data currently available
2

---

Published as a conference paper at ICLR 2022
for training LMs In the LNRE zone, it is difficult obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation (MLE). Accounting for this enormous amount of novelty is thus central challenge in language modeling.
Language modeling A model M of the language L attempts to define distribution pM which closely resembles the true distribution of the language pr.- In a locally-normalized autoregressive model, this distribution is defined by assigning probabilities to variable length sequences æ via chain-rule decomposition:
at
aa
exp pl 11
p()) = p(a; x1s--1)
(I)
EES exp p (1
2=1
where 2 is the vocabulary, and p( 0) is the non-negative score of token n; given the sequence prefix W1 which is computed by neural network with parameters
For PM perfectly approximate pr., expect (ar( pr. (x) for all x E E", where Σn is the set of all strings of finite length (the Kleene closure of Σ). In the LNRE zone, PM is defined over a support containing very large set of sequenees which have never occurred in training corpus (or cquivalently, have all occurred with frequency 0), and which take on a very wide array of differing probabilities. For example, while the sequences 11, 32 E" have likely never occurred in any sample of English, most would agree that 21 is far more probable than Ø2!
Φ1: The East pond in Pare Lafontaine was filled to the brim with Diet Coke.
T22 Certain nak indicate liberationing among theorter codity voters vandalized.
LM Evaluation For perfect LM of English, would expect the estimated probabilities of the sequences æ1 and 22 to match their probabilities under the true distribution Praglish- However, since PEnglish and it's underlying generative process are unknown, it is possible to explicitly evaluate how closely instance-level probability estimates align. As proxy, the mean perplexity of the model on holdout set of sequences DD is typically used, which measures whether the model, on average, assigns high likelihood to unseen instances. This measure does not, however, tell us whether instance- estimates align with their true counterparts, nor is necessarily indicative of performance on rare, idiosyncratic events in DD In this way, the lack of access to the ground-truth distribution severely complicates LM evaluation on the heavy-tail of rare sequences language. The following section introduces methodology to overcome these limitations.
LANGUAGE MODEL EVALUATION IN THE LNRE ZONE
Component
Notation
Description
Generative model Artificial language
L
A LM trained natural instance-level data. The distribution sequences induced sampling scheme from L.
PL
Language model
PM
The distribution of LM trained on sequences sampled from PL-
Target probabilities
(w))
The probability assigned by ps. to the sequence ac The probability assigned by to the sequence æ.
Model probabilities
PM (x)
Table 1: Components of our instance-level evaluation scheme. Training Pas on samples from PL we compare Par (x) to (z) for
E*
We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences. Specifically, we train Transformer LM on sequences sam- pled from a corpus of natural language to define a generative model T.. The distribution over se- quences induced by sampling scheme from J., denoted Pr. is then our artificial language. We ex- pect a model M of this artificial language to assign probabilities pM (x) to sequences x which match
For empirical validation of this claim sample of practical size from the Open OpenWeeTexxt corpus, the Appendix
3

---

four Published as a conference paper at ICLR 2022
the target probabilities PL (a) of under PL. To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from PL, and compare the instance-level probability estimates given by PM to target probabilities under Pr.- We summarize the components of this methodology in Table and overview in greater detail in the following section.
3.1 ARTIFICAL LANGUAGES AS TARGET DISTRIBUTIONS
logpr.
"we'r re very excited to have the opportunity to help them," he says. what's going to happen? to me, fisheries are in midst a global financial crisis.
199.3811
-17.1128
-11.4835
Table 2: Sample of sequences drawn from our artificial language.
To define a generative model L, we train a aundoll-innitiaizeed GPT2 HPT2-medium on 1.5M sentences sampled from the length to be 128 tokens. additionally train a byte-pair-encoding (BPE) tokenizer on this data set with standard GPT2 vocabulary size of 57,256 tokens. For simplicity, this tokenizer is used for all models.
corpus (Gokaslan & Cohen 2019) P
We set the maximum sequence
Using this generative model L. define the target distribution over sequences- the artificial language-as as the distribution induced by an ancestral sampling scheme from L. Thus, we draw instances (#i,..., F12)) from our language PL by recursively sampling from the conditional distribution over tokens at each time step: experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from L with softmax temperature T 0.85 In Section 4.5 explore the effects of different values of 7 when sampling from Pr. Table shows three sequences sampled from this distribution.
pr(- x<z) where
BOS and x, EOS. All
3.2 SEQUENCE PROBABILITY ESTIMATION IN THE LNRE ZONE
Given an artificial language PL. the task of the language model-is define a distribution PM whose sequence-level probability estimates closely align with the sequence-level probabilities given by pz. We refer to any deviation from this desiderata as model estimation error. To quantify the model estimation error for sequence æ, we take the difference between the sequence's 's log probability under M and its true log probability under
error(æ) po (a:) log (z.
This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence is under the language model M. Note that error (x) 0 indicates that M underestimates the probability of æ, whereas error(x:) > 0 indicates that M overestimates the probability of æ. In practice, train M set of sequences Diran sampled from pL, and compute model estimation еггоr on a separate set of sequences Dees sampled from pr. In all cases, we compute the probability of sequence
its chain rule decomposition: p(x) II!=] p(z, | x<()) where Zn = BOS and T|æ|| = EOS. When computing the ground-trum sequence probabilities for PL. take into account any softmax tempering.
3.3 NEURAL LANGUAGE MODELS
We study the estimation performance of two neural LM architectures: the Transformer [Vaswani al. (017)) and the LSTM (Melis al. 2020). When training either architecture, we halve the learning rate if validation loss increases at the end of an epoch. For all model sizes, we use batch size of 128 sequences. Models with the lowest cross-entropy loss on a withheld validation set are used in experiments unless otherwise mentioned.
obtained from Huggingface, and training done XTT-8000 GPUs (depending on model size) with mixed floating point precision.
2 All Transformer implementations
We temper in an effort to define ground-truit distribution whose entropy more closely resembles that of natural language. Note that our findings hold for untempered (T 1.00) ground-truth distributions well (see A.I.T and A.1.2
4

---

Published as a conference paper at ICLR 2022
LSTM
GPT2-small
GPT2-medium
Estimation error
-20
-40
logPx(X)
GPT2-small
-400
-120
GPT2-med
-500
-140
logP:(x)
logPL (x)
logPL(x)
logp)(x)
Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model's 's estimates perfectly align with the target probability; shading the right this line denotes underestimation. Right- most figure: Mcan sequence estimation eπror by target sequence probability.
We use the Huggingface (Wolf al. 2020) implementations of GPT2-small, GPT2-medium and GPT2-large (Radford et al. 2019) as representative Transformer LMs. We use Adam Optimization 4e and 35-5 for GPT2-small, -medium -mediul and
with 1e and learning rates 5e -large respectively. Since these models are in the same model class as our artificial target language PL. we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown. For the LSTM (Hochreiter implementation of the baseline LM described in (Melis et al. 20200). We use 2 layers and adjust the hidden state and embedding dimension (2048 and 1024, respectively) be such that the total number of parameters is approximately equal to GPT2-small (110M).
Schmidhuber 1997). we follow the
RESULTS
4.1 ESTIMATION ERROR WITH FIXED DATA
We begin by exploring model estimation error fixed training set Durin of 1M sequences sampled from PL. We first train LSTM and GPT models on Duains early-stopping as described above. Fol- lowing training, under both the model distribution pM and the true language distribution pr.- From this, we obtain x Diea). If the model M perfectly models the language L, then for each (x), (x)( E Stest we would expect (z = PM (x). Figures and BB) visualize this relationship with the z- and y-axes denoting the true and model estimated sequence probabilities respectively, and dashed line representing equality. To compare probability estimates, we represent the set Sue in the form of a joint histogram over this coordinate space. Histogram bins are shaded based on the number of tuples which lie in the coordinate range they define. Importantly, any deviation of this histogram from the identity line indicates that the model distorts the shape of the distribution of the language.
sample test set Diea of 500,000 sequences from Pr., and score each sequence
set of probability estimates: Sies = {(pr. (a)),
Figures [ and BB provide evidence for distributional distortion the form of underestimation. The majority of probability tuples in Sestt lie the right of the identity line, indicating that LSTM and GPT2 models consistently underestimate the probability of sequences sampled from PL- Fur- thermore, the distance between the identity line and the probability tuples grows non-linearly as function of the true sequence probability, indicating that underestimation error is more severe for rarer sequences in the language. We validate these observations in the right-most plot of Figure 2. which shows mean estimation error decreasing non-linearly as a function of the target sequences probability"" In addition, comparing underestimation behaviour across model size, we find that while @PT2-medium performs slightly better than GPT2-small, these improvements are typically within the range defined by the bootstrapped 95% confidence intervals. See A.1.3 for evidence indicating that this underestimation behaviour also occurs in pre-trained models fine-tuned on D
"To compute this curve, we split the target sequence probability pr (œ) range into N equally sized bins (by probability range). We report the mean estimation error for each bin with > 10 sequences. We additionally compute 95% confidence intervals with 10, 000 bootstraps for each mean, resampling equal to the number of sequences in the given hin.
5

---

Published as a conference paper at ICLR 2022
Train
Test
logputk: -lsppospppp
Meen estimation
J.J. ate underestimates
10
Training epoch [1M sequences)
Training egoch (1M sequences)
Figure 3: Mean model estimation emor by training epoch (GPT2-medium). Each line denotes the mean estimation error for 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches minimum.
4.2 ESTIMATION ERROR ACROSS TRAINING TIME
To understand the training dynamics underlying the previously reported underestimation, we com- pute model probability estimates on subsets of Dimin and Diest at the end of each training iteration
Once computed, we sort each set of probability tuples S and s(1) by their target sequence proba- bilities Pr. (z)) and split the probability tuples into 50 equally-sized bins. We plot estimation curves in Figure 3 Each curve represents 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent
the target probability the sequence.
Figure leff visualizes underestimation enror for sequences seen in training. Around the fifth epoch, estimation error for train sentences converges to zero, that is (PL(x) = par (z)), indicating that GPT2-medium is able almost perfectly recover the target probabilities of training sequences matter their target probability. At the same time, this convergence happens almost simultaneously for all sequences, indicating that complete reduction error during training occurs throughout the entire range of target sequence probabilities.
Figure right visualizes GPT2-medium model's performance on separate of test sequences. First, unlike for Duain. estimation eтror for Diest does not converge to zero, meaning that even when the model has perfectly recovered the target probability of train sequences, the target probabilities for test sequences remain underestimated. Second, in the case of Driin the difference between esti- mation curves of different shades converges to zero, indicating that estimation performance becomes uniform across the distribution of train sequences. We do not see such behaviour in Dies- Instead, the error curves remain at relatively consistent distance from one another, indicating that the dis- crepancy in estimation error at different parts of the distribution is unchanging for sequences not seen during training.
4.3 ESTIMATION ERROR BY AMOUNT OF TRAINING DATA
LSTM
GPT2-medium
GPT2-large
Number sequences seen in training (millons) Number of sequences seen training (millons)
Number sequences seen training (millions]
Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from PL- Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences function of the number of sequences seen in training. both cases, each line denotes estimation behaviour for 50th of the test sequences; darker lines represent less probable sequences.
Our previous experiment trained languages models nation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not enough samples from the target distribution. Here explore how
set of IM sequences. plausible expla-

---

Published as a conference paper at ICLR 2022
Low probability & low perturbation
Number of pertun perturbations performed
Sevene underwatimation
e.g. there are of great bands,, great fens inclusing the omazing corp mariner the dianis, the big dondonas, the snowthow. sommusta, altitar, the tampoon, the jamayyhe, the onezwan and outor and norwhhl
Model estimatioo erorr
Low probability high perturbation
Severe overestimation
intrusions chimed has rivited mocures unteedrancodde and 350ord dlementary innutations and community atotied los angales.
baok@enorrbirree d6 "wrest
472.0
312.0
152.0
True sequence probability logp((x)
Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underes- timation. We also include example sequences from two zones in this sequence space.
estimation error varies as a function the amount of training data. We train GPT2 m-e-um,, GPT2- large and LSTM model in the online "Ideal World" setting (Nakkiran al. 2020 by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from PL. and training M on this sample. Doing so for 60 iterations, we obtain LMs which have been trained 30 million sequences. We compute model estimation error on Prest at the end of each iteration i. Figure 4 vi- sualizes underestimation error throughout training for these LMs. We again split test sequences by their true probability, with darker lines denoting estimation trends for less probable target sequences.
The estimation curves Figure suggest that while increasing the amount of data in training ini- tially leads to lower estimation eтror, this reduction eventually asymptotes. In the insets of Figure we visualize the relative change mean estimation between epochs estimation error eventually fluctuates around 0 (minimal change) for the majority of the distribution. Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution.
and Relative change in
4.4 WHERE DID THE PROBABILITY MASS GO?
In the previous section, saw that even when increasing the amount of training data, PM consis- tently underestimates the probabilities of sequences sampled from the tail of pt. At the same time, we did not find that PM overestimated sequences in the head of pt. Under the assumption that pM is a proper probability distribution 5 that is, Exer- PM (x) 1, these findings suggest that there ex- ists sequences in >** whose probability is overestimated by the model. In this section, we investigate where this probability mass went.
To do so, compute model estimation error on perturbed sequences from Py sequeneces in Σ" which are increasingly far away from the high-probability zones in PL. We build a corpus of perturbed sequences by recursively applying 30 each sequence the set of sequences at perturbation step i can be expressed (t) perturbed (PERTURB(<)) x Di where PERTURB(x) is function which returns novel perturbed version of and (9)
Swap two tokens in æ. Delete token from ar. Insert token from E a position in
random perturbations
D Formally,
Substitute token in w with a token from E,
Table 3: PERTURB(Z)) randomly ap- plies one of these perturbations æ.
portureed Dest- Sequence perturbation operations are shown in Table 3 While is possible that these
operations produce other well-formed strings, we expect this be relatively rare outcome. We score each of these 15M sequences under both the target generative model and the LM PM Note that we use as LM the --r-mmm model from the previous section (trained on 30M sequences).
"See Welleck al. 2020al for discussion consistency in the context of neural language model decoding.

---

Published as a conference paper at ICLR 2022
Mean estime nionnsioon error logguv -loge
Frequency
T=0.00
T=0.85
-00
0.70
Frequency
200
7= 1.00
1.15
0.85
1.00
1.15
-100 True sequence log probability (xgo:(x)
-108
0-433
-100
-303
log
logoL(x)
Figure 6: Model estimation error on test sequences as a function target sequence probability for three different artificial languages. Each line visualizes estimation error for GPT2-medium model trained to model a language with a specific softmax temperature parameter T.
Figure 5 visualizes GPT2-medium's mean estimation error for these 15M sequences across two dimensions. On the x-axis, we plot the target probability of the sequence under pL, and on the y- axis, the number of perturbations performed on the sequence. For example, the bottom-left corner (I) of Figure F visualizes estimation behaviour for rare sequences sampled directly from pr, whereas the top-left corner (2) visualizes estimation error sequences which are equally rare, but which have been perturbed up to 30 times.
Figure [ offers a nuanced characterization of underestimation behaviour. The brown area on the bottom of the figure re-states the underestimation findings of the previous section. When increasing the number of perturbations performed, however, we begin entering into a space of sequences which first well-estimated by PM (the white areas) but then are quickly overestimated by pM (the dark blue areas), confirming that there are indeed sequences in Σ which are overestimated by the language model. Furthermore, these findings suggest that the tail of rare events defined by the language model does not match the tail of the artificial language- the rare events typical in are under-represented in PM in favour of other sequences in Σ* See Section A.1.4 for experiments finding that random sequences from E are also overestimated by PM.
4.5 MODULATING THE SHAPE OF THE TARGET DISTRIBUTION
Up to this point, the target artificial language Pz was given as distribution induced by an ancestral sampling scheme with softmax T 0.85 from the generative model L. In the previous section, saw that PM placed excess probability mass on areas in Σ* with low-probability under PL. Here we modulate the shape of the sequence space defined by Pr. to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that pr. allocates probability mass over V1, control the entropy of the conditional distributions at each generation step & by dividing the pre-softmax logits by temperature value 6
We visualize the effects of T the shape of the distribution in the left of Figure By increasing the value of T, we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in Σ*. We define four artificial languages with varying 2' and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform compara- tively better than models trained on low entropy languages. Estimation error for models trained on languages with greater 7' is less severe, and this holds throughout nearly all target sequence prob- abilities. These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across Σ*
RELATED WORK
This paper contributes to recent work investigating the properties of the distributions defined by
LMs. Prior studies have focused on exploring (Takahashi & Tanaka-Ishil 2019 2017) and develop-
Formally, for the ith component of the length K pre-softmax logits æ, this operation is given as: exp(#)
SOFTMAX(E): ΣK ΣK (+)
8

---

Published as a conference paper at ICLR 2022
ing frameworks (Meister & Cotterell 2021 to better understand whether the large-scale statistical tendencies of natural language, such as Zipf"s s law ZZipi 1949). are captured by LMs. We take a more fine fine-grained approach, proposing methodology which draws off of instance-level evaluation schemes (Zhong et al. 202T) and the experimental control afforded by artificial corpora (White & Cotterell 202T Papadi & Jurafsky (020). Indeed, closely related our work is islKulikov et al. (2 s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling. Our analysis exploring the overestimation of ill-formed sequences extends previous findings on locally normalized conditional models assigning arbitrary probability mass to unlikely sequences (Andor et al., [016)] Goyal al. 2019 Lafferty al.) (001), neural LMs assigning high likelihood to sequences with repetitions (Welleck et al. (0200)), the con- sistency of decoding algorithms (Welleck et al. 2020a), and on machine translation models placing significant probability mass on the empty sequence (Stahlberg & Byrne 2019).
We additionally contribute to the body work seeking to characterize and adapt neural model per- formance rare or novel examples and classes (Horn & Perona 2017, Bengio 2015). In the context of language modeling. Lent et al. (202I) explored performance on under-resourced lan- guages, whereas Oren al. (2019) did
under-represented domains in training corpora. [M-] Coy et al.] (2021) introduced analyses to assess sequential and syntactic novelty in LMs. Focusing on the word frequency distribution, Dudy & Bedrick (2020) found that LMs under-perform when less frequent examples are encountered at test time. In the classification setting, various approaches have been proposed to help alleviate class imbalance in the data distribution, such as data aug- mentation (Sagawa et al.) (2020) or the transfer of knowledge from high-frequency classes to infre- quent ones (Ouyang et al. [ 2016 Zhu et al.) 2014; Chen et al. 202T). Prior to the current neural paradigm (Bengio et al. 2003). multiple approaches have been proposed to deal with the heavy-tail, such as smoothing and back-off approaches in statistical n- n-grams (Chen & Goodman 1999) and
two-stage Bayesian approaches (Goldwater et al. 2006).
6 CONCLUSION
Emerging as a result of a language user's ability produce and comprehend novel expressions, the heavy-tail of rare events is one of the fundamental features of distributions in natural language. In this work, we introduce a controlled methodology to evaluate instance-level LM performance on this set of individually rare but collectively frequent events. We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMS on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language.
Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data. Investigating where this proba- bility mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typ- ical in the target distribution. Comparing model performance on target distributions with varying properties, we find that neural LMs tend provide more accurate approximations of distributions with greater entropy. Interpreted together, these results indicate that autoregressive neural language models have tendency to spread probability mass too uniformly across the space of possible se- quences.
Finally, we would like to acknowledge that we do not know the degree of structural difference between our ceaeffommr--enerrttee ground-truth distributions and the distributions of actual natural languages. It is likely that the distribution defined by our ground truth models is less structured than the distribution of natural language. Therefore, it is possible that some systematic difference between natural language distributions and our ground-truth distributions may affect our results to a certain degree. That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions. and we expect the task of recovering ground-truth truth distribution to be made easier when the target distribution and LM are in the same model class. Nevertheless, future work should seek to conduct similar experiments using ground- truth distributions with more explicit structure.

---

Published as a conference paper at ICLR 2022
REFERENCES
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-base based neural networks, 2016.
R. Baayen. Word frequency distributions. 2001.
R. Harald Baayen. Productivity in language production. Language and Cognitive Processes, 9()):
447-469 1994. doi: 10.
URI.
0169096940840222
Harald Baayen. 41. corpus linguistics in morphology: Morphological productivity. Corpus linguistics, pp. 899-919. De Gruyter Mouton, 2009.
Samy Bengio. The battle against the long tail. In Talk on Workshop on Big Data and Statistical Machine Learming, volume 1, 2015.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3: 137-1 2003.
Terra Blevins and Luke Zettlemoyer. Moving down the long tail of word sense disambiguation with iloss-informed biencoders. CoRR, abs/2005.02590, 2020. URL
wrg.vv.orrraaaoo abs/
2005. 02590
Howard Chen, Mengzhou Xia, and Danqi Chen. Non-parametric few-shot learning for word sense disambiguation. arXiv preprint N0w::2104.2677, 2021.
Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4))359-394, 1999.
N. Chomsky. The Logical Structure of Linguistic Theory. Springer US 1975, 1955. ISBN 9780306307607.
Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. Don't for- get the long tail! comprehensive analysis of morphological generalization in bilingual lexicon induction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP- IJCNLP), pp. 974-983, Hong Kong, China, November 2019. Association for Computational Lin- guistics. doi: Te.11865//1/DD1-... 1090. URL (tRLhhtts:://// /tattps://cllaee http:////caaaaahhaaaaooo wwwholooyo.oooooooooooooo
Daniel D'souza, Zach Nussbaum, Chirag Agarwal, and Sara Hooker. A tale of two long tails. arXiv preprint arXiv22107.13098, 2021.
Shiran Dudy and Steven Bedrick. Are some words worth more than others?, 2020.
Jerry A Fodor. The language of thought, volume 5. Harvard university press, 1975.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy-
sis. Cognition, 28(1 (8(1-2)33771 1988.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. bttp:///kkyLLon007ggiii 1o/ OpeeWeebTextCorpuu 2019.
Sharon Goldwater, Mark Johnson, and Thomas Griffiths. Interpolating between types and tokens by estimating power-law generators. In Y. Weiss, B. Schölkopf, and J. Platt (eds.), Advances in Neural Information Processing Systems, volume 18. MIT Press. 2006. URL SEAFF239acc322b11cc
23l.f Papere
Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. Producing power-law distribu- tions and damping word frequencies with two-stage language models. Journal of Machine Learning Research, 12(68):2335-22 2011. URL http://jmmx: czzg/papers/v1//
gol wwl.wwatrrla htmmm
10

---

Published as a conference paper at ICLR 2022
1. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(3/4):237-266 264, 1953. ISSN 00063444. URL http:/ www. jstor orrg stable/23333e
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrrik, An empirical investigation of global and local normalization for recurrent neural sequence models using continuous relaxation to beam search, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Grant Van Horn and Pietro Perona. The devil is the tails: Fine-grained classification in the wild. CoRR, 0is/1709001450 2017.
01450
F. Jelinek. Interpolated estimation of markov source parameters from sparse data. 1980.
S. Katz. Estimation of probabilities from sparse data for the language model component of speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3):400-401, 1987. doi: 10.1109/AASSP..99.11665
E. Khmaladze. The statistical analysis of a large number rare events. 1988.
R. Kneser and H. Ney. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 181-184 vol.1, 1995. doi: D0.109//CASSP..995..4999
Ilia Kulikov, Sean Welleck, and Kyunghyun Cho. Mode recovery neural autoregressive sequence modeling. CoRR. 100.05599, 2021. URL [BLLpt::://axxt: /ttp://://aaaaa..ooo//////////// wwt.:o/aaaag...ooooo
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings the the Eighteenth International Conference on Machine Learning, ICML '01, pp. 282-289, San Francisco, CA,
USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Heather Lent, Emanuele Bugliarello, Miryam de Lhoneux, Chen Qiu, and Anders Søgaard. language models for creoles, 2021.
R Thomas McCoy, Paul Smolensky. Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. arXiv preprint arXiv:2 11.0509, 2021.
Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity, 2021.
Gábor Melis, Tomás Kočiský, and Phil Blunsom. Mogrifier Istm. In International Confer- on Learning Representations, 2020. URL httpts://operreviiiiii http://ppwnneeeeeeeeeeeeeeeeeeee wwt.:/ene/fformmoco.mm
SJe5P6EYvS
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap: Good online learn- are good offline generalizers. CoRR, abs/2010.08 127, 2020. URL https::/aaxxi.......m
abss20100 08127
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNN.P-IJCNLPP) pp. 4227-4237, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v/D119. 1432. URL
brg/D19-143
Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.

---

Published as a conference paper at ICLR 2022
Isabel Papadimitriou and Dan Jurafsky. Pretraining non non- linguistic structure as tool for analyzing learning bias in language models. CoRR, abs/2004. 14601, 2020. URL abs abs/2004. 14601
Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and llya Sutskever, Language models
are unsupervised multitask learners. 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations, 2020.
Felix Stahlberg and Bill Byrne. On NMT search errors and model errors: Cat got your tongue?
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 2356-3362, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.186533v1/D19-111 1331.
Shuntaro Takahashi and Kumiko Tanaka-Ishii. Do neural nets learn statistical laws behind natural language? PloS one, 12(12)xe019336,, 2017.
Shuntaro Takahashi and Kumiko Tanaka-Ishi.. Evaluating Computational Language Models with Scaling Properties of Natural Language. Computational Linguistics, 45(3)-81-553,, 09 2019. ISSN 0891-2017. doi: 10.1 a URL. htttps::/doi.orgg..
a 00355
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008 2017.
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Consistency of recurrent language model with respect to incomplete decoding, 2020a.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihhod training. In International Conference on Learning Rep- resentations, 2020b.
net. www.:neetoormmmmidsdeeeeeeee
Jennifer C. White and Ryan Cotterell. Examining the inductive bias of neural languag models with artificial languages. CoRR, abs/2106.01044,, 2021. URL http::::/aaavv..orr//abbbs/////a 01044
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Associ- ation for Computational Linguistics. doi: 10. 18653v 200.0.0emml-.dmm........ URL https:
//aclantholoogyorz//2222
Frank Wood, Jan Gasthaus, Cédric Archambeau, Lancelot James, and Yee Whye Teh. The sequence memoizer. Communications of the ACM, 54(2):91-98, 2011.
Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language models uniformly better? comparing performance at the instance level, 2021.
Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2014.
George Kingsley Zipf. Human behavior and the principle of least effort. 1949.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 ADDITIONAL EXPERIMENTS
A.1.1 PRE-TRAINED GROUND-TRUTH MODEL
In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWeh corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine- tuned on 1.5M sequences from the Open WebTeT corpus. We sample from pt with softmax 1.00.
Analogously to the experiment in Section 4.1 we train a randomly-initializee GPT2-medium model on IM sequences sampled from the fine-tuned model. In the center of Figure 7 visualize mean model estimation error for 50, 000 test sequences as a function of true sequence probability. Sim- ilarly experiments in Section 4.4 find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences. Note that this LM obtains test perplexity of 67.97.
Moon eSteedoo CODP M0
10gu
Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estima- tion error by true sequence probability for sequences randomly sampled from E*
Finally, ensure that findings regarding the overestimation of ill-formed sequences hold, compute model estimation error on random sequences sampled from E' (see [A.1.4] for details on how these sequences constructed). Figure center visualizes function of target sequence probability. We find that PM overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using pretrained model.
model estimation
A.1.2 UNTEMPERED (T = 1.00) GROUND-TRUTH DISTRIBUTION
artificial language defined by ancestral sampling scheme with T = 0.85. Section 4.5 we saw that the underestimation findings held regardless T' To provide further evidence that these results hold for other values of similar experiments as in Section 4.3 with an artificial language Pz defined by an ancestral sampling scheme with T 1.00. Specifically,
In Sections 4.1 to we conducted all experiments
conduct
train GPT2-medium and GPT2-large 30M sequences sampled from PL. and we compute model estimation error on a set of withheld test sequences each training iteration.
total of
A.1.3 PRE-TRAINED MODEL ESTIMATION ERROR
In Section we explored how estimation error varies function of the amount of training data, finding that while increased data weakens estimation error, the underestimation behaviour persists. As an alternative way to explore how underestimation varies with increasing data, we fine-tune Hug- gingface's 's pre-trained GPT2-small, -medium and -large models on the set of IM sequences used in Section 4.1 Computing estimation error on set of unseen test sequences, we find, analogously to experiments models trained from scratch, that pre-trate pre-trained models underestimate the proba- bility the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9. Furthermore, in Figure 10 increase the amount
13

---

Published as a conference paper at ICLR 2022
Noa co
GPT2-medium
GPT2-large
Number sequences teen (millinss)
Number sequences training Imillonss
Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T 1.00. Main plots: Mean estimation eгror on test sequences seen in training. Inser plots: Relative change in mean estimation error of test sequences function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.
function of the number of sequences
of fine-tuning data substantially, and plot sequence estimation behaviour the end of each epoch for pre-trained GPT2-medium model. Once again, increased training data lessens but does climinate the underestimation behaviour.
GPT2-small
GPT2-medium
GPT2-large
-300
Estimattion error
-10
logpx(x)
-000
300
GPT2-small
-400
80
GPT2 medium
GPT2-large
-70
-500
-400
-300
-200
-100
6-500
-600
-300
-200
-100
8-500
-400
-300
-200
-100
-300
-20
-100
logP:(X)
logpL(x)
logPL(x)
(xgPL(x)
Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on IM sequences sampled from PL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model's estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.
Mean estimation emrrr
GPT2-medium sequences
training (millionss
fine-tuned 30M sequences sampled from PL- Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in fine- tuning. Insel plots: Relative change in estimation error of test sequences function of the number of sequences seen fine-tuning. In both cases, each line denotes estimation behaviour for 50th of the test sequences; darker lines represent less probable sequences.
Figure 10: Pre-trained
A.1.4 ALTERNATIVE PERTURBATIONS
In Section we study where the language model's underestimated probability mass went by computing model estimation error perturbed sequences. We obtain set of perturbed sequences by (i) sampling a sequence from pı and then (ii) recursively perturbing this sequence according to the perturbations provided in Table
14

---

Published as a conference paper at ICLR 2022
This method provides us with sequences which are inereasingly far away from high-probability zones under PL. However, does so with initial sequences sampled directly from PL. and as a result, produces strings which are edit-adjacent to the high-probability zones (under Pt.) in Σ"
To ensure that our results hold other low- probability subspaces, conduct analogous periment on sequences randomly sampled from Σ". We sample sequence from Σx by first sampling sequence length / from a Poisson distribution with
Mean estimation error
logpx(X) logp, [x)
10. Given this length, we sample tokens from E' and concatenate all tokens to form a sequence. We then score the sequence under both pr. and Pw,
20
and compute model estimation error. We use as ar- tificial language Pr GPT2-medium with and we use as language model pM GPT2-medium model trained on 30M sequences ancestrally sam- pled from PL-
0.85
overestimates p(x)
400
320
200
200
logp: (x)
Figure 11: Mean model estimation error by true sequence probability for sequences ran- domly sampled from Σ
Figure II visualizes mean estimation error as a function of the ground-truth probability of the quence. Similarly to all other perturbation exper- iments, do indeed find that PM overestimates these sequences, regardless of their true sequence probability.
A.1.5 ESTIMATION ERROR BY SEQUENCE LENGTH
Autoregressive neural language models decompose the joint distribution p(x) over sequences into series of conditional distributions p(x; x<z). Generating a sequence of length n, then, involves estimating n conditional distributions. Since the probability of a sequence is inversely correlated with length, our findings that estimation error is worse for rarer sentences may be explained by compounding errors.
GPT2-small
GPT2-medium
logpe(X) logp: (x)
-.0
-10
Mean observed error Expected crror
Sequence length
Sequence length
Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2- medium.
To test this claim, we ask whether the observed estimation error worse than would expected if it was due to error compounding. Specifically, in Figure [2] we plot the expected model esti- mation (black) and the mean observed error (blue) by sequence length. Expected estimation error for sequence length n is computed by multiplying the average token level estimation error by the sequence length, i.e., ne, where
1 D
Σ
log pMp (as (ie)) log pogrr.aa (ara x<s)
Observed estimation error is computed as the mean estimation error for test sentences of length Note that the shaded areas around this curve denote the 95% bootstrapped confidence intervals for this mean. As shown in Figure 12 observed estimation error for both GPT2-small and GPT2- medium is more severe than expected estimation that estimation error for longer (and typically rarer) sequences not solely due to error compound- ing.
increase sentence length. This suggests
15

---

Published as a conference paper at ICLR 2022
A.2 EMPIRICALLY MEASURING THE LNRE ZONE
In section 2 we formally defined the LNRE zone as the range of values of N for which there is non-zero probability of sampling novel event at the N + 1th draw. Here we introduce frequentist estimator for this probability. This in turn allows us to empirically verify if a sample exists in the LNRE zone.
A.2.1 ESTIMATING THE POTENTIAL PRODUCTIVITY
Suppose have set of N events D {a1,.... 1.,...., xx) drawn from some generative process Φ Given D, we aim to obtain an empirical estimate for the potential productivity Py: the amount of probability allocated to unseen events as a function of N. We can do so using the Good Gord-Turing estimate for the probability of an event given frequency (Good 1953).
Specifically, let fæ, D) be function which returns the frequency of the event E in D. Let Nu denote the number of types (unique events) in D for which f(x, D) m. Good-Ture -uriiT says that for large N, the probability of the event given that it has occurred with frequency in our sample of size is equal to
(m + 1)
Nm+1
P( f(x, D) m)
N
Nm.
To obtain an estimate for PN. we set m
0:
N1
PN NoP(x f(x, D) 0)
(4)
N
which states that the total amount of probability mass allocated unseen events is equal to the proportion of events which occurred only once (hapax legomena) in D. This quantity is known as the potential productivity of a linguistic process (Baayen 2009 2001 1994).
A.2.2 THE LNRE ZONE IN O
We apply this method to subset of OpenWebText, popular language modeling corpus. In Figure 13 we plot the empirical estimate of Pv as function of the sample size N for n-grams sampled from subset of Open WebText. Particularly for n-grams with 2 3, we find that there is significant probability of sampling a previously unseen event, even for N 10, 000, 000.
1.0
Unigrams
Distribution
Bigrams
0.6
Trigrams
0.4
4-grams
0.2
5-grams
Number of samples N (log scale)
Figure 13: The probability of sampling a novel item (potential productivity P) function of sample size N. Many distributions in natural language are characterized by a potentially unbounded amount of novelty.
A.3 MODEL. PERPLEXITY VALUES
In this section report relevant perplexity values for all models used. For each model, we report mean perplexity across sentences drawn from (i) the validation set generated by the artificial lan- guage they attempt to model and (ii) the OpenWebText corpus. While include the perplexity values for our models on sentences of English, this is not to claim that our ground-truth models are meant define distribution which closely resembles the distribution of English.
16

---

Published as a conference paper at ICLR 2022
Model
mean PP (val)
mean PP (eng)
LSTM
36.53
158.31
GPT2-small
26.66
129.84
GPT2-medium
25,42
132.56
GPT2-small (pretrained)
21.02
53.79
GPT2-medium (pretrained)
20.87
48.42
GPT2-large (pretrained)
20.90
47.30
LSTM (increased data)
35.01
149.02
GPT2-medium (increased data)
21.60
94.03
GPT2-large (increased data)
21.32
91.28
GPT2-medium (ground-truth model)
73.79
Table 4: PP validation generated by the artificial language the LM attempts model (val), and on real sentences sampled from OpenWeb WebText (eng).
Softmax T
mean PP (val)
mean PP (eng)
0.70
11.61
223.60
0.85
25,42
132.56
75.30
106.21
1.15
290.69
106.29
Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from WebTTet (eng) for models used Section
4.5
A.4 LANGUAGE SAMPLES
logPL(a) (x))
he complete east end player.
-26.399
a former harvard university graduate told that in weeks.. u.s. intelligence officials have begun to gather evidence that trump 's campaign colluded with russia to influence the election.
-61.846
the current study, we examined whether participants in the study performed more or less "cttive in weight loss -Irb- p = 0.05 -rrb-
-51.6566
you, the one that is the republican candidate.
who is taking over the senate and government democrat and who bipartisan democrat,
and you got to able to get done.
-92.703
since the 1970s, the city has been in the midst of landmark urban pride.
-11.9523
Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T 0.70.
17

---

Published as a conference paper at ICLR 2022
log
have doubt that 's big part any kind campaign machine " he told al jazeera. and 's very good idea work with claims in comics and film novels.
-54.9551
-59.7626
according to the new york times, susan macmahon, 54 and her husband, elizabeth bailey, were the vehicle with their son, aged between 12 and 19.
86.5217
anticipating experience
-21.7095
this is the reason 'd like to take advantage of these ideas attempt to transform my life.
-53.1399
Table 7: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T 0.85.
pr.())
beautiful bacon cookies
19.8038
this accident, despite most of its life
is dependent on one of the most precious bodies on this planet, which is comparable to that on mostrughenai mountains.
-142.4362
it not fair to say that we 're not in a position to permit same marriages, and all the same issues that presented regular basis
$8.9437
from video Irb- without editing -rrb- can see original images produced at the official website competitors, and event prices.
100.3791
consumer electronics have become perfect fit for bm 's ultra low cost turn
73.0217
Table 8: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 1.00.
log
zy brace gym fingerprint bom jerozo dell
-94.0307
shortly thereafter, will be buoyant the dynamo bust series starts
-90.5994
disjitor doom efeco became gothicrevolution manager ofby city library for marvel studios
143.6075
earlier this week chicago 's writer andrew o't text
american catholics interested readers the holiday kingsburyobee -Irb- expiration -rr- at for now atmotionweism race and bachelor whitney university.
register as 1852 jim banner of airbus beer and of course smiled at us in formula where he tells friends by name, finnish catholics: the encryptings robot type mothers.
-493.9313
middleware in germany
-27.3111
Table 9: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 1.15.
18


Reviews:
[{"review_id": "jbZABqhe34", "reviewer": "Reviewer_Msuw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.\n\n* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.\n\n* Comments and Questions:\n\n    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible \"natural\" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.\n    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?\n    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?\n\n", "summary_of_the_paper": "The paper conducts experiments to evaluate whether neural sequence models such as LSTMs and Transformers are able to correctly assess the probability of rare sentences, which collectively constitute a large probability mass in natural language productions (heavy-tail phenomenon). In order to do so, it performs experiments in a controlled synthetic environment where a first language model $P_L$ is trained on a corpus of natural sentences, and a second model $P_M$ is trained to emulate the first model. The authors observe that the second model systematically tends to underestimate the probability of rare $P_L$ sentences, the more so the rarer such sentences are, and show that some artificially corrupted sentences tend to receive higher probability from $P_M$ than from $P_L$, partly explaining where the missing probability mass over rare well-formed sentences went.\n\n", "summary_of_the_review": "A nice experimental paper addressing the difficulty of neural LMs to approximate the probability of rare events from an underlying \"teacher\" LM. The paper could be improved by more analysis and discussion of some of the results.\n\n*After authors' response*: thank you for your detailed answers to my questions and for your additions to the paper and to experiments. I aim raising my score and hope the paper will get accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fsoWrQkLwh", "reviewer": "Reviewer_V22X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Interesting background**. The connections between productivity, low-probability sequences, and the limitations of perplexity were clearly written and interesting.\n\n**Well-executed experiments and methodology**. The idea of using a language model as a ground-truth distribution was interesting and well-suited for the analysis done here. The experiments were easy to follow and the analysis was clear.\n\n**Interesting findings**. The main findings related to underestimation, comparison of training-set and test-set dynamics during training, and dependence on training data were interesting and not obvious. The finding that perturbed, unnatural sequence received unusually high probabilities (while test samples receive unusually low probabilities) [Figure 4] was especially interesting. \n\n**Clarity**. The paper was well-written and enjoyable to read. The authors articulated how their work fits in with related work, and the concepts (e.g. the LNRE zone), methodology, and results are clearly written. Well done.\n\n**Additional experiments**: A good addition would be measuring the impact of model size with models larger than the target distribution. What would GPT2-XL's or GPT-3's probability assignments look like on a target distribution from GPT2-medium?", "summary_of_the_paper": "This paper investigates how language models allocate their probability mass, with an emphasis on rare sequences that are part of the 'heavy tail' of the distribution of natural language sequences. The authors use a language model to define a target distribution with access to samples and ground-truth probabilities. A language model is trained on an empirical estimate of the target distribution, and the authors study the gap between the learned model's and target distribution's probability assignments. \n\nUsing this methodology, the authors uncover various interesting phenomena: the model systematically assigns lower probabilities than the target distribution, but assigns unusually high probabilities to unnatural, perturbed sequences (suggesting an explanation for where the probability mass moved to). The authors include several fine-grained analyses with additional interesting findings.", "summary_of_the_review": "This investigatory paper defines the problem that they are studying, develops a methodology for studying it, and clearly analyzes the results. The investigation yields interesting findings related to language models. This paper would make a great addition to ICLR - accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-wWh7RbMtgt", "reviewer": "Reviewer_AtM6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper overall is exceptionally well-written and clear and tables and plots are beautiful and clear---multiple times did I go \"ooh that's a nice way of laying it out.\" Kudos! The abstract tells the entire story and the paper substantiates the claims nicely making what is shown feel easy and obvious, which I consider a great feat.\n\nConcerning content, I am equally enthusiastic about the idea and evaluation and agree that this can be a very interesting piece of evidence that nicely connects to a number of past threads of NLP research (as laid out throughout and in the nice related work section, though I don't think I know enough related literature to truly safely say that the paper is as novel as it claims to be). That said, there are three major methodological concerns that I would like to discuss through with the authors before recommending acceptance:\n\n1. We do not know whether neural-LM-defined origin distributions are \"strange\"---or really, structurally different from actual human natural language distributions that we believe exist---in some systematic way that leads to the results in this paper. One potential intuition here is that a neural LM being a very imperfect model makes very erratic choices for what low-frequency/low-likelihood it assigns probability mass to---erratic choices that may be very hard to learn/emulate with the student LM that the authors train. Maybe real natural language distributions have more \"sensible tails\" that are actually not all that hard to learn and maybe not even underestimated. To be clear, this is a fundamental issue that the authors have no way of *resolving* easily under this paradigm and I think the paradigm is still a worthwhile idea! What I am asking is that this limitation is acknowledged and discussed with the same clarity that is afforded to the other content.\n\n2. Relatedly, currently there is no way to know whether the trained models are actually reasonably trained, overall well-working models or not: the paper is missing test ppl/bpc for both the origin model (on English test data) and for the student model (both on the original English data and on test data sampled from the origin model). The authors make a good point that these scores are not as informative as what this paper presents, but the reason these are crucial is so we know whether the effects the authors find are a property of well-trained models or whether these models (through subpar training or another issue) are just poor models that have little connection to the models we actually use in reality. A secondary idea to easy this concern might be to actually use a pretrained GPT model as the origin model to inspire trust that that at least is a good model---I'm not sure why this road wasn't chosen unless it truly did not matter and this \"home-made\" model performs just as well, in which case, again, I would like to see some quantitative substantiation of that claim.\n\n3. The paper is not really clear on how the sampling/teaching procedure and the tempering of the softmax interact: Footnote 2 claims pure (right?) ancestral sampling is used (as opposed to top-k/nucleus?), but then Footnote 4 says the softmaxes are all cooled to T=0.85, and yet Section 4.5 again implies that no tempering took place up to that point in the study. This is not only slightly confusing messaging, but I am worried that it may belie the promise of fair evaluation, especially because it isn't clear to me whether the tempering was accounted for in the calculation of probabilities as defined at the end of section 3.2. Specifically, if the probability of a sentence under the origin model does not take into account this local tempering, then it is no surprise that the student model learns to underestimate rare events because they just weren't sampled thanks to the locally cooled distribution! I would be somewhat reassured if the probability calculation in section 3.2 takes the tempering into account, but my understanding of the paper is that it currently fails to do so. As slight reassurance, Figure 5 does seem to tell us that even without any tempering (yes?) the described effect is visible, so I think this too should not be an issue that sinks the paper, but I do see it as a critical flaw the way it is right now.\n\nBeyond those three, I see some minor concerns that may be worth addressing:\n- \"autoregressive neural language models\" should also cite the Mikolov RNN paper, perhaps in place of one of the two GPT papers\n- not to Schmidhuber, but citing only Melis et al. (2020) for LSTMs looks a little odd (citing Hochreiter & Schmidhuber or Sundermeyer et al. may be less surprising though I understand the desire to cite the *actual* model and training specification used and agree it should be present)\n- concerning citations, Shiran Dudy & Steven Bedrick's work may be nice to also connect to (despite its relatively low visibility)\n- it may be worth pointing out that the perturbations in section 4.4 may individually also make sentences *more* grammatical and likely and that is just a relatively rare outcome (I don't think much more than that needs to be said on that)\n- I did not quite see what was \"[p]erhaps unsurprisin[g]\" in Section 4.5, but that's a minor nitpick \n- Appendix A.1 measures events, okay... but are those words (as the hapax legomena theory usually assumes) or are they sentences (as I think this experiment does)? If they are sentences, we should not be surprised to see very high probabilities of seeing novel events, after all sentences are rarely the same (excluding formulaic language and \"Resumption of the session\" style sentences). I also would have liked to see some Poisson distributions or other distributions that were more interesting than uniform n-sided dice in Figure 6, but that is more wish than necessity.", "summary_of_the_paper": "The paper proposes to examine the probability mass that a language model trained as usual places on frequent/likely and rare/unlikely sentences by learning from a distribution that is another neural language model itself so instance-level probabilities/NLLs of sentences can be compared. It turns out that all models in all tested circumstances *underestimate* the probabilities of rare events---and an ablation study shows that much of that missing mass can be found in sequences that are also rare and unlikely, but that weren't likely under the origin model.", "summary_of_the_review": "Before author response:\n\nMy comparatively low score is a consequence of my concerns and very much given as a temporary score until we can discuss the three major concerns I have---if they are addressed satisfactorily, I anticipate raising my score significantly to champion the paper if needed as I found this paper a joy to read and thought-provoking in a good way (even if that lead to criticism).\n\n\nAfter author response:\n\nThank you for all the clarifications and edits, they are greatly appreciated. I would strongly like to see this paper accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a very interesting study of using an artificial language (generated using a specific algorithm via a transformer model) and training SOTA transformer and LSTM language models on that language;  the authors show that these LMs underestimate the probability of sequences from this language and overestimate the probability of ill formed sentences, among other observations.  This is a very interesting study that captures the behavior of recent LMs.  All reviewers are supportive of accepting this paper and it is good to see the engagement between reviewers and authors of this paper.
Decision: Accept (Poster)