Revisiting Entropy Rate Constancy in Text
Authors: Vivek Verma
Nicholas Tomlin
Dan Klein
Venue: EMNLP 2023 Findings (2023)
Keywords: Uniform Information Density
Entropy Rate
Large Language Models
Linguistic Theories

Abstract: The uniform information density (UID) hypothesis states that humans tend to distribute information roughly evenly across an utterance or discourse. Early evidence in support of the UID hypothesis came from Genzel and Charniak (2002), which proposed an entropy rate constancy principle based on the probability of English text under $n$-gram language models. We re-evaluate the claims of Genzel and Charniak (2002) with neural language models, failing to find clear evidence in support of entropy rate constancy. We conduct a range of experiments across datasets, model sizes, and languages and discuss implications for the uniform information density hypothesis and linguistic theories of efficient communication more broadly.

Revisiting Entropy Rate Constancy in Text
Vivek Verma"
Nicholas Tomlin'
Dan Klein
Computer Science Division, University of Californa,, Berkeley
(vivekverma, nicholas_tomlin, klein)@berkeley. edu
Abstract
Entropy in Penn Treebank
10
GPT-2 (1.5B)
The uniform information density (UID) hypoth- esis states that humans tend to distribute infor- mation roughly evenly across an utterance or discourse. Early evidence in support of the UID hypothesis came from Genzel and Charniak (2002), which proposed an entropy rate con- stancy principle based on the probability of En- glish text under n-gramm language models. We re-evaluate the claims of Genzel and Charniak (2002) with neural language models, failing to find clear evidence in support of entropy rate constancy. We conduct range of experiments across datasets, model sizes, and languages and discuss implications for the uniform informa- tion density hypothesis and linguistic theories of efficient communication more broadly.
Thigram
8-
Mean HOX)
7-
3-
100
200
300
400
500
Position in Dacument
1 Introduction
Figure 1: Entropy rate of the Penn Treebank under smoothed trigram model and a GPT-2 XL. model (Rad- ford al., 2019), averaged across documents per word position. Genzel and Charniak (2002) showed that en- tropy rate increased under n-gram models and predicted that it would remain constant in models which can con- dition on long-range context. We replicate the former result but find clear evidence supporting the latter.
Linguistic functionalists have often claimed that language is optimized for efficient communication (e.g.. Gibson et al., 2019). One common argument supporting theories of efficient communication is that humans communicate at or near channel ca- pacity (Shannon, 1948); functionalists have argued that because interlocutors can only produce and comprehend fixed amount of linguistic informa- tion per unit of time, and because speakers strate- gically arrange their utterances to convey much information as possible, language should typically have uniform surprisal over time (Aylett, 1999; Bell et al. 2003; Aylett and Turk, 2004; Jaeger and Levy, 2007; Jaeger, 2010). Early evidence for this claim comes from Genzel and Charniak (2002), which used n-gram language models to argue that En- glish documents exhibit entropy rate constancy. In this paper, we identify a limitation in Genzel and Charniak (2002)'s analysis and directly measure its hypothesis using neural language models. We then analyze results across a variety of datasets, lan- guages, and models and discuss their implications for theories efficient communication.
Background
2.1 Efficient Communication
Claims that language optimized for efficient com- munication originated with diachronic arguments about the evolution of language. Zipf (1949) ob- served that frequent words are usually shorter, lead- ing to claims that word lengths are optimized for efficient communication (Piantadosi et al., 201 I). Other work has argued that natural language lex- icons efficiently carve up semantic space, mak- ing reference to the cross-linguistic organization of color terms (Gibson et al., 2017; Zaslavsky et al., 2018) and kinship terms (Kemp et al., 2018). Still other work has focused on syntactic effi- ciency, suggesting that statistical tendencies such

---

CC News
NYT
Al-Ittihad
6.5
7.5
7.0
Mean H(X)
6.0
4.0
Trigram
5.0
Trigram
5.5
Trigram
GPT-2 XL (1.58)
4.5
GPT-2 xL (1.58)
GPT-2 XL (1.58)
4.5
4.0
4.0
3.5-
3.0
100
20D
500
100
200
300
400
Position in Document
Position in Document
Position in Dacument
Figure 2: Entropy rate of the Common Crawl News Dataset, the NYT Annotated Corpus and the Al-Ittihad subset of the Arabic Billion Words Corpus under smoothed trigram model and a GPT-2 model (1.5B) (Radford et al., 2019), averaged across documents each word position. We observe roughly increasing trend for the trigram model across all three datasets, and variety of trends for the GPT-2 models.
as dependency-length minimization (Futrell et al., 2015), adjective ordering preferences (Hahn et al., 2018), and Greenbergian word-order correlations (Hawkins, 2009; Hahn et al., 2020) may have de- veloped because they improve communicative effi-
2003; Meister et al., 2021). Crucially, research of this latter type must operationalize the uniform information density hypothesis order to test its predictions; in the following section, we discuss Genzel and Charniak (2002)'s approach, which operationalized the uniform information density hypothesis at the document level.
ciency. Collectively, these works demonstrate that fixed elements of linguistic structure, such as the lexicon and syntactic rules, often lead to more effi cient communication than unattested alternatives.
2.3 Revisiting Entropy Rate Constancy
Genzel and Charniak (2002) operationalized the notion of information density by claiming that the average per-word entropy of the n-th sentence English documents does not depend on n. In other words, they claimed that entropy remains roughly constant over the course of documents. Genzel and Charniak (2002) referred to this hypothesis an entropy rate constancy principle; use the same terminology for consistency, but we note that it differs from the standard meaning of entropy rate in information theory, as discussed in Section 5. In this section, we briefly restate the argument for entropy rate constancy presented in Genzel and Charniak (2002) and refer the reader to the original paper for more details. Formally, let Xo...., X; be random variables representing words, and let H (Y;) H(X; Ci, Ia) denote the conditional entropy of word X; given its long-distance con- text C, Xo. X;-n and local n-gram con- text L; = X; nil...., X,, Then by the defini- tion of mutual information:
2.2 Uniform Information Density
In contrast, work in psycholinguistics has high- lighted the real-time decisions that speakers make in order optimize communicative efficiency. Early work in surprisal theory (Hale, 2001) demon- strated that the contextual predictability of words determines their processing difficulty (Levy, 2008; Brouwer et al., 2010). This finding led to cogni- tive models of efficient communication such as the smooth signal redundancy hypothesis (Aylett and Turk, 2004) and the uniform information density hypothesis (Jaeger and Levy, 2007; Jaeger, 2010), which states that given the choice between two otherwise identical utterances, speakers tend to choose the one with more uniform distribution of in- formation content. One line of evidence for the uni- form information density hypothesis comes from the analysis of linguistic phenomena such as leni- tion (Aylett and Turk, 2004), syntactic reduction (Jacger, 2010), and word omission (Ast and Dem- berg, 2015), which more likely to appear in pre- dictable contexts. Another line of work uses data- driven analysis of corpora to determine whether or not they exhibit properties associated with uniform information density (Genzel and Charniak, 2002,
H H(x, C.,L) H(,, L) - I(X;;
(1)
(2)
Next, assume that the entropy HYY)) remains con- stant. By the above equations, mutual information

---

CC News
NYT
Al-Ittihad
1750
3500
4000
Number of Dacuments
1500
3000
3000
1000
2000
750
1500
1000
1000
500
500
250
200-300
L300-260
353-506
530-3300
20.00
<29>
250000
10003500
Number of Words in Document
Number of Words in Document
Number of Words in Document
Figure 3: Distribution of document sizes for the CC News, NYT and Al-Ittihad test splits. Throughout this paper. we present word probabilities for the first 500 tokens of each article due to the lack of longer articles in each of these datasets. We provide additional results on longer documents in Appendix D.
I(X;; L,) should increase as contexts become longer, which means that the entropy given only local contexts (X; Lj) must also increase over the course of the document.
NYT Annotated Corpus² (Sandhaus, 2008), the Common Crawl News Dataset (Hamborg et al., 2017), and the Al-Ittihad subset of the Arabic Bil- lion Word Corpus* (EI-Khair, 2016). We present dataset statistics in Figure 3 and describe each of these datasets, well filtering criteria, in the following subsections.
Because Genzel and Charniak (2002) could not directly estimate (Y;) without access to models that effectivvly integrate long-distance context, they instead used n--ramm models to demonstrate that H(;; Li) increases over the course of docu- ments. We replicate their results in Appendix 6.1 but highlight shortcoming of their argument: non- decreasing H (X; L;) is a necessary but not suf- ficient condition for entropy rate constancy, and HY,) could increase or decrease depending on the relative value of [(X;; C, L.). In other words, Genzel and Charniak (2002) confirmed predic- tion of entropy rate constancy but did not provide direct evidence for the hypothesis itself. Because modern neural language models are capable of inte- grating long-distance contexts, can now directly approximate H (Y;) to shine further light on these results. shown Section 6.2, our results do not provide clear evidence for constancy, but rather for sharp decline at the beginnings of documents, followed by constant or slightly declining trend.
preprocessing and
3.1 The New York Times Annotated Corpus
The New York Times Annotated Corpus features over 1.8 million articles written and published by The Times from 1987 to 2007. We randomly sam- ple 120K documents from this corpus and construct data split consisting of 100K train articles, 10K validation articles, and 10K test articles. We con- dition on the title of each article when computing word probabilities and provide additional discus- sion this point in Section 6.5 and Appendix A.
3.2 Common Crawl News
We include a subset of the Common Crawl News Dataset due to its chronological diversity. In par- ticular,
the majority of experiments GPT-2; because articles in the NYT Annotated Cor- pus were published between 1987 and 2007, they may appear GPT-2's training data. To address this concern, we filtered the Common Crawl News Corpus to only include articles which were written after GPT-2 was trained. In total, there are 270996 news articles written after 2018, of which we ran- domly sample 100K training documents, 10K vali- dation documents, and 10K test documents.
3 Datasets
Genzel and Charniak (2002) ran experiments on the Penn Treebank (PTB; Marcus al., 1993). which we replicate in Section 6.1 for complete- ness. However, we run our primary experiments on different datasets, in order to obtain additional data with more chronological diversity, as well non-English data. We run experiments the
hhttps://catalog. 1dc. upenn. edu/LDC2808119 hhttps://huggingfacee httpa//gaa.e.eeeeeettttteeeeeeee
"https://catalog: 1dc. www.n.eeeee.. wtt.:uueeeeeeddDeeeee.... edu/LDC99T42

---

3.3 Al-Ittihad (Arabic Billion Words)
Effect of Model Size in NYT
Lastly, we leverage the Al-Ittihad subset of the Ara- bic Billion Words Corpus (EI-Khair, 20, as a means of comparing trends across languages. Al- though the corpus contains over three million ar- ticles, we employ one subset due to the differing nature of dialects, which would complicate compar- isons. In total, we include 8551 training documents, IK validation documents, and 2K test documents.
6.5
GPT-2 Small (124M)
GPT-2 Medium (345M)
GPT-2 Large (774M)
6.0
GPT-2 XL (1.58)
5,5
Mean HeXn
HXX)
5.0
4 Models
In recent years, neural language models, in par- ticular, transformer-based models (Vaswani et al., 2017) have been shown to greatly outperform n- gram models, due to their ability to scale and model long-distance dependencies. In this work, com- pare the entropy rate of English text under the transformer-based GPT-2 model (Radford et al., 2019) to that of n-gram models.
3.5
3.0
100
200
400
500
Position in Document
Figure 4: Entropy of NYT under four GPT-2 model sizes (124M, 345M 774M 1.5B). We note lower en- tropy values as model size increases but observe con- sistent decline in surprisal values across all model sizes.
4.1 Trigram Model
For each of the three datasets, we train a trigram model on their respective training splits. To provide a fair comparison to prior work, we aim to repro- duce the model in Genzel and Charniak (2002) as closely possible. However, because Genzel and Charniak (2002) did not provide exact details about its approach n-gram modeling, we use parame- ters matching those described in the followup paper Genzel and Charniak (2003). In particular, we use a smoothed trigram model:
4.2 Large Language Models
In addition to training R-gram models, we also fine-tune GPT-2 on both the NYT and CC News datasets, with one epoch the train split and batch size of 8 1024-token length inputs. For fine- tuning on the Arabie Billion Words Corpus, we em- ploy AraGPT-2 Mega (1.5B) (Antoun et al., 2021). We report results across all model sizes ( (124M, 345M, 774M, 1.5B), both with and without fine- tuning, in Section 6. We also report fine-tuning and inference times in Table 1 in the Appendix.
P(x;/ (11...4-11 ~ P(æ: 84-2, 2i-1)
(3)
= Pi, 24-21
(4)
Experiments
+ P(: [T:
34 1)
(5) For each dataset and model, compute the per- token probability of each document in the dataset:
+ (1- - P(xs)
where n; corresponds to the ith word in document, \1 0.5 and A2 0.3 smoothing coefficients matching those Genzel and Charniak (2003), and P a maximum likelihood estimation via counts:
Po(a; æ1,..... (..-1)
(8)
where denotes model parameters. We compute token probabilities using the maximum context length available to each model. Because our tri- gram models are trained on words and the neural models are trained on subwords, we over the log probabilities of subword tokens to obtain word probabilities from neural models (Mielke, 2019):
C(I1...Zs)
((a.(1)-20111) C(
where C(r;..3))) is the number of times Fy... ap- pears in the training data. We train trigram models the word level on closed vocabulary, dis- cussed in Appendix B. As a result, note that exact probabilities may not be directly comparable to those computed by GPT-2 models, but the gen- eral trends between models are still comparable.
end(k)
log Po(w)) =
log Pa Pa(x;)
(9)
i=start(k)
for a word wi consisting of subword tokens . Fend(())) In contrast to Genzel and

---

CC News
NYT
Al-Ittihad
Withaut Fine-Tuning
Without Fine-Tuning With Fine-Tuning
Without Fine-Tuning
With Fine-Tuning
With Fine-Tuning
Mean HDO
8-
3-
0
100 200
300
400
500
0
100
200
300
400
500
100
200
300
400
500
Position Document
Position Document
Position in Document
Figure 5: Entropy rate of CC News, NYT and Al-Tutihad on a fine-tuned GPT-2 XI. (1.5B) compared to a non-fine- tuned model. We note that entropy values sharply decline and have lower values on the fine-tuned models, most likely due to domain adaptation. The difference between the two models is largest at the beginnings of documents.
Charniak (2002), we present results the word- level rather than at the sentence-level to provide additional granularity and insight into trends at the beginnings of documents; additionally, this approach avoids confounding effects of sentence length which were noted in previous work (Keller, 2004; Xu and Reitter, 2018).
strongly negative trend, because word probability under neural models tend to decline sharply at the beginnings of documents, as discussed in the fol- lowing section. We provide additional discussion of this issue, along with results from the Mann- Kendall significance test, in Appendix E.
6 Empirical Results
We then over each position in the dataset to compute the average per-word probability across documents the dataset, each word position
6.1 Replicating Genzel and Charniak (2002)
We first replicate the results of Genzel and Char- niak (2002) and compare them to entropy rates achieved using GPT-2 XL (1.5B). As shown in Fig- ure entropy rates under trigram model tend to increase, as reported in Genzel and Charniak (2002). contrast, average word surprisals under GPT-2 XL sharply deeline at the beginming of docu- ments before leveling off. We note that these values are quite noisy, due to the test split containing only 400 documents. In the following subsections, we run similar analyses much larger corpora.
f(1)
Σ WEW
log Po(w:)
(10)
[W
where w denotes article in a corpus W. Follow- ing Genzel and Charniak (2002), we refer to the slope trend of fz:) the entropy rate.5
In this paper, focus on qualitative analysis of entropy rate. We avoid quantitative measures like correlation coefficients, as used in Giulianelli and Fernández (2021), because they are strongly dependent on the lengths of sampled documents. In particular: for sufficiently long documents, entropy rate must either increase approach constant value, because word probabilities cannot be be- low and must level off asymptotically. Mean- while, for very short documents, we would observe
6.2 Measuring Entropy Rate with GPT-2
We also replicate the results of Genzel and Char- niak (2002) on significantly larger corpora, show- ing that trigram models exhibit increasing entropy rates on both the CC News and NYT datasets, as well as the Al-Ittihad subset of the Arabic Bil- lion Words Corpus. We then compute entropy rate using fine-tuned GPT-2 models condition- ing on the entire document history and observe various decreasing and non-monotonic trends, shown in Figure 2. In particular, average per- word surprisal as measured by GPT-2 sharply de- clines at the beginning of documents in all cor- pora, and then either sharply rises before becoming roughly constant (CC. News), asymptotically de-
'In contrast, given stochastic process {X,},, Cover and Thomas (2012) defines the entropy rate H[X) the time density average entropy given by each random variable in the process, written
1 H(X,, X2, .., X..)
H(X) lim 10 n
(LI)
While the standard definition of entropy rate refers a constant, our usage refers a more general trend over the course of doc- uments. Further, rather than computing the limit as n - co, we estimate the average observed word probabilities for each E [1...., n. where n is the length of the document.

---

clines (NYT), or slowly increases before beginning decrease again (Al-Itihad). This finding sug- gests that ;X;; C₁ L,) and (X; Lj) do not necessarily increase at similar rates and is largely consistent with recent results about how neural lan- guage models integrate context (Khandelwal et al., 2018; O'Connor and Andreas, 2021). Most cru- cially, these findings do not provide clear evidence for entropy rate constancy as predicted by Genzel and Charniak (2002).
Effect of Title in CC News
Not conditioned on title
Conditioned on title
Mean HIX)
6.3 Effect of Model Size
We also fine-tune GPT-2 base (124M), medium (345M), and large (774M) models on the NYT dataset and observe a similar decreasing trend across all model sizes, as shown in Figure 4. As expected, across both datasets, larger models con- sistently cxhibit lower perplexity. We predict that future large language models will continue to im- prove integrating long-distance context and pro- duce similar trends in entropy rate and provide preliminary results on GPT-3 in Appendix C.
100
200
400
500
Position in Document
fine-tuned GPTT-2 XL (1.5B) models, one fine-tuned with the title and one fine-tuned without, omit entropy values for the title in this plot, but condition the model on the title.
Figure 6: Entropy rate of CC News
in Figure 2 can be attributed conditioning the title. We hypothesize that since news titles only provide a limited amount of information, condi- tioning on them does not make the document sig- nificantly easier to predict. Future work might take an information-struucural approach and investigate entropy rate values associated with different parts of articles, such lede paragraphs or conclusions.
6.4 Effect of Fine-tuning
Finally, we also analyze the effect of fine-tuning. We observe that fine-tuning generally results in lower surprisal values, especially at the beginning of documents, shown in Figure 5. As result, entropy rate tends flatten out faster when com- puted with non-fine-tuned models. We hypothesize that this finding may result from domain adapta- tion: during the fine-tuning process, models may learn to attribute most of their probability mass to in-domain vocabulary and conventions. How- ever, models without fine-tuning must determine the domain from the context alone, which may be especially difficult at the beginnings of documents.
Discussion
We clarify that the findings presented in this paper do not necessarily invalidate the uniform informa- tion density hypothesis. Although entropy rate as measured by neural language models may decline over the course of documents, cognitive measures of surprisal might not decline. For example, the recently proposed lossy-context surprisal model of Futrell et al. (2020) posits that surprisal is com- puted with respect to an incomplete representation of context, whereas neural language models may make predictions based on lossless representations of their context windows. This perspective is also consistent with recent findings that the base GPT-2 model (124M) outperforms larger GPT-2 and GPT- models as predictor of human reading time (Shain et al., 2022). In particular, these results point to a discrepancy between surprisal values under a Bayes-optimal language model and cognitively- relevant measures of surprisal. Despite still being
6.5 Effect of Titles
this section, we demonstrate how these results are sensitive to pre-preocessing standards. We fine- tune two GPT-2 XL (1.5B) models on CC News, one by feeding in just the document, and one with the title followed by a new-line and then the rest of the document. We compute word probabilities and only plot those corresponding to the main body of each article. Unsurprssigly, the initial word probabilities are significantly lower when condi- tioning on the title. However, after 100 words they are only marginally better. We note that this com- parison shows that the slight increase in entropy values towards the beginning of the document seen

---

worse than humans at a variety of language-related tasks, we consider it likely that large language mod- els outperform humans at the task of raw language modeling, at least as measured by perplexity. As a result, weaker language models may be better correlated with cogmitive measures of surprisal.
Recent work has also sought to connect cogni- tive theories of efficient communication with tech- niques natural language processing; in particu- lar, operationalizations of the uniform information density hypothesis have been connected to natural language decoding (Meister et al., 2020, 2022) and used as regularizers for language model training (Wei et al., 2021). We hope that an improved un- derstanding of entropy rate constancy will inform such applications the future.
Whether or not our results contradict the entropy rate constancy principle is a matter of interpreta- tion. Genzel and Charniak (2002) would predict that neural language models, which are capable of integrating long-distance context, would exhibit roughly constant entropy rate over the course of documents. Under certain conditions, however, en- tropy rate as computed by neural language models seems to decline even exhibit non-monotonic behavior. While this behavior is mostly isolated to the beginnings of documents, it is impossible for entropy rate to decline forever, because word prob- abilities cannot be less than zero. At the very least, we can conclude that our analyses do not provide clear support in favor of the entropy rate constancy principle proposed by Genzel and Charniak (2002).
Conclusion
In this work, we computed entropy rate using tri- gram models and GPT-2, failing to find clear evi- dence in support of Genzel and Charniak (2002)'s claim of entropy rate constancy in text. We provide results across various model sizes, with and without fine-tuning, and across several datasets, including the Arabic Billion Words Corpus. Our work also provides one of the only analyses of entropy rate constancy in a language besides English, although see Genzel and Charniak (2003) for results in Rus- sian and Spanish. We encourage future work to further investigate the cross-linguisto validity of the uniform information density hypothesis.
8 Related Work
Most relevant to this work are Giulianelli et al. (2021) and Giulianelli and Fernández (2021), which explore the role of entropy rate constancy in dialogue datasets such as the Spoken British Na- tional Corpus (McEnery et al., 2017), the HCRC Map Task (Thompson al., 1993). and PhotoBook (Haber et al., 2019). Giulianelli and Fernández (2021) follows a similar methodological procedure and computes entropy rate using fine-tuned GPT-2 models, claiming to support the entropy rate con- stancy hypothesis in the Penn Treebank but not in the dialogue datasets. In contrast, we focus on significantly larger news datasets, which are also more similar to the Penn Treebank data used in Genzel and Charniak (2002), and compute results across a wider range of model sizes. Using larger datasets enables additional fine-tuning and reduces variance in the results; further, our focus on word- level surprisal provides additional granularity at the beginnings of documents, where entropy rate is least constant. Finally, we note that perplexity is extremely sensitive metric (cf. Appendix 6.5), and large variation in results may be attributable small differences in data. In particular, we do not expect the trends we observe in news articles to always transfer to other domains, such as spoken dialogue in Giulianelli and Fernández (2021).
Limitations
One limitation of this work is that since the train- ing data for GPT-2 was not released, is unknown whether the contents of the NYT Annotated Cor- pus exist in the pre-training data. We circumvent this issue by also evaluating entropy rates on doc- uments from the Common Crawl News dataset, filtered to those published after 2018. However, it is possibility that time generalization may com- plicate the measurement of entropy rate (Lazaridou et al., 2021). Another limitation of our analysis is the sensitivity of word surprisal small changes in text. As shown in Appendix 6.5, results can significantly change when titles are omitted from fine-tuning and inference. Handling of punctuation and other text preprocessing decisions also plays large role in the computation of word probabilities, and consequently these decisions may affect any resultant conclusions about entropy rate constancy. Lastly, although the AraGPT-2 training data does contain the Arabic Billion Words Corpus (Antoun et al., 2021), we utlize due to the unavailability of Arabic-based LLMs and Arabic datasets.

---

Acknowledgments
Richard Futrell, Kyle Mahowald, and Edward Gibson, 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings the Na- rional Academy of Sciences, 112(33):13366-100
Nicholas Tomlin is supported by NSF Graduate Research Fellowship, as well grants from the DARPA LwLL and XAI programs. We are grateful to Uriel Cohen Priva, Roma Patel, the members of the Berkeley NLP Group, as well as anonymous re- viewers for feedback on earlier drafts of this paper.
Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th An- nual Meeting of the Association for Computational Linguistics, pages 199-206, Philadelphia, Pennsylva- nia, USA. Association for Computational Linguistics.
Dmitriy Genzel and Eugene Charniak. 2003. Variation of entropy and parse trees of sentences as a func- tion of the sentence number. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 65-72.
References
Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic lan- guage generation. In Proceedings of the Sixth Aru- bic Natural Language Processing Workshop, pages 196-207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Edward Gibson, Richard Futrell, Julian Jara- Ettinger. Kyle Mahowald, Leon Bergen, Sivalogeswaran Rat- nasingam, Mitchell Gibson. Steven T Piantadosi, and Bevil R Conway. 2017. Color naming across lan- guages reflects color Proceedings of the Na- tional Academy of Sciences, 114(40):10785--072
Fatemeh Torabi Asr and Vera Demberg. 2015. Uniform surprisal at the level of discourse relations: Nega- tion markers and discourse connective omission. In Proceedings of the 11hh international conference on computational semantics, pages 118-128.
Edward Gibson, Richard Futrell, Steven P Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in cognitive sciences, 23(5):889- 407.
Matthew Aylett and Alice Turk. 2004, The smooth signal redundancy hypothesis: A functional ex- planation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31-56. PMID: 15298329.
Mario Giulianelli and Raquel Fernández. 2021. Analysing human strategies of information transmis- sion as a function discourse context. In Proceed- ings of the 25th Conference on Computational Nat- ural Language Learning, pages 647-660, Online. Association for Computational Linguisties.
MP Aylett. 1999. Stochastic suprasegmentals: Relation- ships between redundancy, prosodic structure and syllabic duration. Proceedings of ICPhS-99, San Francisco.
Mario Giulianelli, Arabella Sinclair, and Raquel Fer- nández. 2021. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8271-8283 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier. Cyn- thia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and ut- terance position word form variation in english conversation. The Journal of the acousticul society of America, 113(2)1001-1024,
Harm Brouwer, Hartmut Fitz, and John Hoeks. 2010. Modeling the noun phrase versus sentence coordi- nation ambiguity in Dutch: Evidence from surprisal theory. In Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Lin Linguistics, pages 72-80, Uppsala, Sweden. Association for Com- putational Linguistics.
Janosch Haber, Tim Baumgärtner, Fee Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook dataset: Building common ground through Soually-grounder dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895-1910 1910. Flo- rence, Italy. Association for Computational Linguis- tics.
Thomas M Cover and Joy A Thomas. 2012. Elements of information theory. John Wiley & Sons.
Michael Hahn, Judith Degen, Noah D Goodman, Dan Jurafsky, and Richard Futrell. 2018. An information- theoretic explanation of adjective ordering prefer- ences. In CogSci.
Ibrahim Abu El- Eh-ir. 2016. 1.5 billion words arabic corpus. arXiv preprint 16X 1.04033.
Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information- theoretic model of memory effects in sentence pro- cessing. Cognitive science, 44(3) 12814.
Michael Hahn, Dan Jurafsky, and Richard Futrell. 2020. Universals of word order reflect optimization of gram- mars for efficient communication. Proceedings of the National Academy of Sciences, 11765)23347-2353

---

John Hale. 2001. probabilistic Earley parser as psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguisties.
Tony McEnery, Robbie Love, and Vaclav Brezina. 2017. Introduction: Compiling and analysing the spoken british national corpus 2014, International Journal of Corpus Linguistics, 22(3):3
Felix Hamborg. Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A generic news crawler and extractor. In Proceedings of the 15th In- ternational Symposium of Information Science, pages 218-223.
Clara Meister, Ryan Cotterell, and Tim Vicira. 2020. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, Online. Association for Computa- tional Linguistics.
John A Hawkins. 2009. Language universals and the performanco-rammmn correspondence hypothesis. Language universals, pages 54-78.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisit- ing the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963- 980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
T Florian Jaeger. 2010.
Redundancy and reduction: Speakers manage syntactic information density. Cog- nitive psychology, 61(1):23-62.
T Florian Jaeger and Roger Levy. 2007. Speakers opti- mize information density through syntactic reduction. In Advances in neural information processing sys- tems, pages 849-856.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Typical decoding for natural lan- guage generation. arXiv preprint arXiv:2202 00666.
Frank Keller. 2004. The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 317-324, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Sabrina J. Mielke. 2019. Can you compare perplexity across different segmentations?
Joe O' Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 851 1-64, Online. Association for Computational Linguistics.
Charles Kemp, Yang Xu, and Terry Regier. 2018. Se- mantic typology and efficient communication. An- nual Review of Linguistics, 4:109-128.
Maurice George Kendall. 1948. Rank correlation meth- ods.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural lan- guage models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 284-294, Melbourne, Australia, Association for Computational Linguistics.
Steven T Piantadosi. Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient com- munication. Proceedings of the National Academy of Sciences, 108(9))3526-3) 3529.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Ko- cisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young. and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems, volume 34, pages 29348-29361 Curran Associates, Inc.
Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia. 6 6(12):26752.
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Philip Levy. 2022. Large-scale evi- dence for logarithmic effects word predictability on reading time.
Roger Levy. 2008. Expectation- based syntactic compre- hension. Cognition, 106(3): 1177
Claude Elwood Shannon. 1948, A mathematical theory of communication. The Bell system technical journal, 27(3):39-4233
Henry B Mann. 1945. Nonparametric tests against trend, Econometrica: Journal of the Econometric Society, pages 245-259.
Henry S. Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty Sneddou, Alison Newlands. and Cathy Sotillo. 1993. The HCRC map task corpus: Natural dialogue for speech recognition. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21 21-24, 1993.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Mareinkiewicz. 1993. Building large annotated corpus of english: The penn treebank.

---

Noam Vaswani, Ashish aand Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Atten- tion is all you need.
Jason Wei, Clara Meister, and Ryan Cotterell. 2021. A cognitive regularizer for language modeling. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5191- 5202, Online. Association for Computational Lin- guistics.
Yang Xu and David Reitter. 2018. sity converges in dialogue: Towards information- theoretic model. Cognition, 170:147-163.
Information den-
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naf- tali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the Na- tional Academy of Sciences, 115(31):73377794.
George K. Zipf. 1949. Human behavior and the princi- ple of least effort. Addison- Addison-Wesley Press.

---

A Preprocessing and Fine-tuning Details
NYT GPT-3 Davinci (175B) vs GPT-2 (1.5B)
In this section, we describe preprocessing and fine- tuning details for each of the three datasets. Fine- tuning GPT-2 across all datasets was performed with one epoch and batch size of 8 1024-token length inputs. We outline the fine-tuning and infer- ence times in Table 1. All experiments were run on Quadro RTX 6000 and Quadro RTX 8000 GPUs.
GPT-2 (1.5B)
GPT-3 Davinci (1758)
Mcan H(X)
A.1 NYT
We randomly sample 120K documents from the NYT Annotated Corpus and construct a training set consisting of 100K documents, a validation set consisting of 10K documents, and test split consisting of 10K documents. We feed in each document with the title as the first line, followed by a newline (\n) token, and the body of the article afterwards. For non-finetuned runs, we replace the newline token with a colon (":").
100
200
300
400
500
Position in Document
Figure 7: Entropy rate of NYT under GPT-3 (davinci) and GPT-2 XL (1.5B). Although GPT-3 perplexity val- ues are notably lower than those of GPT-2, the general trend is similar. Neither model was fine-tuned for direct comparison.
A.2 Common Crawl News
Similar to NYT, we randomly sample 120K docu- ments that were written after 2018. For finetuning, construct each document by placing the title in the first line, followed by new line, and the rest of the document afterwards. For the non-finetuned experiments, we place the title the first line, fol- lowed by a colon (":"), new line, and the rest of the document after.
the general trends between models are still compa- rable. Indeed, we observe that the n-gram models are occasionally better predicting words the beginnings of documents, which we attribute the frequency of rare words at the beginnings documents which are often replaced with an <unk> token in glosed-vocabulag models.
A.3 Al-Ittihad
C Large Language Models
We split the Al-Ittihad subset of the Arabic Billion Words Corpus (El-Khair, 2016) into train split, containing 8551 documents, a test split containing 2000 documents and validation split containing 1000 documents. We then finetune AraGPT2-Mega (1,5B) (Antoun et al., 2021) by feeding in the title of each document followed by new line, then the contents of the article after.
We primarily use GPT-2 for our experiments due to (a) its public availability, (b) the ability to fine- tunc and run inference on standard hardware, and (c) the availability of comparable models for Ara- bic. We also provide preliminary experiments us- ing the largest GPT-3 model (175B) but do run on all configurations due to cost considerations. We report results 1000 documents from the NYT Annotated Corpus in Figure 7, observing simi- lar trend as with GPT-2 models. We use the base davinci model rather than the instruction-tuned text -ea-dddvviiiioo because our work focuses on the base language modeling objective.
B Constructing Closed Vocabulary
We follow additional preprocessing steps to con- struct a closed vocabulary for the trigram models. We first tokenize each document by splitting on whitespace and lowercasing all alphabetical char- acters. We then form closed vocabulary by re- placing each word which appears in the training data less than five times with the <unk> token. As a result of lowercasing and <unk>ing, we note that exact perplexity values may not be directly com- parable to those computed by GPT-2 models, but
D Modeling Longer Documents
We also attempt feed in longer documents and therefore compute entropy rates on WikiText-2 (Merity et al., 2016) using GPT-2 XL. As shown Figure 8, these results show a non- non-monotonic trend

---

Entropy Rate WikiText
highly sensitive to the length of documents, espe- cially since entropy is least constant the begin- nings of documents. As a result of this sensitivity, we focus primarily on qualitative evaluations of the observed trends rather than on significance tests.
GPT-2 Finetuned
4.50
4.25
4,00
Mean HO0
3.75
3.50
3.25
3,00
2.75
1000
2000
3000
4000
5000
Position in Document
Figure 8: Entropy rate of WikiText-2 on GPT-2 XL (1.5B), up to the first five thousand words. We run this experiment WikiText because it has larger number of very long documents than our primary evaluation corpora. However, we note that entropy values are not accurate due fixed context size and still suffer from noisiness due to a lafck of long-form documents.
even past 1000 words. We note that this result is not an accurate representation of entropy values, due to the fixed 1024 context window of GPT-2. In order to get around the fixed context window, we use a stride of 64 tokens. As result, we expect entropy values to increase in the long run, follow- ing Genzel and Charniak (2002)'s argument for n-grams. We attribute the noise in these results to the lack of documents with more than 5000 words.
E Significance Testing
We report entropy values a per-word basis for both n-gram models and GPT-2. We also apply the non-parametric Mann-Kendall test (Mann, 1945; Kendall, 1948) determine whether entropy rate is monotomically increasing decreasing through- out the course of document. We note that this method not intended to compare the relative sizes of trends and that is sensitive to hyperparame- ters such as the length of perplexity timeseries and choice of tokenization scheme. We omit these find- ings from the main body of the paper primarily due the x-axis cutoff. We
how sensitive they present the results and significance figures in Ta- ble 2. We further note that other methods, such as correlation coefficients or mixed effects models as used in Giulianelli and Fernández (2021) are also

---

Dataset
Model
Fine-tuning Time Inference Time
GPT-2 Small (124M)
3 hours
7 minutes
GPT-2 Medium (345M)
7.5 hours
12 minutes
CC-News
GPT-2 Large (774M)
15 hours
26 minutes
GPT-2 XL (1.5B)
25 hours
37 minutes
Trigram
7 minutes
minute
GPT-2 Small (124M)
3 hours
6 minutes
GPT-2 Medium (345M)
7.5 hours
12 minutes
NYT
GPT-2 Large (774M)
14 hours
24 minutes
GPT-2 XL (1,5B)
24 hours
36 minutes
Trigram
8 minutes
minute
AraGPT-2 Mega (1.5B)
3 minutes
minute
Al-Ittihad
Trigram
1 minute
minute
Table 1: Fine-tuning and inference times across all datasets under each model size.
Dataset
Fine-tuned
Model
Trend
p-value
N/A
Trigram
Increasing
8 10-8
Yes
GPT-2 XL (1.5B)
Decreasing
1 10-5
Yes
GPT-2 Large (774M)
Decreasing
0.4
CC News
Yes
GPT-2 Medium (345M)
Decreasing
0.01
Yes
GPT-2 Small (124M)
Decreasing
9 10-9
No
GPT-2 XL (1.5B)
Decreasing
1 10-17
N/A
Trigram
Increasing
1.10 17
Yes
GPT-2 XL (1.5B)
Decreasing
1-10-13
Yes
GPT-2 Large (774M)
Decreasing
10
NYT
Yes
GPT-2 Medium (345M)
Decreasing
1-10-1
Yes
GPT-2 Small (124M)
Decreasing
1.10 17
No
GPT-2 XL (1.5B)
Decreasing
1 10-10
No
GPT-3 Davinci (175B)
Decreasing
0.1
N/A
Trigram
Increasing
6 10-16
Al-Ittihad
Yes
GPT-2 XL (1.5B)
Increasing
1- 10-14
No
GPT-2 XL (1.5B)
Decreasing 1 10
Table Results of running the Mann-Kendall on each of the experimental conditions in this paper. In general, we observe a decreasing trend for neural models, and an increasing trend for n-gram models. Although this test is non-parametric, we caution that results are highly dependent on the length of the input time-series.


Reviews:
[{"review_id": "UaGuhv5jG9", "reviewer": "Reviewer_8v3T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We’re glad that you found the topic interesting and well-motivated, and that you appreciated our use of larger datasets (as well as the Arabic dataset).\n\nIn response to your point about the contributions being unclear: as you say, because current LMs might be better predictors of word probability than humans, these results don’t provide *negative* evidence for UID (and in fact, it’s hard to disprove such theories). However, prior work like G&C ’02 has used LMs as *positive* evidence for UID, and we argue against this line of reasoning. This is a fairly nuanced point, but we’ll try to make it more clear in any future versions of our paper.\n\nThanks also for the presentation tips. Our graphs are currently dual-coded (with both shape and color), but we’ll experiment with different palettes to fix the contrast issue. Your interpretation of surprisal and probability is also correct — “average word probabilities decline“ was an oversight which we will fix, and we’ll aim to provide some additional intuition about the definitions of surprisal and entropy rate in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It's an interesting area to study, as computational analyses of information in large corpora can inform and be informed by psycholinguistic studies.\n\n- Since the world of language modelling has changed so much in recent years, it makes sense to revisit questions that were investigated using previous technology. Using large datasets (50 million words in each experiment so about 10x what Genzel and Charniak used) allows for more robust results.\n\n- Studying Arabic as well as just English strengthens the generality of the insights", "reasons_to_reject": "- The contributions are slightly unclear. The patterns found are interesting but I'm not sure what implications they have. As the authors acknowledge LLMs may be better predictors of word probability than humans, just as n-gram LMs may be worse than humans. If the psycholinguistic insights that can be achieved are limited due to this mismatch, what else can we use these results for? The paper would be stronger if the authors can help the reader see the contribution.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper reimagines an analysis done by Genzel and Charniak in 2002 to glean insights from statistical language models about the hypothesised constancy of information across a text. Unlike Genzel and Charniak we now have access to LLMs such as GPT which are much better predictors of language than the n-gram models of two decades ago. Taking three corpora in two languages, the authors observe the patterns in word probability across positions in a document; they do gather insights into word distribution, though they do not conclusively prove (or disprove) the entropy rate constancy hypothesis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The figures are hard to read on a non-colour printout as all the plots are similar shades of grey\n\n- For NLP researchers without expertise in this specific area, it will be helpful to give some intuitions about how to interpret notions like entropy rate and surprisal. This seems key to following the core contribution of the paper. \"High probability = low surprisal\" is the way I naively think about it, but I'm still not sure how that tallies with the statements in Section 6 that \"average word probabilities under GPT2-XL sharply decline at the beginning of documents before leveling off\" and \"average per-word surprisal as measured by GPT-2 sharply declines at the beginning of documents in all corpora\", i.e. probability and surprisal decline together."}}, {"review_id": "BQ0derIpxS", "reviewer": "Reviewer_vesJ", "summary": "", "questions": "- Do you assume that neural language models capture not only H(X|L) but also I(X;C|L)? \n- Can you employ any quantitative metrics to quantify the uniformity of information density, which can directly support the main conclusion of this paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We’re glad that you appreciated our inclusion of several different datasets, models, and languages in this analysis.\n\nIn response to your concern about the quantitative measures: unfortunately, any quantitative measure of entropy rate constancy is (necessarily) parametrized by the length of documents considered. For example: G&C ’02 chose an arbitrary cutoff at the 25-th sentence mark. Because many of the entropy rate curves in our paper are monotonically decreasing, they must eventually appear constant or nearly constant with a long enough x-axis (i.e., with long enough documents). However, it’s undeniable that there is some non-constant behavior around the first 50-100 tokens; when we say the conclusion is “a matter of interpretation,” we specifically mean that whether you consider these first 50-100 tokens important is up for debate and is ultimately a matter of how entropy rate constancy is defined. We provide a partial discussion of this issue in lines 274-285 and can elaborate further in any future version of the paper.\n\nHowever, note that we do actually include quantitative results from the non-parametric Mann-Kendall significance test in the appendix, which says whether a time-series is increasing, decreasing, or neither. These show an increasing trend for the trigram model and decreasing trends for almost all neural language models. However, these results come with the above disclaimer about document length and are highly dependent on our choice of x-axis cutoff (in this case, the first 500 tokens). Because of their sensitivity to document length we chose to keep them in the appendix rather than in the main body of the paper.\n\nThanks also for your presentation suggestions. In response to your question about mutual information: we assume that LLMs directly approximate H(X | C,L) which can be used to compute the mutual information based on equation (2) in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To the best of my knowledge, this is the first systematic investigation of the original formulation of the UID hypothesis across datasets (CC News, NYT, Al-Ittihad), models (Trigram, GPT-2), and languages (English, Arabic).", "reasons_to_reject": "- As the authors themselves also point out, the main conclusion of this paper is \"qualitative\" or \"a matter of interpretation\", where no quantitative metrics demonstrate that the original formulation of the UID hypothesis is not supported. In fact, the information density looks relatively uniform to my eyes across three datasets.", "questions_to_authors": "- Do you assume that neural language models capture not only H(X|L) but also I(X;C|L)? \n- Can you employ any quantitative metrics to quantify the uniformity of information density, which can directly support the main conclusion of this paper?", "paper_topic_and_main_contributions": "The main contribution of this paper is to investigate the uniform information density (UID) hypothesis with neural language models, and demonstrates that the original formulation of the UID hypothesis is not supported.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l.120: I(X;C,L) -> I(X;C|L)\n- l.290: Figures are distributed across the paper, so that the readers should go back and forth between the pages. For example, Figure 1 on p.1 is first mentioned in l.290 on p.5."}}, {"review_id": "pCk3Gr0eL1", "reviewer": "Reviewer_sMtY", "summary": "", "questions": "A. How does your token probability computation (Eq. 8) differ from the other related papers (Giulianelli et al., 2021; Meister et al., 2021)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We’re glad you consider this an important finding and appreciated our inclusion of various datasets, model sizes, and preprocessing approaches. We’re also glad you found our presentation of related work to be thorough and accurate. In response to some specific concerns:\n\n> entropy values do not decrease a lot after the initial dip between ~0th-50th\n\nTo some extent, this is inevitable (see our response to vesJ), but you are right that some models approach constancy faster than others. This is also an effect of fine-tuning (cf. Figure 5). We hypothesize that this occurs due to domain adaptation and can further elaborate on these findings in any future version of the paper.\n\n> How does your token probability computation differ from the other related papers?\n\nOur token probability computation is the same as in other work, just parametrized by different language models. The main difference is that we keep these values in terms of token probabilities rather than converting them into sentence probabilities, as in Xu & Reitter (2018) and Giulianelli and Fernandez (2021). This allows us to avoid some additional normalization based on sentence length and also provides a more fine-grained look at the beginnings of documents (256-262).\n\n> LMs integrate long range context\n\nBy this, we simply mean that nearby context strongly affects word probabilities while distant context continues to weakly affect word probabilities. The cited paper by Khandelwal et al. 2018 goes into further detail on this point.\n\nAs noted in our response to 8v3T, we’ll provide a clearer definition of entropy rate and surprisal in any future draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Important finding for the areas of NLG/LM evaluation, dataset/corpora analysis and specifically measuring surprisal/entropy.\n* Detailed analysis featuring various data and model sizes as well as preprocessing steps, potentially sparking interesting discussions and future work in this direction.\n* Very nicely contextualized within related work.\n\t* Similarities and differences between this and other work (mostly the papers by Giulianelli and Meister) have been thoroughly explained (§ 8, § 2).\n\t* To my knowledge, no relevant references are missing, which is rare in such a prolific area of work.", "reasons_to_reject": "The analyses of the (mostly unsurprising) findings remain somewhat shallow (l. 329, l. 337, l. 357, l. 367) and there is no \"deep dive\" into why certain outcomes occur. I don't feel like I've learned much about how the corpora and their analyses tie into the properties of the GPT models and such takeaways are a rather crucial part of papers in this domain. This is different for e.g. Giulianelli et al. (2021) who point out the organization of context in dialogues, which can be used to \"inform the development of natural language generation models.\" The diversity of results between corpora, as depicted by Fig. 2, might be an aspect requiring further analysis than that \"these findings do not provide **clear evidence** for entropy rate constancy\" (l. 321), because, from this high-level perspective, this empirical study is not very actionable. The paper often seems to cut off at points before it gets really interesting.\n\n* One of the most surprising points to me is how entropy values do not decrease a lot after the initial dip between ~0th-50th position. This curve is even flatter for GPT-3 (Fig. 7). This kind of analysis on models with much larger parameter size (l. 773: \"preliminary experiments\") is very intriguing and should be fleshed out and be part of the main paper.\n\n* The claim in l. 317-320 that the findings are consistent with how LMs integrate context needs to be more detailed. The behavior of LMs \"Integrating long-range context\" (l. 401) is mentioned quite often, but the paper does not really show specific results in this direction, or it should be better explained how the Figures are related to that claim.\n\n* The interpretation of entropy values and surprisal is a bit opaque as it's currently described and depicted through Fig. 4, § 6.4, § 7.", "questions_to_authors": "A. How does your token probability computation (Eq. 8) differ from the other related papers (Giulianelli et al., 2021; Meister et al., 2021)?", "paper_topic_and_main_contributions": "This paper empirically evaluates the uniform information density (UID) hypothesis, originally formulated by Genzel & Charniak (2002), stating that humans tend to distribute informational content roughly in a uniform manner across an utterance. While the original claim was supported by empirical evidence from local n-gram language models, this work tests whether this pattern remains robust in light of recent neural language models (NLMs) that are able to condition on much longer contexts. In particular, authors evaluate the entropy rate for GPT-like language models over non-English and chronologically diverse datasets similar to Penn Treebank, finding evidence that goes against the original UID claim. In the discussion, authors speculate that this might be due to NLMs lossless context access compared to human memory retrieval.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The term \"surprisal\" is never properly introduced (l. 069, Fig. 4, l. 310)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Meta Review

This paper revisits the entropy rate constancy hypothesis, by Genzel and Charniak (2002), investigating it using modern language models.
They use autoregressive language models (3-grams and GPTs) to compute the average surprisal of words in a document for several document positions; they do this under a number of experimental conditions.
They then analyse whether those average surprisal values follow a constant, decreasing, or increasing trend (mostly in a qualitative manner).

The reviewers agree this is an important research question to investigate.
They also agree that this paper’s detailed analyses—including a number of models, datasets, and preprocessing steps—make a significant contribution which might spark debates and future work.
The reviewers, however, also point out that the paper lacks a deeper analysis of its results, which makes some of its contributions slightly unclear.
Overall, I agree with the reviewers that this work is interesting and that its analyses might spark future debate.
I also agree with the reviewers that a "deeper dive" into some of its results would strengthen the paper.

# Typos and Other Comments

Line 46: I believe that Zipf only observed that frequent words were shorter, and that Piantadosi et al. observed that contextually predictable sentences are shorter.

Eq 9: I think these should be log-probabilities, instead of probabilities.

Line 264: “per-word probability across”. Again, should this be log-probability instead of probability?

Eq 10: I think this is missing a $\frac{1}{|W|}$, right?

Related Work Section: I think briefly mentioning the similarities/differences between this work and Xu and Reitter (2018) here could be useful.

Line 460: “our work also provides the first analysis of entropy rate constancy in a language besides English”. See Genzel and Charniak (2003).

# References

Genzel and Charniak. 2003. Variation of Entropy and Parse Trees of Sentences as a Function of the Sentence Number.
Decision: Accept-Findings