Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives
Authors: Mario Giulianelli
Sarenne Wallbridge
Raquel Fernández
Venue: EMNLP 2023 Main (2023)
Keywords: surprisal
alternatives
acceptability
reading times
predictability
TL;DR: We present a measure of utterance predictability and show its psychometric predictive power is either stronger or complementary to aggregates of token-level surprisal.

Abstract: We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.

Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives
Mario Giulianelli Sarenne Wallbridge Raquel Fernández Instiiutee for Logic, Language and Computation, University of Amsterdam Centre for Speech Technology Research, University of Edinburgh m. m.g.ilianeellle giulianellieuva.n s13017300ed, ac. uk raquel. fernandez@uva, nl
Abstract
information content, of unit u (Shannon, 1948), perhaps the most widely used measure of informa- tion: [()) = - logg p(u). Predictable units carry low amounts of information -ee., low surprisal as they are already expected to occur given the context in which they are produced. Conversely, unexpected units carry higher surprisal.
We present information value, a measure which quantifies the predictability of utterance rel- ative to a set of plausible alternatives. We in- troduce a method to obtain interpretable esti- mates of information value using neural text generators, and exploit their psychometric pre- dictive power to investigate predictability that drive human comprehension behaviour. Information value is a stronger pre- dictor of utterance acceptability written and spoken dialogue than aggregates of token-level surprisal and it complementary to surprisal for predicting eye-tracked reading times.'
dimensions of
Proper estimation of the surprisal of an utterance is intractable, it would require computing probabilities over a high-dimensional, structured, and ultimately unbounded event space. It is thus common to resort to chaining token-level surprisal estimates, nowadays typically obtained from neural language models (Meister et al., 2021; Giulianelli and Fernández, 2021; Wallbridge et al., 2022). However, token-level autoregressive approximations of utterance probability have few problematic properties. A well-known issue is that different realisations of the same concept or communicative intent compete for probability mass (Holtzman et al., 2021), which implies that the surprisal of semantically equivalent realisa- tions is overestimated. Moreover, token-level surprisal estimates conflate different dimensions of predictability. As evidenced by recent studies (Are- halli et al., 2022; Kuhn et al., 2023), this makes difficult appreciate whether the information carried by utterance is result, for example, of the unexpectedness of its lexical material, syntactic arrangements, semantic content, or speech act type.
1 Introduction
When viewed as information transmission, suc- cessful language production can be seen as an act reducing the uncertainty over future states that a comprehender may be anticipating. Saying a word, for example, may cut the space of possibilities in half, while uttering whole sentence may restrict the comprehender's expectations to a far smaller space. Measuring the amount of information carried by linguistic signal is fundamental to the computational modelling of human language processing. Such quantifications are used in psycholinguistic and neurobiological models of language processing (Levy, 2008; Willems et al., 2016; Futrell and Levy, 2017; Armeni et al., 2017), study the processing mechanisms of neural language models (Futrell al., 2019; Davis and Schijndel, 2020; Sinclair al., 2022), and as a learning and evaluation criterion for language modelling (under the guise of 'cross-entropy loss' or "perplexity'). The amount of information carried by a linguistic signal is intrinsically related to its predictability (Hale, 2001; Genzel and Charniak, 2002; Jaeger and Levy, 2007). This connection is summarised in the definition of the surprisal, or
We propose an intuitive characterisation of the information carried by utterances, information value, which computes predictability over the space of full utterances to account for potential commu- nicative equivalence, and explicitly models multi- ple dimensions of predictability (c.g., lcxical, syn- tactic, and semantic), thereby offering greater in- terpretability of predictability estimates. Given a linguistic context, the information value of an terance is a function of its distance from the set contextually expected alternatives. The intuition is
Shareed first authorship.
"http://ggthubb.com/mg wwwp://mgilllliifoomaatoooaaaaaa

---

Ar siz piena the othr day.
Verma et al., 2023) and for the design of processing and decision-making mechanisms that reproduce them in natural language generation systems (Wei et al., 2021; Giulianelli, 2022; Meister et al., 2023).
So, what do you feel tiee eating then?
burgars?
B: airzady hail harger yorknday.
suruly will, How much a this word carving?
X
dt Well. enting lugges
2 Background
I
Good
All nigh. rs go get on onder of those right now.
Surprisal theory. Expectation-baset theories of language processing define the effort required to process a linguistic unit as a function of its predictability. Surprisal theory, perhaps the most prominent example, posits direct relationship between effort and predictability, quantified surprisal (Hale, 2001). The theory is supported by broad empirical evidence across domains and languages (Pimentel et al., 2021; de Varda and Marelli, 2022). and serves as foundation for quantitative principles of language production and comprehension such as Entropy Rate Constancy (ERC; Genzel and Charnia,, 2002) and Uniform Information Density (UID; Levy and Jaeger, 2007).
LM
44 Hle Chinos food
dg hangry
a
dg
44
Figure I: The information value J(Y y X x) of target utterance y is lower and its predictabil- ity higher-when is closer to the set of plausible al- ternatives A, (a, ....... Here, alternatives are generated by LM conditioned context x.
The psychometric predictive power of surprisal. Without direct access to the 'true'2 conditional probabilities of linguistic units, psycholinguists have relied on statistical models of language to estimate surprisal (Hale, 2001; McDonald and Shillcock, 2003). More recently, large-scale lan- guage models have emerged as powerful estimators of token-level surprisal, reflected by their ability to predict different aspects of human language comprehension behaviour (their psychometric predictive power). Psychometric variables include self-paced and eye-tracked reading times (Keller, 2004; Goodkind and Bicknell, 2018; Wilcox et al.. 2020; Meister et al., 2021; Shain et al., 2022; Oh and Schuler, 2023), acceptability judgements (Lawrence et al., 2000; Heilman et al., 2014; Lau et al., 2015, 2017: Warstadt et al., 2019; Wallbridge et al., 2022), and brain response data (Frank et al., 2015; Schrimpf et al., 2021).
that if an utterance differs largely from alternative productions, it is unexpected contribution to dis- course with high information value (see Figure 1). We obtain empirical estimates of information value by sampling alternatives from neural text genera- tors and measuring their distance from a target utter- ance using interpretable distance metrics. Informa- tion value estimates are evaluated in terms of their ability to predict and explain human reading times and acceptability judgements in dialogue and text.
We find information value have stronger psy- chometric predictive power than aggregates of token-level surprisal for acceptability judgements in spoken and written dialogue, and to be com- plementary to surprisal aggregates as predictor of reading times. Furthermore, we use our inter- pretable measure of predictability gather insights into the processing mechanisms underlying human comprehension behaviour. Our analysis reveals, for example, that utterance acceptability in dialogue is largely determined by semantic expectations while reading times are more affected by lexical and syn- tactic predictions.
To obtain estimates utterance surprisal, differ- ent aggregates of token-level surprisal have been proposed, motivated by psycholinguistic theories like ERC and UID. However, their behaviour is far less understood (e.g., Wallbridge et al., 2022). For example, divergences between how model characteristics affect predictive power for different comprehension tasks (e.g., Meister et al., 2021) raise questions about whether leven-level
Information value is new way to measure predictability. As such, next to surprisal, it is powerful tool for the analysis of comprehension behaviour (Meister al., 2021; Shain al., 2022; Wallbridge et al., 2022, 2023), for the computa- tional modelling of language production strategics (Doyle and Frank, 2015; Xu and Reitter, 2018;
In this context, true refers to the-if at all existing- unattainable conditional prohabilities of linguistic units that human may experience during language comprehension.

---

aggregates appropriately capture expectations over utterances in human language processing.
distance of an utterance from its alternatives, and (3) a means with which summarise distributions of pairwise distances. We discuss these three elements turn in the following paragraphs.
Alternatives in semantics and pragmatics. Our proposed notion of information value takes inspi- ration from the concept of alternatives in seman- ties and pragmatics (Hom, 1972; Grice, 1975; Stal- naker, 1978; Gazdar, 1979: Rooth, 1996; Levin- son, 2000). Reasoning about alternatives has been argued to be at the basis of the use of questions (Hamblin, 1976; Groenendijk and Stokhof, 1984; Ciardelli al., 2018), focus (Rooth, 1992; Wagner et al., 2005; Beaver and Clark, 2009), and implica- tures (Carston, 1998: Degen and Tanenhaus, 2015, 2016; Zhang et al., 2023). Recently, alternative sets generated with the aid of language models have been used to provide empirical evidence that pragmatic inferences of scalar implicature depend on listeners' context-driven uncertainty over alter- natives (Hu et al., 2022, 2023). Hu et al. (2022) generate sets of plausible words in context, within scalar constructions, then embed and cluster the resulting sentences simulate conceptual alter- natives. Reasoning over word- and concept-level alternatives is operationalised through surprisal and entropy. To our knowledge, ours is the first study to use language models for the generation of full utterance-level alternatives.
Generating alternative sets. Since the 'true' al- ternative sets entertained by human comprehen- der are not attainable, we propose generating them algorithmically, via neural text generators. Be- ing able to guarantee the plausibility, or human- likeness of the generations is crucial. Our approach builds on recent work (Giulianelli al., 2023) find- ing the predictive distribution of neural text gen- erators to be well aligned to human variability, measured with the same distance metrics used in this paper (see next paragraph): while not all gen- crations are guaranteed to be of high quality, their low-dimensional statistical properties (e.g., n-gram, POS, and speech act distribution) match those of human productions. This should allow us to obtain faithful distance distributions d(y, Ag) and thus accurate estimates of information value.
Measuring distance from alternatives. We quantify the distance of target utterance from an alternative production using three interpretable distance metries, defined by Giulianelli al. (2023). Lexical: Fraction of distinct n- n-grams in two utterances, with n E 2, (i.e., the num- ber of distinct n-gram occurrences divided by the total number of n-grams in both utterances). Syn- tactic: Fraction of distinct part-of-speech (POS) n-grams in two utterances. Semantic: Cosine and euclidean distance between the sentence embed- dings of two utterances (Reimers and Gurevych, 2019). These distance metrics characterise alter- native sets varying levels of abstraction (Katzir, 2007; Fox and Katzir, 2011 I; Buccola et al., 2022), enabling an exploration into the representational form of expectations over alternatives in human language processing.
3 Alternative-Based Information Value
Given a context speaker may produce num- ber of plausible utterances. We refer to these as Az. the alternative set. We define the information value of an utterance y in a context as the real random variable which captures the distribution of distances between and the set of alternative productions Ag. measured with a distance metric d:
(Y= d(y, Ax)
(1)
This distribution characterises the predictability of y in its context. Large distances indicate that y differs substantially from expected utterances, and thus that y a surprising next utterance.
Summarising distance distributions. Informa- tion value is a random variable that describes a distribution over distances between an utterance y and the set of plausible alternatives (Equation I). To summarise this distribution, we explore mean the expected distance (under uniform distribution over alternatives) as the distance from proto- typical alternative, and min as the distance of y from the closest alternative production, implicating that proximity to a single alternative sufficient to determine predictability.
3.1 Computing Information Value
In Equation 1, we define information value as a statistical measure of the unpredietability, or unexpectedness of utterance. In practice. computing the information value an utterance requires (1) a method for obtaining alternative sets Az, (2) a metric with which to measure the

---

4 Experimental Setup
judgements which examine grammaticality as well as semantic and pragmatic plausibility.
4.1 Language Models
SWITCHBOARD and DAILYDIALOG. Partici- pants were presented with short sequence of dia- logue turns followed by potential upcoming turn, and asked to rate its plausibility in context on a scale from to 5. Judgements were collected by Wallbridge et al. (2022) for (transcribed) spoken di- alogue and written dialogue from the Switchboard Telephone Corpus (Godfrey et al., 1992) and Dai- lyDialo (Li et al.. 2017), respectively. For cach corpus, 100 items are annotated by 3-6 participants. Annotation items consist of 10 dialogue contexts, each followed by the true next turn and by turns randomly sampled from the respective corpus.
We generate alternative sets using neural autore- gressive language models (LMs). For the dialogue corpora, we use GPT-2 (Radford et al., 2019), Di- aloGPT (Zhang al., 2020), and GPT-Neo (Black al., 2021). For the text corpora, we use GPT-2, GPT-Neo, and OPT (Zhang et al., 2022). The text models are pre-trained, while dialogue models are fine-tuned on the respective datasets. Further de- tails on fine-tuning and perplexity scores are in Appendix A. The resulting dataset, which contains 1.3M generations, is publicly available.?
Generating alternatives. To generate an alter- native set Ag, we sample from PI.M (Y =r). We experiment with four popular sampling algo- rithms to ensure that the quality of our informa- tion value estimates not dependent on par- ticular algorithm-or, if it is, that we are not overlooking it. We select (1) unbiased (ances- tral or forward) sampling (Bishop, 2006; Koller and Friedman, 2009), (2) lemperature sampling (a E (0.75, 1.25)), (3) nucleus sampling (Holtz- man al., 2019) (p E [0.8, 0.85, 0.9, 0.95)), and (4) locally typical sampling (Meister et al., 2023) (r E [0.2, 0.3, 0.85, 0.95)), for a total of sam- pling strategies. We post-process alternatives to ensure that each contains only a single utterance. 4
CLASP. Participants were presented with sen- tences from the English Wikipedia in and out of their document context and asked to judge accept- ability using 4-point scale (Bemardy et al., 2018). The original sentences are round-trip translated into languages to obtain varying degrees of accept- ability: the context not modified. This dataset contains 500 stimuli, annotated by 20 participants.
4.2.2 Reading times
Previous literature regarding the predictive power of language models for reading behaviour has fo- cused on the relationship between per-word sur- prisal and reading times (Keller, 2004; Wilcox et al., 2020; Shain et al., 2022; Oh and Schuler, 2023). We define utterance-level reading time as the total time spent reading the constituent words of the utterance. This approach has been taken by pre- vious studies of utterance-level surprisal (Meister et al.. 2021; Amenta al., 2022).
4.2 Psychometric Data
Using five corpora, we study two main types of psychometric variables that rely on different under- lying processing mechanisms (Gibson and Thomas, 1999; Hofmeister et al., 2014): acceptability judge- ments and reading times.
PROVO. This corpus consists of 136 sentences (55 paragraphs) of English text from a variety of genres. Eye movement data was collected from 84 native American English speakers (Luke and Chris- tianson, 2018). We use the summation of word- level reading times (IA-DWELL-TIME, the total du- ration of all fixations on the target word) of con- stituent words to obtain utterance-level measures.
4.2.1 Acceptability judgements
Stimuli for acceptability judgements typically consist of isolated sentences that are manipulated automatically or by hand to assess a grammatical notion of acceptability (Lau et al., 2017; Warstadt et al., 2019). The effect of context acceptability is still relatively underexplored, yet contextualised judgements arguably capture a more natural, in- tuitive notion of acceptability. In this study, we use some of the few datasets of in-context acceptability
This corpus consists of self- moving window reading times for 450 sentences (12 passages) from the Brown corpus of American
BROWN.
AltGen: https://doi.coo wrt.10..280/0000 (0..528//zenda 10006413.
5 'Acceptability ratings available https::/daats cstr.
"We use spaCy's s sentence segmentation algorithm (Honni- bal et al., 2020) for the text corpora and split dialogue utter- ances hased on the position of the turn separator.
ed. ac. uk/ htt///renne//NN MwM/aarrem///TEEPPEEEHH0222 INTERSPEECH2222/.
"We only use judgements collected in context, available at /tnps://githin com//U- BLL.-CCLLLPBPBLLL111111

---

Mean cusine
Min
Information value
Surprisal
-064
Acceptability (s
-3.00
SWITCHBOARD
0.702. (semantic)
0.506 (uperrinnar, k
4.50
DAILYDIALOG
-0.584 (xemantic)
-0,457 (zsperlineor k 2.5)
-3%
2.5g
(syntactic)
(meun)
Reading times x v PROVO
0.421 (symtactic)
0.495 (varionce)
0.50
0.223 (lesicut)
0.220 (meon)
3.4.0
-0.4%
Table 1: Correlations of the most predictive variants (in parentheses) of surprisal and information value across model, sampling algorithm, and alternative set size with psychometric data: mean acceptability judgements and length-normalle reading times.
10 20
20
40
30
100
10
20
30
40
70
06
15 A1
91
Allessatvv Stae
Aliszative
Medel
Mudion
Madian
CorENee 1.11
GPT2Exge
DRIGGPTIN Lorge
Figure 2: Spearman correlation between semantic in- formation value and mean acceptability judgements in SWITCHBOARD, Confidence intervals display variabil- ity over II sampling strategies.
5.1 Predictive Power
For every corpus, we obtain moderate strong Spearman correlation between information value and the psychometric target variable. For example. estimates of semantic information value correlate with acceptability judgements at strengths approximately between -0.1 and -0.7 for SWITCHBOARD and between -0.3 and -0.6 for DAILYDIALOG across models and sampling strategies from Section 4.1 (see Figure for SWITCHBOARD; Appendix for all datasets). Estimates obtained with the best information value estimators for each corpus, shown in Table 1, yield substantially higher correlations with acceptability in dialogue than the best token-level aggregates of utterance surprisal, both as computed in our experiments and reported in prior work (Wallbridge et al., 2022, 2023). Reading times, on the other hand, which are aggregates of word-level psychometric data points, should naturally be easier to capture with word-level measures predictability. Nevertheless, our best information value estimates correlate with reading times only slightly less strongly comparably to surprisal: and additionally, they give indications about the dimensions of unexpectedness (in this case, lexical and syntactic) that mostly affect reading behaviour.
English. Reading times were collected from 35 native English speakers (Smith and Levy, 2013).
5 Psychometric Predictive Power
We begin by evaluating empirical estimates of information value in terms of their psychomet- ric predictive power: can they predict comprehen- sion behaviour recorded as human acceptability judgements and reading times? We test the robust- ness of this predictive power to the alternative set generation process and compare it to previously proposed utterance-level surprisal aggregates in- cluding mean, variance, and range of summation strategies; Appendix B for full definitions.
For each corpus in Section 4.2, measure the correlation between information value and the re-
spective psychometric variable, which is the av-
erage in-context acceptability judgement for DAI- LYDIALOG, SWITCHBOARD, and CLASP, and the total utterance reading time normalised by utter- ance length for PROVO and BROWN.? Alterna- tive sets are generated using the language models and sampling strategies described in Section 4.1. Lexical, syntactic, and semantic distances puted in terms of the distance metrics presented in Section 3.1, for alternative sets of varying size ([10, 20,... 100).. The distributions of similarities in Equation! are summarised using mean and min, thus yielding scalar estimates of information value.
Overall, beyond building trust our information value estimators, this evaluation demonstrates the benefit of their interpretability. The predictive power for lexical, syntactic, and semantic distances varies widely between corpora. Semantic distances much more predictive for dialogue datasets than lexical syntactic distances, while the inverse is true for the reading times datasets. We explore differences between the underlying perceptual processes employed for these two comprehension tasks further in Section 6.
WW normalise by utterance length us is an obvious corre- late of total reading time and would have confounding effects on this analysis. In Section 6, we confirm our findings using mixed effect models that include utterance length as a predic- for and total unnormalised reading time as a response variable.

---

5.2 Robustness to Estimator Parameters
using them as explanatory variables in linear mixed effect models to predict per-subject psychometric data. For the dialogue corpora and CLASP, our mixed effect models predict in-context acceptabil- ity judgements. For the reading times corpora, our models predict the total time spent by subject reading sentence, as recorded in self- slee-paced reading and eye-tracking studies. This is the sum, over sentence, of word-level reading times (more details in Appendix F). We include random intercepts for (context, target) pairs in all models.
We now study the extent to which our estimates are affected by variation in three important pa- rameters of alternative set generation: the alterna- tive set size ([10, 20, 100), the language model, and the sampling strategy. We find slight pos- itive, asymptotic relationship between predictive power, reflected by correlations between informa- tion value and psychometric data, and alternative set size for semantic information value in the di- alogue corpora- intormatioooinnnnnn value estimates be- come more predictive alternative set size in- creases (see, e.g., Figure 2). Set size does not sig- nificantly affect correlations for the reading times corpora. Moreover, while we do observe differ- ences between models, and larger models tend to obtain higher correlations with psychometric vari- ables, these results are not consistent across corpora and distance metrics (Figures 4 and 5, Appendix C). In light of recent findings regarding the inverse re- lationship between language model size and the predictive power of surprisal (Shain al., 2022; Oh and Schuler, 2023), we consider an encourag- ing result that the predictive power of information value does not decrease with the number of model parameters." 8 We do not observe a significant im- pact of decoding strategy on predictive power, re- gardless of alternative set size, as indicated by the confidence intervals in Figures 2, 4 and 5.
Analysis Procedure. For every corpus, we first test models that include single predictor beyond the baseline: i.e., information value measured with each distance metric and either mean and min as summary statistics (see Section 3.1). Based on the fit of these single-predictor models, we select the best lexical, syntactic, and semantic distance metrics (with the corresponding summary statisties) to instantiate ftree-predicton models for each of the derived measures of information value.
Following Wilcox et al. (2020), we evaluate each model relative to baseline model which includes only control variables. Control variables are selected building on previous work (Meister et al., 2021): solely the intercept term for acceptability judgements and the number of fixated words for reading times (more details in Appendix F). As an indicator of explanatory power, we report LogLik, the difference in log-likelihood between model and the baseline: a positive AoogLik value indicates that the psychometric variable is more probable under the comparison model. We also report fixed effect coefficients and their statistical significance. The full results are shown in Table 6 (Appendix F), according to which the best metrics for each linguistic level are selected and used throughout the rest of the paper.
In sum, estimates of information value do not display much sensitivity to alternative set generation parameters." 9 Therefore, for each corpus, we select the estimator (a combination of model, sampling algorithm, and alternative set size) that yields the best Spearman correlation with the psychometric data (Table 5 in Appendix E). We use these estimators throughout the rest of the paper.
6 In-Depth Analysis of Psychometric Data
6.1 Derived Measures of Information Value
Using information value, we now study which dimensions of predictability effectively explain psychometric data. This allows us to qualitatively analyse the processes humans employ while read- ing and assessing acceptability. We also examine the effect of contextualisation on comprchension behaviour by defining two additional measures derived from information value (Section 1. and
Inspired by information-theoretie concepts used in previous work to study the predictability of utter- ances (e.g., Genzel and Charniak, 2002; Giulianelli and Fernández, 2021; Wallbridge et al., 2022), we define two additional derived measures of infor- mation value and assess their explanatory power. Out-of-contexi information value is the distance between an utterance y and the set of alternative productions A. expected given the empty context e:
'It remains to be seen whether this trend extends larger language models, for which we lack computational resources. "We ohtain similar evidence of rohustness to parameter settings using an intrinsic evaluation, reported in Appendix D.
T(Y=y) = -l =
(2)

---

Switchbourd
is by far the most predictive variable (Table 6, Appendix F), especially when min is used as a summary statistic. Responses to the same dialogue context can exhibit great variability and being close to single expected alternative-in terms semantic content and dialogue act type-appears to be sufficient for utterance to be considered acceptable. Our analysis of derived measures (Fig- ure 3) further indicates that acceptability is mostly determined by the in-context predictability of an utterance. The high explanatory power of context informativeness (almost twice that out-of- context information value) suggests that contextual cues override inherent isolated plausibility.
30
DailyDialog
Clasp
1
40
Prevo
Brown
&Tog1.3
30
20
10
N(Y y y|x x] -|=
=
Figure 3: Explanatory power of information value and derived measures (out-of-context information value and context informativeness; defined in Section 1.
CLASP. Syntactic information value is the best explanatory variable for acceptability judgements in CLASP (Table 6, Appendix F). This suggests that comprehenders entertain expectations over syntactic structures (here, represented as POS sequences) result which could complement find- ings on the processing of lexicalised constructions in reading (e.g., Tremblay et al., 2011) and eye- tracking studies (c.g., Underwood et al., 2004). In contrast to the dialogue corpora, estimates of in- context information value are less predictive than their out-of-contex counterparts (Figure 3), which may be due to the previously discussed artificial nature of the CLASP negative samples. In sum, our results indicate that the acceptability judgements in the CLASP corpus, even collected context, are mostly determined by the presence of startling surface forms rather than by semantic expectations.
It reflects the plausibility of y regardless of its context of occurrence. An analogous notion is decontextualised surprisal.
Context informativeness is the reduction infor- mation value for contributed by context x:
C(Y - x-x) = ((Y-)))((Y-
(3)
This quantifies the extent to which a context re- stricts the space of plausible productions such that y becomes more predictable. An analogous notion is the pointwise mutual information.
6.2 Acceptability
We generally expect inverse relationship between information value and in-context accept- ability judgements: information value is lower when target utterance is closer to the set of alternatives a comprehender may expect in a given context (see Figure 1). Furthermore, we expect grammaticality and semantic plausibility- factors known to affect acceptability (Sorace and Keller, 2005; Lau et al., 2017) to play different roles in dialogue and text. For the dialogue corpora, we expect semantic-level variables have high explanatory power, as they can identify utterances with incoherent content such as implausible underlying dialogue acts (Searle, 1969, 1975; Austin, 1975). Lexical and syntactic information value may be more explanatory of acceptability in CLASP, where stimuli are generated via round-trip translation and thus may contain disfluent or ungrammatical sentences (Somers, 2005).
6.3 Reading Times
When reading, humans continually update their ex- pectations about how the discourse might evolve (Hale, 2001; Levy, 2008; Yan and Jaeger, 2020). This is reflected, for example, in the faster process- ing of more expected words and syntactic structures (Demberg and Keller, 2008; Smith and Levy, 2013). High predictive power for lexical and syntactic information value would support these findings. However, comprehenders also reason about seman- tic alternatives, c.g., to compute scalar inferences (Van Tiel et al., 2014; Hu et al., 2023). Our inter- pretable measures of information value help clarify the contribution of different types of expectations.
PROVO and BROWN. Syntactic information value is strong predictor of eye-tracked reading times in PRovo, while lexical information value (in particular, based on trigram distances) is the
SWITCHBOARD and DAILYDIALOG. For both dialogue corpora, semantic information content

---

SWITCHBOARD
DAILYDIALOG
PROVO
the most predictive surprisal and information value estimators (one per linguistic level) in isolation and jointly as fixed effects. Table 2 summarises the results of this analysis.
Surprisal
6.63
5.08
59.04
Information value
Lesicol
8.32
10.88
12.17
Syntactic
2.49
6.71
21.80
Semantic
34,20
30.41
6.86
In isolation, information value is a better pre- dictor for the dialogue corpora. Including lexical, syntactic, and semantic information value on top of the best surprisal predictor (Joint) improves model log-likelihood substantially. Separately including each linguistic level reveals that semantic distance is largely responsible for improved fit, suggest- ing that surprisal fails to capture expectations over high-level linguistic properties of utterances such as speech act type, which are crucial for modelling contextualised acceptability in dialogue. This is true regardless of the aggregation function used.
All
43.11
35.42
45.19
Joint
Lexical
14.08
10.23
72.60
Syntactic
9.77
8.05
75.70
Semantic
34.37
26.98
68.61
All
44.11
30.55
93.08
Table 2: LogLLi for surprisal, information value, and joint mixed effect models. We report the best infor- mation value metrics (as per Section 6) and surprisal aggregates for each dataset: maximum for SwrrcH- BOARD, and super-linear for DAILYDIALOG (k- 1.5) and PROVO (A: = 0.5).
For PROVO, surprisal is the best explanatory variable. However, including the best information value predictors further improves model fit by 58%, demonstrating the complementarity of the two measures in predicting reading times (Table 2). Separately adding information value predictors shows the strongest boost comes from syntactic factors, which are known to have higher weight in human anticipatory processing than in language models' (Arehalli et al., 2022).
only significant explanatory variable for the self- paced reading times in BROWN (Table 6), and only weakly so. Expectations over full semantic alter- natives have a limited effect on reading times in both corpora, suggesting anticipatory processing mechanisms at play during reading operate at lower linguistic levels. For both corpora, out-of-context estimates are at least as predictive as in-context estimates and higher than context informativeness (Figure 3), indicating that context modifications only moderately dampen the negative effects of un- usual syntactic arrangements and lexicalised con- structions on reading speed.
Overall, combining predictive information value with surprisal yields better models for all tested cor- pora, indicating that these measures capture distinct and complementary dimensions of predictability.
7 Relation to Utterance Surprisal
7.2 Effects of Discourse Context
We have shown alternative-based information value to be a powerful predictor for contextualised acceptability judgements and reading times. In fact, information value is substantially more predictive of acceptability than utterance surprisal (Section 5). We conclude with a focused comparison between these measures, considering whether they are complementary and why they might diverge.
While language comprehension is known to be a function of context (e.g.. Kleinschmidt and Jaeger, 2015; Chen et al., 2023), little attention has been given to its impact on surprisal estimates. We ex- amine whether the dissimilar predictability esti- mates of information value and surprisal stem from differences in their sensitivity to context, compar- ing how they behave under congruent, incongru- ent, and empty context conditions. In each con- dition, alternative sets and token-level surprisal are computed in the truc context (congruent), context randomly sampled from the respective cor- pus (incongruent), or with no conditioning (empty) as used to compute out-of-context information volue 10 We quantify effects on the best informa- tion value and surprisal predictors as AogLiik, us- ing single-predictor models described in Section 6.
7.1 Complementarity
Differences in predictive power between informa- tion value and surprisal (see Table 1) may reflect variations between the dimensions of predictability captured by the two measures. To investigate this possibility, use both measures jointly for psychometric predictions. We focus on the dialogue corpora and PRovo, where we observed the highest explanatory power for information value (Section 6). For each corpus, we fit lincar mixed effect models with control variables, using
"To ensure that all stimuli in this analysis are contextu- alised, first sentences in PROVO paragraphs were excluded.

---

Duteci
Samm.
Manic
Congmennt
Empty
Iromggree
token-level surprisal, information value captures
Loioi
Trigam
8,32
7.18
variability above the word level by explicitly ac-
SWITCHBONRD
Meze
Mia
Semvntic
Systemi
PIS Ingram
cosine
34.30
2.49
7.64
3,00
10.90
2.5
counting for more abstract communicative units
Surpesal (tt conrent. max
6,63
2,56
3112
like speech acts (Searle, 1969, 1975; Austin, 1975).
Lzorell
Regram
18,8
3.18
1.42
DAILYDALLO
Mext
Mis
Systzciic
Scuratlic
POS Unigram
Cusise
30.41
6.71
6.89
1.43
6.16
290
We validate our measure by assessing its psychome-
trie predictive power, its robustness to parameters involved in the generation of alternative sets, and its sensitivity to discourse context.
Supuisi (o comtest supellizar
1.5)
5.08
0.99
235
Mext
Lesocal
Trgram
12,97
12.94
11.86
PROVO
Mese
Meat
Semontic
Syntzche
IOS Tnggare
Euclideen
25,86
8,53
10.85
15:30
12:94
8,33
Surprtol (In contect, seperinear é 0.30
35.75
17.85
38.00
Using interpretable measures centred around in- formation value, we investigate the underlying di- mensions of uncertainty in human acceptability judgements and reading behaviour. We find that acceptabilityy judgements factor in base rates of ut- terance acceptability (likely associated with gram- maticality) but are predominantly driven by seman- tic expectations. In contrast, reading time is more influenced by the inherent plausibility of lexical items and part-of-speech sequences. We further compare information value to aggregates of token- level surprisal, finding differences in the dimen- sions of predictability captured by each measure and their sensitivity to context. Information value is stronger predictor of acceptability in written and spoken dialogue and is complementary surprisal for predicting eye-tracked reading times.
Table 3: LogLLik of angle-preddctor models for infor-
mation value and surprisal across context conditions.
Table 3 displays results for SWITCHBOARD, DAILYDAAOG, and PROVO. Congruent context produces substantial effect on the predictive power of semantic information value for both di- alogue datasets; for DIALOG, we see a 20- fold increase over the empty context condition. Sur- prisal shows a similar trend, though far less pro- nounced. Syntactic information value is the least affected by context modulations. Though surprisal is a powerful predictor for reading times in PRovo, the incongruent and empty context conditions are more predictive than the true context. Perhaps most concerning the fact that estimates in incongruent contexts are the most predictive. In contrast, the most predictive information value (syntactic) is sig- nificantly more predictive for congruent contexts. Interestingly, information value in the control con- ditions is not uninformative, likely reflecting the inherent plausibility of utterances.
Information value is defined in terms of plau- sible continuations of the current linguistic context, taking inspiration from the tradition of alternatives in semantics and pragmatics (Hor,, 1972; Grice, 1975; Stalnaker, 1978). Although the ideal set of alternatives would be derived directly from humans, neural text generators have demonstrated their potential to act as useful proxies, particularly when multiple generations are considered. Vari- ability among their productions has been shown to align with human variability (Giulianelli et al., 2023), and decision rules that operate over sets of alternative utterances, rather than next tokens, have been shown to improve generation quality (e.g., Eikema and Aziz, 2022; Guerreiro et al., 2023). We release our full set of 1.3M generated alternatives. obtained with a variety of models and sampling algorithms, to facilitate research in this direction.
Both information value and utterance surprisal display sensitivity to context, however, the effects on surprisal are less predictable and perhaps even undesirable for certain psychometric variables.
8 Discussion and Conclusion
Humans constantly monitor and anticipate the tra- jectory of communication. Their expectations over the upcoming communicative signal are influenced by factors spanning from the immediate linguistic context to their interpretation of the speaker's 's goals. These expectations, in turn, determine aspects of language comprehension such as processing cost, as well as strategies of language production. We present information value, measure which quan- tifies the predictability of an utterance relative to a set of plausible alternatives; and we introduce a method to obtain information value estimates via neural text generators. In contrast to utterance predictability estimates obtained by aggregating
Our information value framework allows consid- erable flexibility in defining alternative set gener- ation procedures, distance metrics, and summary statisties. We hope will enable further investiga- tion into the mechanisms involved in human lan- guage processing, and that will serve as basis for cognitively inspired learning rules and inference algorithms in computational models of language.

---

Limitations
Ethics Statement
Our framework for the estimation of utterance in- formation value allows great flexibility. Modellers can experiment with a variety alternative set gen- eration procedures, distance metrics, and summary statistics, While our selection of distance metrics characterises the relation of an utterance to its al- ternative sets at multiple interpretable linguistic levels, there is large space of metrics that have not tested in this paper. Syntactic distances, for example, can be computed using metrics that capture structural differences between utterances in a more line-grained manner (e.g., tree edit dis- tance or difference in syntactic tree depth); seman- tic distances can be computed with a more taxo- nomical approach (e.g., Fellbaum, 2010) or using NLI models to capture semantic equivalence (Kuhn al., 2023); and distances between dialogue act types can be detected using dialogue act classifiers (Stasaski and Hearst, 2023). We chose metrics based on prior work validating them as probes for the extraction of uncertainty estimates from neural text generators (Giulianelli et al., 2023), but we hope future work will explore this space more ex- haustively. Similarly, though the current work has been constrained to English data, framework can be directly applied to other languages. We hope see work in this direction.
In the Limitations section, we have mentioned that an important direction for future work is designing more computationally efficient methods for the es- timation of information value. This is crucial to the application of this method larger datasets, which may be prohibitively expensive in some research communities and any case, perhaps unnecessar- ily, envirommentally unfriendly.
The limited size of the corpora of psychometric data used in this paper has further ethical implica- tions, as the corpora have not been collected to be representative of wide and diverse range of com- prehenders. We hope to see efforts this direction.
Acknowledgements
We thank the ILLC's Dialogue Modelling Group for helpful comments and discussions. MG and RF are supported by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agree- ment No. 819455).
References
Simona Amenta, Jana Hasenäcker, Davide Crepaldi, and Marco Marelli, 2022. Prediction at the intersection of sentence context and word form: Evidence from eye- movements and self-paced reading. Psychonomic Bulletin & Review.
Moreover, due to computational constraints, we selected single information value estimator per corpus for our analyses in Sections 6 and 7. Al- though we assessed the sensitivity of information value to parameters of alternative set generation extensively Appendices C and D, the effect of estimator parameters on the explanatory power of information value predictors can be assessed more widely in future work.
Subas Archalli, Brian Dillon, and Tal Linzen. 2022. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities. Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 301-313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
A further aspect of our method for the estimation of information value that we have not highlighted in the paper is its computational cost. Because it involves drawing multiple full utterance sam- ples from language models, our method is clearly less efficient than traditional surprisal estimation, which requires only single forward pass. While we have observed that the psychometric predictive power of information value reaches satisfactory lev- els even with relativefy low numbers of alternatives and small language model architectures (see, e.g.. Figure 2), designing more efficient methods for the estimation of information value is an important direction for future research.
Kristijan Armeni, Roel M. Willems, and Stefan L. Frank. 2017. Probabilistic language models in cognitive neuroscience: Promises and pitfalls. Neuroscience & Biobehavioral Reviews, 83:579-588.
John Langshaw Austin. 1975. How to do things with words. Clarendon Press, Oxford.
David Beaver and Brady Clark. 2009. Sense and sensitivity: How focus determines meaning. John Wiley & Sons.
-han-Philippe Bernardy, Shalom Lappin, and Jey Han Lau. 2018. The influence of context on sentence acceptability judgements. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 456-461,

---

Melbourne, Australia, Association for Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2022. Sampling- based approximations to minimum Bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 19978-10993, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Informatinn Science and Statis- (ics). Springer-Verlag. Berlin, Heidelberg.
Sid Black, Gao Leo. Phil Wang. Connor Leahy. and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive I Language Modeling with Mesh- Tensorflow.
Christiane Fellbaum. 2010. Wordnet. In Theory and ap- plications of ontology: Computer applications, pages 231-243. Springer.
Brian Buccola, Manuel Križ, and Emmanuel Chemla. 2022. Conceptual alternatives: Competition in lan- guage and beyond. Linguistics and Philosophy, 45(2)2265-291.
Danny Fox and Roni Katzir. 2011. On the characteri-
zation of alternatives. Natural language semantics, 19:87-107.
Stefan I. Frank, Leun J Otten, Giulia Galli, and Gabriella Vigliocco. 2015. The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140:1-11.
Robyn Carston. 1998. Informativeness, relevance and scalar implicature. Pragmatics And Beyond New Series, pages 179-238.
Richard Futrell and Roger Levy. 2017. Noisy-conter surprisal as a human sentence processing cost model. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 1. long papers. pages 688 698.
Sihan Chen, Sarah Nathaniel, Rachel Ryskin, and Edward Gibson. 2023. The effect of context on noisy-channel sentence comprehension. Cognition, 105:00.503.
Ivano Ciardelli, Jeroen Groenendijk, and Floris Roelof- sen. 2018. Inquisitive semantics. Oxford University Press.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 32-42, Minneapolis, Minnesota. Association for Computational Linguistics.
Forrest Davis and Marten van Schijndel. 2020. Dis- course structure interacts with reference but not syn- tax in neural language models. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learning, pages 396-407.
Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non non-native reading. In Findings of the Associa- tion for Computational Linguistics: AACL-ICCNLP 2022, pages 144. Association for Computational Linguistics.
Gerald Gazdar. 1979. Pragmatics, implicature, presupo- sition and logical form. Critica, 12(35).
Dmitriy Genzel and Rugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 106-
Judith Degen and Michael K Tanenhaus. 2015. Process- ing scalar implicature: A constraint-base approach. Cognitive science, 39(4)6667-710.
Edward Gibson and James Thomas. 1999. Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical. Language and Cognitive Processes, 14:225-248.
Judith Degen and Michael K Tanenhaus. 2016. Avail- ability of alternatives and the processing of scalar im- plicatures: A visual world ere-tracking study. Cogni- tive science, 40( I): 172-201.
Mario Giulianelli. 2022. Towards pragmatic produc- tion strategies for natural language generation tasks. In Proceedings of the 2022 Conference on Empiri- cal Methods Natural Language Processing, pages 7978-7984 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Vera Demberg and Frank Keller. 2008. Data from eye- tracking corpora as evidence for theories of syntactic processing complexity. Cogmition, 109(2):193-2)))
Gabriel Doyle and Michael Frank. 2015. Shared com- mon ground influences information density in mi- croblog texts. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 1587-1596, Denver, Colorado. Association for Computational Linguistics.
Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? Evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. Association for Com- putational Linguistics.

---

Mario Giulianelli and Raquel Fernández. 2021. Analysing human strategies of information transmis- sion function of discourse context. In Proceed- ings of the 25th Conference on Computational Nat- ural Language Learning, pages 647-660, Online. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text de- generation. In International Conference on Learning Representations.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7038-7051, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Mario Giulianelli, Arabella Sinclair, and Raquel Fer- nández. 2021. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8271 -271, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
John J Godfrey, Edward C Holliman, and Jane Mc- Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceed- ings of the 1992 IEEE international conference on Acoustics, speech and signal processing- seressinn-Voluumee 1, pages 517-520.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength natural language processing in Python.
Laurence Robert Homn 1972 On the semantic prop- erties of logical operators in English. University of California, Los Angeles.
Adam Goodkind and Klinton Bicknell. 2018, Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics ( (CMCL 2018), pages 10-18, Salt Lake City, Utah. Association for Computational Linguisties.
Jennifer Hu, Roger Levy, Judith Degen, and Sebastian Schuster. 2023. Expectations over unspoken alterna- tives predict pragmatic inferences. Transactions of the Association for Computational Linguistics. To appear.
Jennifer Hu, Roger Levy, and Sebastian Schuster. 2022. Predicting scalar diversity with context-driven un- certainty over alternatives. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 68-74, Dublin, Ireland. Associa- tion for Computational Linguistics.
Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
Jeroen Antonius Gerardus Groenendijk and Martin Jo- han Bastiaan Stokhof. 1984, Studies on the Seman- fics of Questions and the Prugmatics of Answers. Ph.D. thesis, University of Amsterdam.
T. Florian Jaeger and Roger Levy. 2007. Speakers optimize information density through syntactic reduc- tion. In Advances in neural information processing systems, pages 849-856.
Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for needle in a haystack: A com- prehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the A Association for Com- putationaa Linguistics, pages 1059- 1075, Dubrovnik, Croatia. Association for Computational Linguistics.
Roni Katzir. 2007. Linguistics and philosophy, 30:669-690.
Structurally-definee alternatives.
John Hale. 2001. probabilistic Earley parser as psy- cholinguistic model. In Second meeting of the North Americun Chapter of the Association for Computa- tional Linguistics.
Frank Keller. 2004. The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 317-324.
Charles L Hamblin. 1976. Questions in montague En- glish. In Montague grammar, pages 247-259. Else- vier.
Dave F Kleinschmidt and T Florian Jaeger. 2015. Ro- bust speech perception: recognize the familiar, gen- eralize to the similar, and adapt to the novel. Psycho- logical review, 122(2):148.
Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 174-180, Baltimore, Maryland. Association for Computational Linguistics.
Daphne Koller and Nir Friedman. 2009, Probabilistic gruphical models: Principles and techniques. MIT press.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: I Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations.
Philip Hofmeister, Laura Staum Casasanto, and Ivan A. Sag. 2014. Processing effects in linguistic judg- ment data: (Super- Jadditvviyy and reading span scores. Language and Cognition, 6:111 145.

---

Joy Han Lau, Alexander Clark, and Shalom Lappin. 2015. Unsupervised prediction of acceptability judgements. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 1618-1628.
Tiago Pimentel. Clara Meister, Elizabeth Salesky, Si- mone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A currrsal-duration trade-off across and within the world's languages. In Proceedings of the 2021 Conference on Empirical Methods in Natu- rul Language Processing (EMNLP). Association for Computational Linguistics.
Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cogni- live Science, (1(5):1202-13 1241.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nlya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Steve Lawrence, C Lee Giles, and Sandiway Fong. 2000. Natural language grammatical inference with recur- rent neural networks. IEEE Transactions on Knowl- edge & Data Engineering, 12(01):12-1140.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Stephen C. Levinson. 2000. Presumptive Meanings: The Theory of Generalized Conversational Implica- ture. The MIT Press.
Roger Levy. 2008. Expectation-hased syntactic compre- hension. Cognition, 106(3):1126-1177
Mats Rooth. 1992. A theory of focus interpretation. Natural language semanticx, pages 75-116.
Roger Levy and T. Florian Jaeger. 2007. Speakers opti- mize information density through syntactic reduction. Advances in Neural Information Processing Sys- tems, volume 19. MIT Press.
Mats Rooth. 1996. Focus.
The handbook of contempo- rary semantic theory, ed. by Shalom Lappin.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Ca- rina Kauf, Eghbal A. Hosseini, Nancy Kanwisher. Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative mod- eling converges on predictive processing. Proceed- ings of the National Academy of Sciences, 118(45).
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pages 986 995.
John R Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge University Press.
Steven G. Luke and Kiel Christianson. 2018 The Provo Corpus: A large eye-tracking corpus with predictabil- ity norms. Behavior Research Methods, 50(2):826- 833.
John R Searle. 1975. A taxonomy of illocutionary acts. Language Mind, and Knowledge, 7.
Scott A. McDonald and Richard C. Shillcock. 2003, Low-level predictive inference in reading: the influ- ence of transitional probabilities on eye movements. Vision Research, 43(11))1735-177
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Philip Levy. 2022. Large-scale evi- dence for logarithmic effects of word predictability on reading time.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisit- ing the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963- 980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-423.
Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022. Structural persistence language models: Priming as a window into abstract language representations. Transactions of the Associ- ation for Computational Linguistics, 10:1031
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally Typical Sampling. Transac- tions of the Association for Computational Linguis- tics, 11:102-122 121.
Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3)302-319.
Byung-Doh Oh and William Schuler. 2023. Why Does Surprisal From Larger Transformer-Based Language Models Provide Poorer Fit to Human Reading Times? Transactions of the Association for Com- putational Linguistics, 11:3363300.
Harold Somers. 2005. Round-trip translation: What is it good for? In Proceedings of the Australasian Language Technology Workshop 2005, pages 127- 133, Sydney, Australia.

---

Antonella Sorace and Frank Keller. 2005. Gradience in linguistic data. Lingua, 115(11):1497-155)
of neural language models for human real-time com- prehension behavior. In Proceedings of the 42nd An- nual Meeting of the Cognitive Science Society, pages 1707-1713. Cognitive Science Society.
Robert Stalnaker. 1978. Assertion. In Pragmatics, pages 315-332. Brill.
Roel M Willems, Stefan L Frank, Annabel D Nijhof, Pe- ter Hagoort, and Antal Van den Bosch. 2016. Predic- tion during natural language comprehension. Cere- bral Cortex, 26(6):2506-251)
Katherine Stasaski and Marti A Hearst. 2023. Pragmat- ically appropriate diversity for dialogue evaluation. arXiv preprint arXiv:2304.02812.
Antoine Tremblay, Bruce Derwing, Gary Libben, and Chris Westbury. 2011. Processing advantages of lexi- reading and
Yang Xu and David Reitter. 2018. Information den- sity converges in dialogue: Towards an information- theoretic model. Cognition, 170:147-163.
cal bundles: Evidence from sentence recall tasks. Language learning, (2)2:669 613.
Shaorong Yan and T Florian Jaeger. 2020. Expecta- tion adaptation during natural reading. Language, Cognition and Neuroscience, 35(10):1394-44) 1422..
Geoffrey Underwood, Norbert Schmitt, and Adam Galpin. 2004. The eyes have it: An eye movement study into the processing of formulaic sequences. In Norbert Schmitt, editor, Formulaic Sequences: Ac- quisition, Processing and Use, pages 153-172. John Benjamins.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 00060..
Bob Van Tiel, Emiel Van Miltenburg, Natalia Zevakhina. and Bart Geurts. 2014. Scalar Diversity, Journal of Semantics, 33(1):137-175.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao. Jianfeng Gao, Jingjing Liu, and William Dolan. 2020. DIALOGPT: Large- scale generative pre- pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational System Demonstrations, pages 270-278.
Vivek Verma, Nicholas Tomlin, and Dan Klein. 2023. Revisiting entropy rate constancy in text. arXiv preprint 230.:::::00..00::
Linguistics:
Michael Wagner et al. 2005. Prosody and recursion. D. thesis, Massachusetts Institute of Technology.
Zheng Zhang, Leon Bergen, Alexander Paunov, Rachel Ryskin, and Edward Gibson. 2023. Scalar implica- ture is sensitive to contextual alternatives. Cognitive science, 47(2)
Sarenne Wallbridge. Peter Bell, and Catherine Lai. 2022. Investigating perception of spoken dialogue accept- ability through surprisal. In Interspeech 2022: The 23rd Annual Conference of the International Speech Communication Association, pages 4500- In- ternational Speech Communication Association.
A Language Models
For the dialogue corpora, we use GPT-2 (Radford et al., 2019), DialoGPT (Zhang et al., 2020), and GPT-Neo (Black al., 2021). For the text corpora, we use GPT-2 (Radford et al., 2019), GPT-Neo (Black et al., 2021), and OPT (Zhang et al., 2022). The text models are pre-trained while the dialogue models are fine-tuned for 5 epochs with early stop- ping on the respective datasets, using "
 
" as turn separator. Preliminary experiments on the pre-trained models show that 
 
 is the turn separator that yields lowest perplexity the dialogue datasets. For text models, using no sepa- rator is the option that yields the lowest perplexity. When generating out of context, we set x to be either the dialogue turn separator "
 
" or a white space for the text models.
Sarenne Wallbridge, Peter Bell, and Catherine Lai. 2023, Do dialogue representations align with perception? an empirical study. In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 3696-2713, Dubrovnik, Croatia. Association for Computational Linguistics.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Jason Wei, Clara Meister, and Ryan Cotterell. 2021. A cognitive regularizer for language modeling. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 10th International Joint Conference Natural Language Processing, pages 5191-5202, Association for Com- putational I. Linguistics.
LM validation: perplexity. Table 4 reports the perplexity of these models on the SWITCHBOARD and DAILYDIALOG test sets, as well as on the Wiki- Text test set (the CLASP dataset and the reading
Ethan Gotlieb Wilcox, Jon Gauthier. Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power

---

DailyDialog
Switchboard
WikfText
We experiment with & E [0.5, 0.75....... 5].
GPT-2 Small (124M)
7.34
11.86
25.62
Maximum surprisal captures the idea that a highly surprising element drives the overall sur-
GPT-22 Medium (355M)
6,03
10.50
19.69
GPT-2 Large (774M)
5.26
10.09
17.39
GPT-Neo 125M
7,39
12.54
25.37
prisal of an utterance:
GPT-Neoo 1.38
4.94
10.11
14.01
DialoGPT Small
7.94
12.50
Smaz (y|x) max[s(yn)]
(7)
DialoGPT Medium
6.53
10.96
DialoGPT Large
6,23
11.00
OPT 125M
17.80
22.68
46.85
Surprisal variance across an utterance has been defined in a number ways; we consider surprisal variance as the regression to the utterance-level mean and surprisal difference as the variability between contiguous token-level estimates:
14.88
21.46
12.58
20.30
27.45
Table 4: Language model perplexity results. The models tested on the dialogue datasets are finetuned for 5 epochs with early stopping: the models tested on WikiText are pre-trained.
N
Spariance Sy)riano:yy(x)) N
Σ (s(n)) - Smean (y)) n=
time datasets are too small to allow for robust eval- uation, but their style is sufficiently similar enough that of WikiText). Perplexity scores the low- est for the dialogue datasets. This is to expected as the dialogue models are fine-tuned. The perplex- ity of the pre-trained models on WikiText is line with stateoof-the-ar results; OPT obtains higher perplexity than GPT-2 and GPT-Neo, but still in an appropriate range.
(8)
N Σ n=
faifference (y|x)
((()) s(yn-1)l (9)
C Psychometric Predictive Power and Sensitivity Information Value Estimates
B Utterance-Level Surprisal
We study the extent to which our estimates of infor- mation value are affected by variation in three main factors: the alternative set size ([10, 20, .... 100)), the language model, and the sampling strategy. Figures 4 and 5 show Spearman correlation be- tween information value and psychometric data, averaged over subjects. These results complement Sections 5.1 and 5.2 the main paper.
Given an utterance y as sequence of tokens in context x, token-level surprisal can be defined as a(yr) - log y x). Multiple works have proposed quantifying utterance-level surprisal as functions of token-level surprisal (Genzel and Char- niak, 2002; Keller, 2004; Xu and Reitter, 2018; 8; Meister et al., 2021; Giulianelli et al., 2021; Wall- bridge et al., 2022). We compare the predictive power of information value number of these utterance-level surprisal aggregates.
D Intrinsic Robustness Analysis
Mean surprisal and total surprisal account for all token-level surprisal estimates with and without normalising by utterance length:
In Section 5.1, we evaluate the robustness of infor- mation value to parameters involved in the alterna- tive set generation in terms of its psychometric pre- dictive power. We additionally assess their intrinsic robustness by measuring the correlation between information values assigned target utterances by estimators with different parameter settings.
1 N
Smean (y|x) = Σ
ss (2n))
N
1=1
The parameters which we consider are alter- (5) native set size ((10, 20. 100)). the generative model, and the decoding strategy. Models and de- coding strategies each of the corpora described in Section 4.2, we compute the information value for the target utter- ances based on alternative sets generated under dif- (6) ferent parameter settings. Robustness is quantified through the distribution of the pairwise Spearman
Stotal (y) xx)
Σ [s())]]
detailed in Section 4.1. For
Superlinear surprisal posits superlinear effect of token-level estimates:
8 Ssuperlincar, (y|x)=
Σ
R=1

---

Switchbeard
Lexical information (meas)
Synractic informatio value (mens)
(menn)
Bark-Crem Cil--rrrolsannn
-0.33
1105
0.55
0.10
40.05
0.15
Lexical information value (minn)
Syntactic information value (tiin)
Scumatic information value (min)
Rusk Coretation
0.45
4.1
-0.90
-02
-3.0
40
60
70
99
100
30
40
10
100
30
40
50
10
100
Abomotive Set Size
Altemaiiv Set Size
Altenative Set Size
Dialcar1 Large
GPT2 Latge
LrPESe 1.15
DuaceP Mediun
GPT-2 Malizn
SPPNNcn 128M
Small
Small
Dailydialog
Lesical information value (mesz)
Synractic information value (mern)
Semantic informscion value (mecn)
-0.5
RankConcintes
0.0
015
0.10
0045
0.05
20
100
100
20
100
Lexiesl information value (minn
Syntactic information Value (min)
Sementic informatiin value (min)
Raik- Baik-Coone Connelaaa
0.1
41
0.51
-0.2
100
Lemastie Size
Allemative Sizz
Aliemative Size
DiavcTT Large
GPT Luge
GPTNoo 7BB GPTS0o 1253
DiakGP1 Melino
Medium
Scall
Clasp
exxical informstion valne (mecn))
Syntactic in information valne (mecon)
Semasntc formatioo valne (mwran)
-0.04
Rusk Raak Constatoo
0.00
0.000
0005
-8.05
0.025
-.10
10.99
-0.10
0.12
10
30
90
70
50
103
10
32
20
90
60
71
19
90
100
80
30
30
90
60
100
(mia)
(min)
(min)
Ruik-Corelans
0.025
0.10
-0.05
OPT 125M
1253
350M
GPT Swill
OPT L3B
Figure 4: Spearman correlation between information value and average acceptability judgements.

---

Lesical infommation value (mean)
Syntactic information value (menn)
Semantic information value (meun)
0.30
Rua-Conebniee
0.29
0.20
015
0.15
010
100
Letical Information value (min)
Syntnctic information value (minn)
Semantic information value [min)
161
Ruil Buik-Conelan Coneeabon
0.88
0.2
0.1
0.84
20
100
20
30
100
20
30
100
Aliemativee Sei
Aliemative Stine
Allernaivee
Lxgg
GPT2 Mediun
CPT 1250
OPT 390M
GPT Smoll
OPT
Brown
exxical forommaiiin value (mvan)
Syntactic infermation value
Semantic informe value (mean)
Bark Bark-Coenlann Cievelatin
01
0.65
Lexical infommation value (min)
Syntactic information value (min)
Semantic information value (min)
Raak Raak-Cundiin Conelanee
0125
0.100
6075
0.64
-0.4
10
20
30
10
40
60
71
30
100
11
211
30
31
81
50
30
10
30
100 40
60
101
W0
Allecantive Set Size
Alcenotive Set
5ize
Allenative Set Size
OPFS Madiu
GPVNNce 125M
OPT 180M
(PP52 Smaal
Figure 5: Spearman correlation between information value and average reading times (length- normalised).

---

correlation p obtained between the information val- ues for each parameter setting; strong pairwise cor- relation indicates that information value is robust to the varying parameter. Results are displayed in Figures 6 and 7.
with z-score 3 when the distribution of reading times is modelled as log-linear (following Meister et al.. 2021).
Control predictors. Following Wilcox et al. (2020), we evaluate each model relative to base- line model which includes only control variables. Control variables are selected building on previous work (Meister et al., 2021): we include solcly an intercept term as a baseline for acceptability judge- ments and the number of fixated words for reading times. Meister et al. (2021) report similar trends when including summed unigram log probability sentence length baseline predictors of acceptabil- ity judgements, and word character lengths or word unigram log probabilities for reading times. For reading times, we also test sentence length as a pre- dictor but baseline models that include, instead, the number of fixated words (readers sometimes skip words while reading) achieve higher log bog-likeihhod..
Information value defined as lexical, syntactic, and semantic distance becomes highly robust as alternative set size increases; mean correlations be- tween decoding strategies for each model converge towards perfect correlation as alternative set size increases. This pattern holds for all datasets. De- coding strategies do not produce much variation across correlations, regardless of alternative set size (see confidence intervals in Figures and 7). For mean-based definitions and models, information values generated from different decoding strategies correlate at strengths 0.8 from sets with fewer than 50 alternatives. Although their mean correla- tions still converge 0.9, the dialogue datasets are slight exceptions.
As expected, correlations between parameter set- tings for min-based distances are more variable. Although they converge to weaker correlations as alternative set size increases when compared to mean- mean-based distances, we still find strong to very strong correlations between decoding strategies for large alternative sets across all models.
G More Derived Measures of Information Value
We also tested the following measures derived from information value but found them be less predic- tive than those in the main paper.
Expected information value. The expected dis- tance of plausible productions given context from the alternative set:
E Selecting the Best Information Value Estimators
For each corpus and each surprisal type (lexical, syntactic, semantic), we select the estimator that yields the best Spearman correlation with the psy- chometric data. An estimator is combination of model, sampling algorithm, and alternative set size. Psychometric data are in-context acceptabil- ity judgements for DailyDialog, Switchboard and Clasp, and the mean of all word reading times in a sentence for Brown and Provo. Table shows the best estimators.
B(I(||x=x)) = Ea = (x,==,
(10)
We assume a uniform probability distribution over alternatives. This quantifies the uncertainty over next utterances determined by the context alone. Because the alternative set Ag is the set of plausi- ble productions given x, in practice, we compute expected information value using only one alterna- tive bot-both in the expectation EaEA, and in the distance calculation d(y. Ar).
F Linear Mixed Effect Models
Deviation from the expected information value. The absolute difference between the information value for the next utterance y and the expected information value for any next utterance:
In this section, we include further details about the linear mixed effect models used in Sections 6 and 7. All results for single information value predictors are in Table 6. Results for the comparison with surprisal and joint models are in Table 2.
D(Y = x = !I = x
Response variables. For PROvO, we use the to- tal dwell time, i.e., the cumulative duration across all fixations on given word. We filter away any ob- servation that contains 'outlier' words, i.e., words
-(I (11)
This quantifics the information value of an utter- ance relative to the information value expected for

---

Switchbeard
Lexical (meen)
(menn)
(meuc)
Bunk Crerelsan
0:35
06
0.30
Lexical afforaaiion valur (mnin)
Syntactic information value (min)
Semsrctc informmtion value (min)
69
Baak-C Colk-Coottttooo
0.80
20
10
40
70
90
30
40
70
103
Altecative Size
Altecaatire Set Size
Altiivtire Set Size
Dialccr1 Large
GPTE Latge
LrPE-ee 1.15
DisceP Mediun
GPT-2 Malium
SPPENNn 128M
Small
Small
Dailydialog
Loxical information value (meun)
Syntuctic information value (meen))
Semuntic information value (meun)
69
0.5
Baak-Cnnciite
03
1090
677
30
10
20
103
Levical information value (min)
Synsectic infommation value (min)
Semartic information value (min)
Sarii Conecoon
0.00
0.51
0.5
0.50
0.6
Allemativve
Sel Stne
Allemativv Sct Siet
DiavOTT Eirge
GPT Luge
GPTNco ..B GPTSoo 1253
Medino
Mediam
Clasp
Lexical information value (mesn)
Spotaetic information value (mean)
Semantic informstion value (mean)
Ruak Raak Cocotatoo
0.55
10515
0.03
1990
005
0035
0.83
0.30
0.000
0.35
20
70
60
88
10
18
100
30
20
30
50
70
90
100
10
33
20
10
80
50
101
Lexical
0.8
0.
0.8
0.56
OPT 125M
GPT2 Smill
OPT L38
Figure 6: Intrinsic robustness evaluation acceptability judgements corpora.

---

Provo
Lesical infommation value (mean)
Syrttcticc information value (mean)
Semandic information value (mean))
Sank Bank- Con--Conllninn
0.90
0.5
0.90
0.75
20
60
90
70
80
90
100
20
30
40
50
70
50
901
100
30
30
31
50
75
50
90
103
Lexioal
Syntactic (mnin)
(min)
0.9
Rusk Cocredatin
0.54
0.3
0.7
0.6
0.3
0.15
100
100
OPT
GPF22 Sarill
OPT 1.38
Lexical infvemat ion value (mesn)
Spstaetie infirmation value (menn)
Semraniic infarmatiion valne (menn)
Radko Conotetto
0.90
D19
1150
0.5
035
0.35
0.75
0:8
20
100
30
50
100
30
55
103
Lexical Information value (min)
Syntuctic information value (minn)
Semantic information value (min)
0.50
0.7
0.55
066
20
50
20
50
100
20
30
50
70
50
100
20
30
59
50
100
Allemstivve Set Size
Allemative
Stzze
Allernative Sel Sio
Lxgg
GPT2 Mediun
GPTNeo 125M
OPT 390M
GPT Smol
OPT
Figure Intrinsic robustness evaluation on reading times corpora.
Corpus
Level
Metric
Summary
N
Language Model
Sampling
p
Lexical
Bigram
Min
70
DialoGPT Medium
Temperature 1.25
-0.436*
SWITCHBOARD
Syntactic
POS bigram
Min
100
DialoGPT Small
Ancestral
-0.440*
Semantic
Cosine
100
DialoGPT Large
Temperature 1.25
-0.702
Lexical
Unigram
Min
80
DialoGPT Small
Ancestral
-0.383*
DAILYDIALOG
Syntactic
POS trigram
Min
90
DialoGPT Large
Temperature 1.25
-0.359*
Semantic
Cosine
Min
100
GPT-2 Large
Nucleus 0.9
-0.584*
Lexical
Trigram
Min
90
GPT-2 Large
Temperature 1.25
-0,210*
CLASP
Syntactic
POS Bigram
100
GPT-2 Large
Nucleus 0.95
-0.234*
Semantic
Cosine
Min
90
OPT 1.3B
Temperature 0.75
-0.221
Lexical
Unigram
Min
10
OPT 125M
Typical 0.3
0.379*
PROVO
Syntactic
POS Trigram
Min
10
GPT-2 Small
Nucleus 0.95
0.421
Semantic
Fuclidean
Min
100
OPT 125M
Nucleus 0.95
0.181
Lexical
Bigram
90
GPT-2 Small
Typical
0.223*
BROWN
Syntactic
POS Trigram
Mean
10
GPT-2 Medium
Typical 0.3
0.185*
Semantic
Cosine
100
GPT-Nco 125M
Nucleus 0.95
0.048
Table 5: Best information value estimator per corpus and metric. Spearman correlatii coefficients p. statistical significance (p < 0.001) is marked with star. The highest correlations per dataset are in bold; estimators (a combination of set size N. model, and sampling strategy) that generate them taken as the 'best estimators' for that corpus and are used in Sections 6 and 7.

---

SWITCHBOARD
DAILDDIAOG
CLASP
PROVO
BROWN
Summ,
Level
Metric
ALoglik
ALogLik
g
ALogLik
ALogLik
ALogLik
Unigram
0.594
Lexical
Bigram
-133
4,687
2,761
7.179
-1,573
2.717
1,976
10.971
1.622
9.076
Trigram
2.401
8,315
-1844
7.089
-25144
5.857
1.982
12.169
1.891
11.974
Mean
POS Unigram
0,204
0.505
3.399Me
6.707
0.914
0.106
1,902
8.413
0.958
6.627
Syntactic
POS Bigram
13.861
POS Trggrmm
-0.159
2,488
0.767
2,505
-2.011
2.274
5.527
21.798
1445
8.194
Cusine
8.664
29.0341
-6. 6
21.207
-1.235
0.030
0.237
6.661
0.714
6.633
Semantic
Eaclidean
3.665
29. 29.263
-7.11
21.994
-1.535
0.617
0.221
6.864
0,766
6.833
Unigram
4x454
10.244
2.39
Lexical
Bigram
-1017
1,629
4.614**
10.876
-1,937
1,639
1.882
9.490
2.121
8.426
Trigram
1.774
3.396
-1969
3.311
-2.757
3.714
1.997
10.337
3.689*
13.913
Min
Unigram
-0.52
0,356
1.985
-2.9319
5,45-
21.187
1.947
8.633
Syntuctic
POS Bigram
1,052
0.901
2.933*
5.067
-S.356
13.539
4,494**
16.292
1:404
6.854
POS Trigram
0.758
0.993
-3.26*
5.732
-3.104*
4.124
4.956
18.394
1,362
6.706
Cosine
-9.88
34.204
-9.01
30,408
-1,982
0.979
0.548
6.476
0.661
6.164
Semantic
Euclidean
*7
-8.901 SM
6.020
Table 6: Results of single-predictor linear mixed effect models: fixed effect coefficients 3 and AlogLik. Statistical significance of fixed effects marked with one (p < 0.01) or two stars (p < 0.001). Information value estimates are obtained according to Equation For each corpus and each level (lexical, syntactic, and semantic), the best ALogLik is marked in bold. These are the level-specific metrics used whenever we talk about 'best predictors' the main paper.
plausible productions given x. An analogous no- tion is the deviation of surprisal from entropy. The token-level version this forms the basis of the local typicality hypothesis (Meister et al., 2023).
Expected context informativeness. The pected informativeness of context æ is the reduction in information value contributed by with respect any plausible continuation:
IBC((Y - y;; = E((() -y))
-E(((Y [y) - -x))
(12)
This quantifies the extent to which context re- stricts the space of plausible productions. An anal- ogous notion the expected pointwise mutual in- formation between X æ and Y, where the value fixed. Similarly to out-of-context infor- mation value, out-of-context expected information value E(I(Y= (0(Y=))) -)) is computed with respect to the alternative set A..


Reviews:
[{"review_id": "aaEhWCUVzc", "reviewer": "Reviewer_fDKN", "summary": "", "questions": "In figure 1 caption, it says that 11 sampling strategies are used while in the main text, only 6 sampling methods are mentioned, it is necessary that author(s) give an explanation on that.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We answer your questions below.\n\n1. We think information value is a stronger predictor of dialogue acceptability judgements because, when estimated via semantic distance metrics, (1) it can more robustly identify semantically incoherent dialogue utterances and (2) it can capture expectations over linguistic signals at higher levels of abstraction, such as speech act types (Austin, 1962; Searle 1969, 1975), more explicitly than token-level surprisal aggregates. We have mentioned these two reasons in lines 496-500 and 596-602, but we agree they can be given more prominence! We will discuss them more explicitly in the camera-ready, including the citations to speech act theory.\n2. We present our hypotheses about the predictive power of different linguistic levels in the introductory paragraphs of Sections 6.2 (lines 487-504) and 6.3 (lines 540-552). These are based on linguistic and psycholinguistic theory regarding the potentially different processing requirements of the different tasks studied in this work, but also on the construction of the corresponding datasets (e.g.lines 500-504). They are largely confirmed by our analysis.\n3. Except for ancestral (unbiased) sampling, all other sampling algorithms deployed (temperature sampling, nucleus sampling, and typical sampling) bias away from the underlying conditional distribution of the language model and make the resulting probability mass function hard, or impossible, to characterise. This is why we do not see any principled reason to restrict the set of sampling algorithms. *In practice*, we do observe some variability among sampling strategies (see confidence intervals in Figure 1) yet we find that this variability has no impact on predictive power (see again Figure 1), so we select only one sampling strategy per corpus for our analyses in Sections 6 and 7 (lines 417-421). \nBeyond the confines of this paper, we agree that continued analysis of the differences between sampling strategies is important. We have presented our preliminary findingsin Appendix E (intrinsic evaluation) where we evaluate the variability of alternative set sizes for each sampling strategy. We think your suggestion of making a statistical report about the distributions of sentences generated via different sampling algorithms is an interesting piece of follow-on work. We will publicly release our large dataset of generations to encourage this line of research.\n\nRegarding your question about the caption of Figure 1: we use 4 sampling methods with varying parameter configurations: 1 configuration for ancestral sampling (no parameters here), 2 for temperature sampling, 4 for nucleus sampling, and 4 for typical sampling—for a total of 11 “sampling strategies”.\n\nLastly, thank you also for your presentation improvement suggestions. We will consider extending the abstract in the camera-ready, and we will include the English-only limitation in the respective section.\n\n———  \nAustin, J.L., 1962. _How to do Things with Words_, 2nd edition, J.O. Urmson and M. Sbisá (eds.), Cambridge, MA: Harvard University Press.\n\nSearle, J., 1969. _Speech acts: an essay in the philosophy of language_. Cambridge: Cambridge University Press.\n\nSearle, J., 1975. A taxonomy of illocutionary acts. In K. Gunderson (ed.), _Language, Mind and Knowledge_, Minneapolis, MN: University of Minnesota Press, 344–369.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.Solid and complete overview of previous relevant literature is highly appreciated in term of long paper.\n\n2.The experiments performed in this paper is thorough and have taken relevant previous works into full consideration.\n\n3.Although the usage of distance is not a new idea, the way of using it to represent information value instead of using entropy value is interesting and according to the experiment results, quite promising.\n\n4.The predictability of utterance is evaluated from three perspectives: lexical level, syntactic level and semantic level, which is linguistically thorough and coherent.\n\nOverall, this is an information-intense while interesting paper and I found no strong reason to reject it, it is more about either having it in finding or main venue.", "reasons_to_reject": "1. Have the author(s) thought about the reason why, information value is \"stronger predictor\" for dialogue(Complementarity in page 7 or discussion in page 8), is there any already existing linguistic theory which could explain it. If so, adding that will make this one a stronger paper.\n\n2.It turns out to me that different information value serves as a strong predictor among the 5 chosen corpora, for example, in PROVO it is the syntactic information value. Again have the author(s) already had a potential hypothesis for this phenomenon?Is it practical to do a linguistic analysis on this 5 corpora to find the reason?\n\n3. Is it possible that by increasing the set size of A_(x),  the generated sentences sampled from \"ancestral sampling\" could have already covered some/all of the samples from other sampling methods, e.g., \"temperature sampling\"? As far as I know, both of the two mentioned  sampling methods are based on the conditional probability, while typical sampling is a comparatively new sampling method which could cut off the dependence on the conditional probability to some degree and helps to generate more creative and diversify next sentences instead of entering receptive loop(Meister 2023).\nBased on that,I would suggest the author(s) making a statistic report about the distributions of generated sentences from different sampling methods and maybe then making a selection of just two representative sampling methods based on the observation. Also a reference to temperature sampling is needed.", "questions_to_authors": "In figure 1 caption, it says that 11 sampling strategies are used while in the main text, only 6 sampling methods are mentioned, it is necessary that author(s) give an explanation on that.", "paper_topic_and_main_contributions": "This paper tests the predictability of utterances, which could be considered as an incremental work toward ERC/UID theory.\nOne of the merits of this work is that, it puts forward a new method to measure the predictability of utterance(or beyond it), which is based on the distance among the \"ground truth\" next utterance and the \"candidate sentences\" generated by LLM(the distance measurement is cosine similarity score according to my reading). \nThe effectiveness of the new method is further tested by two types of experiments: acceptability measurement and reading time measurement based on eye gaze tracking.  \nBased on the experiments, this paper further proves the correlation between Information Value(represented as distance in three different linguistic aspect, namely syntactic, lexical and semantic) and acceptability/predictability in an utterance level, as well as a more advanced level, i.e., dialogue act level.\nOverall, this paper introduces a new methodology, in which its potential usage on the evaluation of NLG could be expected.\nIt orients itself quite well in the direction of computationally-aided linguistic analysis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Below are the two suggestions related to presentation improvements:\n\n\n1.The abstract should be definitely improved so that the readers could grasp the skeleton of the story of the paper better and more easily, especially in term of such long paper.\n\n2.One of the important concerns towards the selected corpora is that, they are all English Corpus. It is necessary to mention this point in the limitation section, i.e.,  about the potential consequence of using only English language data."}}, {"review_id": "i4DPwKBW91", "reviewer": "Reviewer_TJwo", "summary": "", "questions": "A Is there a motivation behind the terminological choice “information value”? This term would suggest a close relation to information theory, but as far as I can tell there is no such relation described in the paper.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions. We try to address them below point by point.\n\n> I have some concerns regarding the information value metric as presented in Sec 3. In particular, it is unclear to me how information value is a probability distribution. If it is, I assume you would mean to normalize your distance metrics such that your function fulfills the axioms of a probability distribution.\n\nGiven a context $x$, an utterance $y$, a probability distribution $p_{Y|X=x}$ prescribed by a language model, and a choice of distance metric $d$, $d(y, Y)$ is a real random variable that captures the distribution of distances between $y$ and any output drawn conditionally from $p_{Y|X=x}$. We propose sampling a fixed number of alternatives, which make up the alternative set $A_x$, and define information value as the distribution of distances between $y$ and all the alternatives in $A_x$ (Eq. 1). The word \"distribution\" here means… a distribution, i.e., “simply a collection of data, or scores, on a variable” (Urdan, 2016); in our case the variable is distance (cosine distance, n-gram overlap, or POS n-gram overlap). In other words, information value is not a probability distribution (with pdf, cdf, etc.) and indeed this is never stated in the paper.\n\n> Furthermore, it seems to me that the number of alternatives in your alternative set and how these are sampled would have a significant effect on your distribution, which I miss a discussion on.\n\nThis is certainly true. As such, we have a subsection (Section 5.2) as well as two appendices (B and E) that investigate the sensitivity/robustness of our information value estimates to estimator parameters (i.e., the number of alternatives, the language model, and the sampling strategy). \n\n> Overall, I find this key section of the paper hard to read and believe it would benefit from another round of revision.\n\nThank you for the feedback, we will try to clarify all the aspects above using the additional space available in the camera-ready.\n\n\n> Furthermore, I would have appreciated a more careful discussion on how this type of approach solves some of the problems with previous methods. In the introduction you write that aggregating token-level surprisal estimates leads to different realizations of the same concept or communicative intent competing for probability mass, but why is this a problem? And how does your method solve it? As far as I can tell there is no technique used here that in some way guarantees that the different utterances in the alternative set would be distinct in terms of communicative intent.\n\nIn the Introduction, we mention two downsides of token-level aggregates: (1) different realisations compete for probability mass, (2) different dimensions of predictability are conflated. We acknowledge that we give more weight to our solution to the second issue throughout the paper (with information value allowing us to disentangle different dimensions of predictability thanks to the use of interpretable distance metrics). Regarding the first issue, consider the following case. Let $x$ be a dialogue context and $y$ the true upcoming utterance “I gotta go”. Say the language model assigns a probability p = 1e-7 to $y$ (e.g., because the use of “gotta” is not in line with the formality level of the dialogue so far) and that it assigns the following probabilities to alternative continuations: p(“I have to go”) = 0.001; p(“I must go”) = 0.0005; p(“I need to go”) = 0.0005. The LM’s surprisal for “I gotta go” will arguably be too high in this scenario. Now consider using information value for the same case. Utterances like “I have to go”, “I must go”, and “I need to go” will be much more likely to be sampled in the alternative set and, in turn, the semantic predictability of “I gotta go” more likely to not be underestimated (its lexical predictability may still be rather low, which again attests to how information value addresses problem 2). Now, due to the stochasticity of the sampling algorithms, to the limited size—in practice—of alternative sets, and to the fact that the sampling algorithms we test are themselves token-level, it might still be possible that none of the three higher likelihood alternatives might be sampled because of their “competition” for probability mass. Here, utterance-level decoding algorithms (e.g., Eikema and Aziz, 2022; as cited in the paper) might be of help. Furthermore, we do not aim to guarantee that alternatives are distinct in terms of communicative intent as certain intents are simply more likely for particular contexts, but rather we aim to account for variation at this level of abstraction more explicitly than is currently done by aggregates of token-level surprisal estimates. There exist some very recent approaches for monitoring the variety of intents in the LM’s output space (e.g, Kuhn et al. 2023; Stasaski and Hearst 2023; as cited in the paper); we think they are a promising solution to this problem. In any case, *in the camera-ready, we will tone down our claims about information value solving problem 1 and discuss them more carefully.* \n\n> Is there a motivation behind the terminological choice “information value”? This term would suggest a close relation to information theory, but as far as I can tell there is no such relation described in the paper.\n\nWe chose “information value” to signal the connection of our measure of predictability to the information-theoretic notion of information content or surprisal, while avoiding the terms “information content” and “surprisal”. The relation between predictability and information (Hale, 2001; Genzel and Charniak, 2002; Jaeger and Levy, 2007, as cited in the paper) is described in the first paragraph of the Introduction (lines 27-37). Moreover, the term “information value” resonates with attempts to develop a quantitative measure of information that go beyond the probability of outcomes (Howard, 1966). We also considered referring to our measure of predictability as “surprise”, resonating, this time, with the vocabulary of active inference and the free variational energy principle (Friston, 2010; Friston et al., 2017).\n\n> Presentation Improvements\n\nFinally, we appreciate your feedback regarding presentation improvements. We provide references to a number of previous results in reading time predictions in the Reading Times section (4.2.2) but will aim to present these more clearly in a camera-ready version, as well as clarify the other points of definition that you highlighed (“low-dimensional properties”, and “uniform distribution”).\n\n———   \nUrdan, T.C., 2016. _Statistics in plain English_. Routledge.\n\nHoward, R.A., 1966. Information value theory. _IEEE Transactions on systems science and cybernetics_, 2(1), pp.22-26.\n\nFriston, K., 2010. The free-energy principle: a unified brain theory? _Nature Reviews Neuroscience_, 11(2), pp.127-138.\n\nFriston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P. and Pezzulo, G., 2017. Active inference: a process theory. _Neural computation_, 29(1), pp.1-49.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental setup is sound and has a high degree of breadth, experimenting with several language models, distance metrics, alternative set sizes and sampling techniques. Measuring predictability by making use of alternatives is an interesting idea, appears novel and provides insight into what type of information is considered under different aspects of human language processing.", "reasons_to_reject": "~~I have some concerns regarding the information value metric as presented in Sec 3. In particular, it is unclear to me how information value is a probability distribution. If it is, I assume you would mean to normalize your distance metrics such that your function fulfills the axioms of a probability distribution. Furthermore, it seems to me that the number of alternatives in your alternative set and how these are sampled would have a significant effect on your distribution, which I miss a discussion on. I assume that the event space of your distribution is the sentences in the alternative set but the interpretation is not entirely clear to me. Overall, I find this key section of the paper hard to read and believe it would benefit from another round of revision.~~\n\nFurthermore, I would have appreciated a more careful discussion on how this type of approach solves some of the problems with previous methods. In the introduction you write that aggregating token-level surprisal estimates leads to different realizations of the same concept or communicative intent competing for probability mass, but why is this a problem? And how does your method solve it? As far as I can tell there is no technique used here that in some way guarantees that the different utterances in the alternative set would be distinct in terms of communicate intent.", "questions_to_authors": "A Is there a motivation behind the terminological choice “information value”? This term would suggest a close relation to information theory, but as far as I can tell there is no such relation described in the paper.", "paper_topic_and_main_contributions": "This paper introduces the notion of information value, which is a measure for the predictability of a sentence. It is distinguished from other measures based on surprisal in that it accounts for alternative plausible sentences: information value is a distribution over the distance between the incumbent sentence and a set of plausible alternative sentences given the context, and precise measures are obtained as summary statistics of this distribution. They perform a grid search type study to find how information value correlates with and predicts acceptability judgements and reading times. Their results indicate that information value provides a strong predictor of acceptability judgements, particularly when using a semantic distance metric. Overall I think this is an interesting experimental study that has some room for improvements when it comes to presentation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "L106 seems to misrepresent Hale (2001) as a work that uses Markov chains for estimation of surprisal; he uses an Earley parser."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Sec 5.1 discusses correlations of the information values with psychometric variables compared to those of previous work using surprisal. It would be useful to have a table displaying the results of previous works as well as your own experiments using surprisal with more details than Table 1. The reading times discussion further lacks any references to previous work that the reader can refer to for comparison. \n\nL196 you may want to refer to a specific paragraph\nL206-8 is confusing to me. What low dimensional properties?\nL229-30 it would be useful to see an explanation on what is meant by “under a uniform distribution of alternatives”.\nL232-4, is that a sneak peek of the results?\nL248 I would change notation since Y is not related to the target utterance y."}}, {"review_id": "NLQx2aL4Hv", "reviewer": "Reviewer_i1qv", "summary": "", "questions": "This has nothing to do with my evaluation, but I think your weighted distance measure might be equivalent to surprisal (log prob) of a multivariate Gaussian (exponential) probability model centered on the mean semantic/syntactic/lexical vector, where the distribution of distances is doing the job of Mahalanobis distance (distance in terms of standard deviations) in a Gaussian.  Is it?  If so, it might be an intuitive way to explain why this new information measure adds and subtracts like Shannon information.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interesting comments!\n\n**Token-level comparison of information value vs. surprisal.** This paper focuses on utterance predictability (and utterance-level psychometric variables) but in principle, our measure of predictability can be extended to the token level. This can be done, for example, by measuring the semantic distance between contextualised word representations. We are very curious to see, in follow-up studies, how token-level information value would fare in comparison to token-level surprisal.\n\n**Surprisal of a multivariate Gaussian.** We need to think more about this connection, but thank you for the hint :)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This new, more interpretable measure of information may allow finer-grained exploration of the relationship between expectation and human reading time.\n\nI did not notice any issues with the evaluation.", "reasons_to_reject": "The predictor is only evaluated against utterance-level surprisal, but one advantage of surprisal over this new predictors is that it can be estimated at the token level and might be a better predictor when estimated that way, whereas I'm not sure this new predictor can.  If some of the information value measures can be estimated at the token level, are they as good?", "questions_to_authors": "This has nothing to do with my evaluation, but I think your weighted distance measure might be equivalent to surprisal (log prob) of a multivariate Gaussian (exponential) probability model centered on the mean semantic/syntactic/lexical vector, where the distribution of distances is doing the job of Mahalanobis distance (distance in terms of standard deviations) in a Gaussian.  Is it?  If so, it might be an intuitive way to explain why this new information measure adds and subtracts like Shannon information.", "paper_topic_and_main_contributions": "This paper describes a new measure of information to replace Shannon information (surprisal).  Evaluations find it to be complementary to surprisal.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WduJp97Ckx", "reviewer": "Reviewer_RsrJ", "summary": "", "questions": "- Why is a uniform mean used to reduce d(y, A_x) to a number? Why not weighted by probability, which would seem most natural? Or distance to the most probable member of A_x, etc.\n- How long are the alternative utterances?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, questions, and suggestions!\n\n**Large number of free parameters.** We will consider moving Table 5 to the main paper and including a more exhaustive version thereof in the Appendix. \n\n**The framing with respect to surprisal theory.** We think of surprisal (or Shannon information content) as a measure of predictability, where “predictable” is used out of jargon for an event that is regarded as likely to happen. Furthermore, we see surprisal theory as asserting that the processing effort incurred by a linguistic signal is proportional to the predictability of that signal in context as captured by the specific measure of surprisal. What we are proposing is an alternative measure of predictability—not of surprisal (that would indeed not make much sense, as it would be as stating that information value is a measure _of a measure_ of predictability). We see information value as a new way to measure predictability, next to surprisal.\n\n**Why a uniform mean?** For simplicity, we chose to treat generators as sources of alternative sets and to disconnect information value from the language model probabilities. We agree it would also be interesting to study other reduction strategies, such as the weighted mean, the distance from the most probable alternative, or strategies that rely on the clustering of alternatives (e.g., Kuhn et al., 2023). We will include this point in the discussion.\n\n**How long are the alternative utterances?** The mean and standard deviation of distributions of length as number of words for all alternatives generated by the best estimator for each dataset (as described in lines 415-425) are as follows:\n- Switchboard: 13.90 +- 15.01 (note that this distribution is heavily skewed towards short utterances due to the high proportion of backchannel responses in conversational speech)\n- Dailydialog: 23.53 +- 4.16\n- Clasp: 18.72 +- 12.24\n- Provo: 18.87 +- 12.38\n- Brown: 17.99 +- 11.71  \n\nWe will include these statistics in the camera-ready, probably in the Appendix for reasons of space. In addition, we will release our large set of generated alternatives as we believe characterising these to be an interesting direction to explore.\n\n**Missing References and Presentation Improvements**\nWe will include Smith & Levy (2011) and further specify the connection with the ERC and UID principles in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new concept that can be used in models of language processing.\n- The proposed measure is easy to calculate and understand.\n- The results show that the proposed measure is effective (modulo concerns about multiple comparisons below).\n- The paper is well-connected with the processing literature.", "reasons_to_reject": "- The large number of free parameters (which distance metric, which aggregation method, mean or variance for surprisal) is a bit unsettling. I think a more comprehensive reporting of the results in the main text than is shown in Table 1 is in order.\n- The framing with respect to surprisal theory should be adjusted. Currently the introduction presents information value as an alternative theory of *predictability*, but it seems that it’s really an alternative theory of processing difficulty as reflected in acceptability judgments and reading times, contrasting with surprisal rather than measuring the same thing in a different way. In some parts of the intro it seems that information value is being proposed as an alternative way to measure *surprisal* (eg. “Proper estimation of the surprisal of an utterance would require computing probabilities over a … event space”), but it is not a measure of surprisal: I don’t see any formal relationship between d and surprisal. Rather, it seems that d is a new, separate and complementary predictor of processing difficulty.", "questions_to_authors": "- Why is a uniform mean used to reduce d(y, A_x) to a number? Why not weighted by probability, which would seem most natural? Or distance to the most probable member of A_x, etc.\n- How long are the alternative utterances?", "paper_topic_and_main_contributions": "This paper proposes “information value” as an alternative to surprises a measure of predictability as it relates to human acceptability judgments and reading times. Information value is based on the distribution of distances between a token y and a set of alternative tokens A_x in context x generated from a language model. Three different distance metrics are tried, as well as two ways of aggregating the distances into a single number (mean and min). Information value is found to predict acceptability ratings better than surprisal, and is about as good as surprisal at predicting reading times (and complementary to it). The authors also examine some derived measures such as non-contextual information value.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Smith & Levy (2011): https://escholarship.org/uc/item/69s3541f This is relevant to the motivation for information value as a kind of predictability which isn’t captured by token surprisals. In this paper they find that human-predictable words are influenced not only by corpus frequencies but also by semantic similarity. This seems relevant to semantic information value.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 186: Should be alternative sets A_x, not |A_x|\nLine 109-113: “and serves as a foundation for quantitative principles of language production and comprehension such as ERC and UID” — Only if the linking function from surprisal to processing difficulty is superlinear.\n"}}]

Author rebuttal:

Meta-review: This paper combines fundamental linguistic and cognitive questions with sound modeling. It is well situated in previous literature and makes insightful and sound contributions.
Decision: Accept-Main