TEMPERATURE-SCALING SURPRISAL ESTIMATES IMPROVE FIT TO HUMAN READING TIMES – BUT DOES IT DO SO FOR THE “RIGHT REASONS”?

Authors: tong  liu
iza  škrjanec
vera  demberg
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-08 19:23:29

Abstract: A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word’s negative log probability in context. However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. One reason could lie in the fact that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. 
In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. Firstly, we show that calibration of large language models typically improves with model size, i.e., poorer calibration cannot account for poorer fit to reading times. Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89\% improvement in delta log likelihood), across several reading time corpora. Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens.

Main paper: 
 introduction: 
 In psycholinguistics, a key finding is that words with higher surprisal (= negative log probability of the word in context) require more time for processing (Hale, 2001; Levy, 2008) . Numerous studies provided experimental evidence supporting this theory, demonstrating that surprisal is a powerful predictive measure of processing complexity (e.g., Demberg & Keller, 2008; Wilcox et al., 2020; 2023; Shain et al., 2022) , and that the relationship between surprisal and reading times (RTs) indeed seems to be linear (Smith & Levy, 2013; Wilcox et al., 2020; Shain et al., 2022) .However, prior work implicitly made the assumption that human predictability estimates would be similar to the actual probability of a word occurring in a given context, and that therefore, surprisal values estimated from models that achieve lower perplexities should also approximate human processing difficulty better (Goodkind & Bicknell, 2018; Merkx & Frank, 2021) .Recent research has however found that this is not true -surprisal values from very large LLMs provide in fact a very poor fit to reading times. Oh & Schuler (2023) hypothesize that this might be due to LLMs being "too confident" in their estimates of rare named entities compared to humans, thanks to their manifold larger exposure to data and greater memory capacity compared to humans. Furthermore, work on NLP applications like question answering has reported that probability estimates from pretrained language models are often overconfident, i.e. they are higher than the ground truth probability (Si et al., 2022; Kumar, 2022) . These findings hence beg the question whether current LLMs are well-calibrated with respect to "objective" word occurrence probabilities. Relat-edly, we ask whether LLM probability estimates are overconfident compared to human estimates (as observed in reading times).One approach to address calibration problems is to use temperature scaling, as done e.g., in vision tasks (Guo et al., 2017; Hendrycks et al., 2019) . Temperature-scaling with a temperature T > 1 has the effect that the probability distribution is flattened such that it becomes more similar to a uniform distribution. Temperature-scaling hence incorporates uncertainty into the probability estimates from LLMs.We note that the idea to work with flattened distributions instead of the original probability distributions from LLMs is also related to contextual Rényi Entropy as discussed by Pimentel et al. (2023) . Pimentel et al. find that Rényi entropy with α = 0.5 can explain RTs better than Shannon entropy (α = 1). Similarly to temperatures greater than 1, α values lower than 1 have the effect of making the distribution more similar to a uniform distribution. We'd like to point out however that our work and that of Pimentel et al. (2023) differ in the assumed underlying reasons for why a slightly flattened distribution may be more suitable, and whether this change in distribution is applied when calculating surprisal vs. when calculating entropy.Our experimental results show that scaling probabilities can largely improve the fit to reading times in all 12 settings (3 corpora × 4 neural LMs). Our contributions are summarized as follows: (1) We propose temperature-scaled surprisal, where surprisal is calculated from temperature-scaled probabilities. (2) We demonstrate that temperature-scaling with temperature T≈2.5 improves predictability of human reading times of English texts compared to T=1. (3) We identify linguistic phenomena that correlate with the benefit of temperature-scaled surprisal by analyzing residual errors from regression models. (4) We relate temperature-scaled surprisal to contextual Rényi entropy. 
predictive power for reading times: 
 In psycholinguistics, RTs on a word are believed to correlate with its processing difficulty. RTs can be gathered using different paradigms, including eye-tracking while reading text on a screen (Rayner, 1998) , self-paced reading (Aaronson & Scarborough, 1976; Mitchell & Green, 1978) and the Maze task (Forster et al., 2009) .The most common procedure for predicting words' RT is first to select a set of predictor variables thought to impact RTs v = [v (1) , ..., v (d) ] ⊤ ∈ R d , which include, e.g., the length of a word w t , |w t |, the frequency of a word freq(w t ). Let f ϕ : R d → R be a regression model parametrized by ϕ used to fit these predictors for the prediction of human RTs rt: rt(w t |w <t ) ∼ f ϕ (v), given the previous context w <t . The performance of f ϕ is quantified by its log-likelihood, with a higher log-likelihood indicating a better psychometric predictive power for human RTs (Frank & Bod, 2011; Fossum & Levy, 2012) .Besides the word length |w t | and word frequency freq(w t ), a word's surprisal (i.e., its negative logprobability in context) Hale (2001) ; Levy (2008) has been shown to be predictive of RTs (Demberg & Keller, 2008; Goodkind & Bicknell, 2018; Wilcox et al., 2020; Shain et al., 2022) . 
methods: 
 In this section, we delve into key aspects of information-theoretic measures in language comprehension. We start with surprisal, a method connecting processing difficulty to word predictability. As word predictability is empirically estimated by LLMs, we introduce the notion of calibration errors, metrics quantifying how good the estimation of word predictability is. Further, we lay out temperature-scaled surprisal, and the relation between varying temperature vs. varying α in contextual Rényi entropy.Starting from Shannon (1948) , the information conveyed by a word w t has been quantified as the negative log probability of the word w t given its previous context w <t . In Surprisal Theory (Hale, 2001; Levy, 2008) , this quantity is called surprisal s(w t ) and proposed to be predictive of the word's processing difficulty, typically quantified as its RT. Surprisal values are typically estimated from language models p(w t |w <t ).s(w t ) = -log 2 p(w t |w <t ),(1)Definitions Let D = {(x i , y i )} N i be a data set where x i ∈ X is an sample (i.e., context) and y i ∈ K = [K] is a category label. Let g θ and ẑi = g θ (x i ) denote a language model parametrized by θ and the output logit vector of sample i, respectively. The predicted class label ŷi for sample i is given by ŷi = arg max k∈K g(x i ) k and confidence for sample i is given by pi = max k∈K g(x i ) k . A model is perfectly calibrated when the confidence p is equal to the frequency of correctness, i.e., P(ŷ i = y i |p i = p) = p holding for all p ∈ [0, 1] and any sample i. Any difference between the left and right sides of the above equation indicates there exists a calibration error.Expected calibration error (ECE) (Guo et al., 2017 ) ECE is the most popular calibration metric, which empirically approximates the calibration error by discretizing the probability interval into a fixed number of bins (B m with m ∈ {1, 2, ..., M }), and measures the gaps of averaged confidence and averaged accuracy in each bin B m .EQUATIONwhere 1 is the indicator function. However, it does not necessarily measure the actual-word probability, which is the probability required for calculating surprisal in Eq. 1. It focuses only on the top-label probability for a given sample.Classwise-ECE (CECE) (Kumar et al., 2019; Kull et al., 2019) In comparison, CECE measures probabilities of all classes. For each bin and every class k, it assesses the difference between the average confidence of samples for class k and the actual proportion of class k. If assuming all classes weigh equally, we have:EQUATIONwhere pi,k is the predicted probability of sample i for class k.Human-likeness calibration error (HCE) We define the HCE as the Kullback-Leibler divergence (KL divergence) between predicted probability p from a neural LM and actual probability p * of human language model. HCE = D KL ( p||p * ).(4) Empirically, since p * is not directly observable, we approximate it by the estimates of a temperaturescaled model that best fits reading times (as discussed later). We denote the approximated HCE using such a method as HCE TS .Temperature scaling (Guo et al., 2017) is a widely-used method to improve model calibration. Given the output logit vector ẑi for sample i, a single scalar T > 0 is applied to rescale ẑi before the softmax activation:EQUATION)where qi is the calibrated confidence for sample i, and σ SM is the softmax function. Scaling by a scalar T does not alter the ranking; hence, the predicted label ŷi remains unchanged. As T > 1, it "softens" the probability distribution (i.e., makes the distribution more uniform), increasing uncertainty and entropy of the probability distribution, while T < 1 peaks the distribution. The parameter T in research on calibration is optimized by minimizing the negative log-likelihood on the validation set. In our experiments of fit to human RTs, we manually tune this temperature with T > 1.Temperature scaling has been successfully applied in several applications: In knowledge distillation (Hinton et al., 2015) , temperature scaling (with T > 1) is used to "soften" the knowledge (i.e., probability distribution) provided by the teacher model; in text generation, temperature is used to shape the probability distribution to ease certain aspects of the problems of top-k sampling (e.g., choosing an appropriate k value across varying contexts) (Ficler & Goldberg, 2017; Fan et al., 2018) . Temperature tuning inherently shifts the model's output in the generation's quality/diversity spectrum (Caccia et al., 2018) , with higher temperature decreasing the quality of generation while improving its diversity. This also aligns with our consideration of a possibility that human probability distributions might be flatter than the ones learned by language models and thus increasing the predictive diversity of surprisal provided by LLMs could potentially yield more human-like distributions.Given Eq. 5, temperature-scaled surprisal is:EQUATIONwhere ẑwt and k * = y wt denote the logit vector and the actual word w t class, respectively. For given t ∈ (0, ∞), we simply denote s T (w t , T = t) as s T | T =t . A temperature T with its best performance of final fit to RTs is denoted as T * . The extent to which a word's surprisal is affected by temperature scaling depends on the distribution and thus correlates with the entropy at word w t . Consider an example of two five-class probability distributions p i = [0.8, 0.05, 0.05, 0.05, 0.05] and p j = [0.8, 0.2, 0, 0, 0], for which the word indicated by the first position in the probability vector has identical surprisal in both p i and p j . Notably, p i is more uniform and p j is more peaked, resulting in distinct entropy characteristics: H(w i |w <i ) > H(w j |w <j ), where the entropy defined as the expectation of surprisal of current word w t over vocabulary, H(wt |w <t ) = E w ′ ∼p(•|w<t) [s(w ′ )] = -w ′ ∈W p(w ′ |w <t ) log 2 p(w ′ |w <t ), where W = W ∪{EOS} denotes the set of vocabulary W with EOS token. Fig. 1 illustrates a greater increase in surprisal for a word with a more uniform distribution than with a more peaked distribution. This figure also anecdotally shows that the effect of applying temperature scaling with T > 1 is similar to the effect of setting α < 1 in Rényi entropy.We will discuss the relationship between these parameters in more detail in the next section.While a lot of work has investigated the effect of next word entropy on reading times (Hale, 2003; 2006; Linzen & Jaeger, 2014; Angele et al., 2015; van Schijndel & Linzen, 2019; Aurnhammer & Frank, 2019; Pimentel et al., 2023) , we will here focus on contextual Rényi entropy (the entropy of the probability distribution at the current time stamp, which is parameterized by α), as proposed in Pimentel et al. (2023) to represent human anticipatory reading process. Pimentel et al. (2023) find that Rényi entropy with an optimal α * in the range of (0, 1) (around 1/2) obtains the best performance in reading time prediction (compared to Shannon Entropy (α = 1) or compared to unscaled surprisal estimates).Mathematically, Contextual Rényi entropy (Rényi, 1961) is defined as:H α (w t | w <t ) = lim β→α 1 1 -β log 2 w∈W (p(w|w <t )) β . (7) For given α ′ ∈ (0, ∞), we simply denote H α (w t | w <t )| α=α ′ as H α | α=α ′ .Theorem 1 (Monotonicity of s T (w t , T ) and H α (w t | w <t )). Given any probability distribution p with actual-word probability p wt > 1/K, where K is the number of classes, temperature-scaled surprisal s T (w t , T ) is strictly monotonically increasing inEQUATIONEQUATION) where T * is the optimal T of fit to RTs in the range of ∆ T .Theorem 2 Rényi entropy with α = 0 is equivalent to temperature-scaled surprisal with T → ∞.EQUATION)Theorem 3 For K ≥ 2, the expectation of the L1 norm between Rényi entropy with α = 1 and temperature-scaled surprisal with T = 1 has an upper bound. E[|s T | T =1 -H α | α=1 |] < 1 4 log 2 (K -1) + 1 (11) 
datasets: 
 We conduct analyses on two self-paced reading corpora, the Natural Stories Corpus (Futrell et al., 2018) and the Brown Corpus (Smith & Levy, 2013) , as well as on the Dundee Corpus (Kennedy et al., 2003) of eye-tracking measures. We follow previous work with respect to the preprocessing steps for each corpus (Kuribayashi et al., 2022; Shain et al., 2022) . Appendix C shows details about the preprocessing steps of each corpus.Recent observations showed that surprisal provided by LLMs with more parameters and lower perplexity is less predictive of self-paced reading times and eye-gaze durations (Shain et al., 2022; Oh & Schuler, 2023) ; across different experiments, GPT-2 (Radford et al., 2019) surprisals were found to predict human RTs the best. Therefore, we take four variants of pretrained GPT-2 (small, medium, large, xl) as our language models in all experiments. Following prior work, we obtain the surprisal for words composed of more than one subword by summing up the surprisal estimates of the subwords.We measure the predictive power of surprisal estimates from different language models, which is denoted as the log-likelihood difference per data point between a linear mixed-effects (LME) regression model with a predictor of surprisal estimates (target model) and a model without surprisal (base model), following Goodkind & Bicknell (2018) ; Wilcox et al. (2020) . More specifically, the metric of delta log-likelihood is defined as:EQUATIONwhere v tgt is target predictor variables that include baseline predictor variables as well as predictor variables of our interest, such as surprisal or temperature-scaled surprisal. v base is base predictor variables only including baseline predictor variables. The greater the value of ∆ llh , the more valuable the additional surprisal estimates are for predicting human reading times.For the calibration error evaluation, we set the number of bins M to 15 for both ECE and CECE, aligning with prior literature, such as works by Guo et al. (2017) ; Kumar et al. (2019) ; Rahimi et al. (2020b) , to ensure consistency in addressing problems where comparable probability ranges are relevant. The calibration metrics (ECE and CECE) are evaluated separately on each of the reading time corpus D. For simplicity, our calibration evaluation is conducted at the token level. Given that many words have extremely low probabilities and thus are often grouped into a single bin, we also evaluate the calibration error under the log probability binning scheme. For other descriptions regarding the metrics and evaluation, see Appendix. G. 
calibration of llms: 
 Table 1 shows ECE and CECE in log binning scheme for GPT-2 models of different sizes. LLMs are in general well calibrated on language modeling. Besides, LLM calibration improves with scale. Larger LMs are more calibrated. This conclusion is consistent with calibration investigation evaluated in BIG-bench multiple-choice tasks in Srivastava et al. (2023) as well as in several tasks including language modelling in Zhu et al. (2023) .We evaluate the predictive power of temperature-scaled surprisal. We scale T in the range of [1, 10] and measure ∆ llh , see Fig 3 . First, a confirmatory observation regarding the relationship between model size and predictive power: At T = 1, GPT-2 small exhibits the best predictive performance, and as the model size increases, ∆ llh declines, which is consistent with previous studies (Shain et al., 2022; Oh et al., 2022; Oh & Schuler, 2023) . Secondly, scaling the surprisal with T > 1 can significantly improve the predictive power across all corpora and LLMs. With optimal T * , on Dundee, Natural Stories, and Brown, the ∆ llh improvement is 23-43%, 60-89%, and 14-24%, respectively. We also observe a consistent pattern: when increasing T , ∆ llh first rises then declines; the optimal value T * falls within the range of (2, 3) (around 2.5) across all models and corpora in our setting. At T * , even though the impact of model size on final performance is not fully recovered, the disparity diminishes. Smaller models continue to outperform, but the extent of model sizes influencing performance is reduced.Finally, larger LMs typically have a larger human-likeness calibration error, shown in Table 1 . Larger LMs also require a higher value of T to reach their best performance and have a greater increase by temperature-scaled surprisal.Table 2 shows ECE and CECE in both equally-spaced and log binning schemes when T equals 1 and T * on three corpora. Probability distribution shaped by an optimal T * learnt for fit to human RTs drastically hurts the model calibration regarding these two metrics. ECE and CECE with T * are more than 10 times worse than values with T = 1. This discrepancy can be attributed to the different minima of deviations in LM human RT prediction and expected calibration error. The former is minimized towards words where LMs surprisal significantly deviates from human processing difficulty, while the latter is typically minimized with respect to the negative log-likelihood on a hold-out dataset (Guo et al., 2017; Rahimi et al., 2020a) . 
linguistic analysis: 
 Next we want to gain insight into what words benefit the most from temperature scaling. To this end, we analyze residuals from fitting LME regression models, identifying data points where scaling the temperature parameter notably enhances the fit of human RTs. Specifically, we quantify the improvement in fit by comparing the mean squared error (MSE) before and after adjusting the temperature to its optimal value as follows:EQUATIONwhere MSE T =T ′ (x F ) is the MSE calculated by all the data x F under the linguistic factor F . The difference ∆ MSE (F ) thus quantifies the impact of scaling relative to the linguistic factor F . A higher ∆ MSE (F ) signifies a greater influence of temperature-scaled surprisal of factor F . To ensure sufficient data in each subset, we only consider subsets including more than 1% of the data in each corpus.Given that temperature scaling enhances human likeness by shaping the probability distribution, it is natural to think about investigating whether there exists an inherent relationship between the distribution of probability and ∆ MSE . Specifically, one might ask questions like if samples with low probability gain more from temperature scaling or the other way around. We find that high surprisal words benefit more from temperature scaling than low surprisal words, across all corpora, see Fig 9in Appendix H.We investigate the effects of word-level properties, which include:Named entities. Research has substantiated that named entities (NEs) require increased reading time for humans since during the processing of such words (Damasio et al., 2004; Wang et al., 2013) . Oh & Schuler (2023) showed that NEs are among the top two significant factors contributing to the discrepancies of large and small LMs across all corpus-by-LM combinations. Therefore, we were wondering whether the effect of temperature-scaling might be driven by NE. To test this, we automatically tagged NEs using a BERT base model (Devlin et al., 2019) fined-tuned for NER2 .Part-of-speech tags. Similarly, previous research has argued that the poor fit of large LMs is primarily due to assigning too low surprisal estimates to open-class words like nouns and adjectives (Oh & Schuler, 2023) . We POS-tagged the corpora using the NLTK toolkit (Bird et al., 2009) Results. The result, as shown in Table 6 .2, shows primary factors responsible for the benefit of using s T (w t , T ) for each corpus-by-LM combination. The top three influential subsets for each corpus are underlined. Among all datasets and models, named entities perform to be the most beneficial word-level attribute. In contrast, closed-class words profit least from temperature scaling.Performance trends are consistent across different model variants on the same corpus.We also measured empirically how often temperature scaling increased vs. decreased the surprisal estimate of a word. Our results show that for ca. 90% of words, surprisal estimates are increased through temperature scaling across all word classes. For the subset of named entities, a slightly smaller percentage exhibits increased surprisal estimates. For a full analysis across different corpora and models, see Table 3 in Appendix B. A further analysis reveals that the primary benefit of temperature-scaled surprisal arises from the increase of low surprisal values, particularly noticeable in the case of named entities, see Table 4 in Appendix B.A fact that is often ignored (but see Nair & Resnik, 2023) is that modern LLMs use subword tokenization. This means that long words may consist of several tokens. In this case, the probability of the complete word is calculated by multiplying the probabilities of the subword tokens (and the word's surprisal is correspondingly calculated by adding the surprisals of the subwords). While this may often not matter, whether a word is tokenized into a single subword or several subwords can make a remarkable difference when applying temperature scaling: imagine a long / difficult word which has a low probability (and correspondingly a high surprisal). If this word were to be represented as a single subword token, temperature scaling might have the effect that the probability of this word gets increased during temperature scaling, and its surprisal estimate is hence decreased at T > 1.If, on the other hand, the same word were to be composed of two subword tokens based on the LM's subword vocabulary, one or both of the subword tokens can be expected to have a higher probability (than a hypothetical single subword token), and it is possible that during temperature scaling, the probabilities of the subword tokens would each be decreased at T > 1, such that the sum of the surprisals of the subword tokens would be much higher, compared to the word's surprisal estimate at T = 1.Figure 5 : Relationship between ∆ llh of GPT-2 s on three corpora and corresponding temperature. T is scaled from 1.0 to 10. Dashed vs. full lines denote only scaling tokens in single-token words and scaling tokens in both singletoken and multiple-token words, respectively.To summarize, whether the surprisal of a certain word would increase or decrease after temperature scaling could depend on whether that word happens to be included in the subword token vocabulary or not. Distributions of surprisal for single vs. multiple token words before and after temperature scaling are provided in Figure 8 in Appendix H. In order to quantify to what extent subword tokenization affects surprisal estimates, we conducted several analyses. Fig. 5 shows ∆ llh under various conditions: scaling all words (consistent with experiments in Section 5.2) vs. taking into the analysis only the subset of singletoken words. The comparison between the dashed and full curves highlights that the benefit of temperaturescaled surprisal comes primarily from the scaling of multiple-token words.Next, it is interesting to consider for what percentage of multiple-token words temperature-scaling increases the surprisal. We find that the ratio of surprisal increase is higher for multi-token words than across single-token words by ca. 6% on Dundee and Brown, see Table 7 in Appendix H for more details. 
discussion: 
 We found that temperature scaling is particularly effective for fitting RTs of named entities and open-class words; our later analysis on subword tokenization showed that this effect might be driven especially by words that are composed of several subword tokens. Of course open class words and named entities tend to be more complex and less likely to be contained in the vocabulary as single-word tokens, hence the observed correlation is to be expected.So what does all of this mean for surprisal estimates from LLMs and reading time prediction? Firstly, it is possible that indeed the effect is driven by humans failing to accurately estimate the probability of rare words, because they do not reach sufficient language experience or because human language models do not track these probabilities well, in line with Oh & Schuler (2023) ; in this case, temperature-scaling rare words to which the LLM assigns a high probability (and hence a low surprisal) would be a good strategy to counteract the discrepancy between humans and LLMs.Secondly, it is possible that the beneficial effect of temperature scaling is an artifact of subword tokenization, and that it would disappear if all words were composed of only a single subword token. In order to test this hypothesis, one would have to re-train a GPT-2 model from scratch using a vocabulary that at least includes all words that are contained in the reading time corpora, and then re-running the analysis to check whether a beneficial effect of temperature scaling can still be found. Finally, it is also possible that the splitting of a word into subwords coincides with the reader fixating a word several times, and that these added fixations lead to an overestimate in RTs compared to the actual surprisal experienced by a human reader. Future work could investigate this hypothesis by analysing RTs on subwords instead of aggregated words. This would require re-calculating reading measures, and comes at the caveat that subword tokens are not cognitively plausible units. 
conclusion: 
 This paper studies the prediction of human RTs from the perspective of probability distribution. We make the following contributions: (1) We demonstrate that the prediction of RTs can be significantly improved via temperature scaling of LLM probability estimates. (2) We establish that temperature-scaled surprisal is related to Rényi entropy. (3) We demonstrate that the primary benefit of temperature-scaled surprisal is driven by words composed of several subword tokens. These words also tend to be rarer / long open-class words. Future work should investigate the interaction of subword tokenization and temperature scaling, as well as the issue of tokenization in the analysis of eye-tracking data. 
 
 Appendix: Theorem 1 (Monotonicity of s T (w t , T ) and H α (w t | w <t )). Given any probability distribution p with actual-word probability p wt > 1/K, where K is the number of classes, temperature-scaled surprisal s T (w t , T ) is strictly monotonically increasing inEQUATIONEQUATIONwhere T * is the optimal T in the range of ∆ T .Proof. Eq. equation 8 can be easily verified by considering the monotonicity of temperature-scaled softmax output σ SM ( ẑwt /T ). The second part of Eq. equation 9 can be rewritten as:EQUATIONwhere for the step from Eq. equation 16 to Eq. equation 17 we use AM-QM inequality and K is the number of classes in tokenizer. The first part of Eq. equation 9 can be rewritten as:EQUATIONEQUATIONwhere from Eq. equation 19 to Eq. equation 20 we use AM-GM inequality.Theorem 2 Rényi entropy with α = 0 is equivalent to temperature-scaled surprisal with T → ∞.EQUATIONProof. By plugging in α = 0, Contextual Rényi entropy recovers to be the entropy that readers concentrate on the count of potential words with nonzero probabilities, which is defined in Eq.(5) in Pimentel et al. (2023) . As T → ∞, temperature-scaled surprisal converges to the surprisal induced by random guessing. Given the assumtion that p(w|w <t ) > 0 for each word w ∈ W, LHS becomes:EQUATION)where K is the number of classes. As T → ∞, RHS becomes:RHS = -lim T →∞log 2 e zw t /T w∈W e zw/T(24)EQUATIONTheorem 3 For K ≥ 2, the expectation of the L1 norm between Rényi entropy with α = 1 and temperature-scaled surprisal with T = 1 has an upper bound.EQUATIONProof. With Jensen's inequality, we have:EQUATIONwhere Var[•] denotes the variance. The last inequality is shown by Lemma 4, completing the proof of this theorem.Lemma 4 (Maximum variance of the surprisal). (See Theorem 8 and Lemma 15 in (Reeb & Wolf, 2015) ). Let ρ = diag(p 1 , p 2 , ..., p d ) be a state on a d-dimensional system. Let -log p i be the surprisal of the output i in this system. Define N d to be:EQUATION)For d ≥ 2, the variance of surprisal has a tight upper bound:EQUATIONB FURTHER ANALYSIS IN SECTION 6.2We observe that larger LMs exhibit an increased ∆ MSE by utilizing temperature-scaled surprisal, as shown in the average column (Avg.) of Table 6 .2. Specifically, on Dundee, the top 2 models achieving the largest improvement through temperature scaling are GPT-2 medium and xl, while GPT-2 large and xl have the most benefit on Natural Stories and Brown. This result is consistent with previously observed ∆ llh improvement (∆ llh +) across the corpus-by-LM reported in Table 1 , suggesting a correlation between model likelihood and MSEs of the regression models. We do not observe a mismatch between them, as posited by Oh & Schuler (2023) that LME models achieve similar MSEs irrespective of obvious differences in model likelihood.Regarding the effect of the change (increase or decrease) of actual-word probability on the final fit to RTs, we first analyzed the ratio of probabilities decreasing (or increasing) for all words, as well as for subsets with specific word-level properties, choosing named entities as the representative, as shown in Table 3 . We observed that probabilities of the majority of words (around 80-90%) decrease by temperature scaling. Compared with the average across all word types (as indicated in the 'Avg.' column), named entities exhibit a lower ratio of probability reduction. Larger LMs tend to have a higher ratio, especially the ratio for named entities, likely because smaller models may lack the specific knowledge of less common terms, such as named entities. We further investigate the benefit of temperature-scaled surprisal (quantified by ∆ MSE ) given the subset of words whose probability decreases (or increases). The results are in Table 4 . On Dundee, the main gain arises from the reduction of large probabilities via temperature scaling. Conversely, for Natural Stories, the primary benefit comes more strongly from words with originally very low probability, which get more probable. For Brown, the effects are evenly split. This variation aligns with our theoretical intuition that temperature scaling enhances the fit performance by making probabilities more smoothing and uncertain, which means not only making high probabilities lower but also making super low probabilities higher and close to 1/K, since a super low probability also means the model is confident in the incorrectness of certain classes. For named entities, the story is converse on Dundee vs. on Natural Stories and Brown, where for the latter two corpora, the advantage is primarily due to reducing the probabilities of highly predictable entities. We shed light to the possible reason of such a discrepancy in Fig 6 , which displays the top 15 frequent words for GPT-2 small on each corpus. Notably, Natural Stories and Brown show a marked lack of words with increased probabilities (blue bins) compared to Dundee. This lack weakens the overall impact of rising probabilities (denoted by ∆ MSE (p wt ↑)). Specifically, on Brown, only 4 out of 15 top frequent words have the composition of increased probabilities, correlating with the largest discrepancy in ∆ MSE between probabilities that decreased (329.7) and those that increased (-170.6 ) in Table 4 . On Dundee ET corpus (Kennedy et al., 2003) , we use the first-pass gaze duration. Following prior work (Kuribayashi et al., 2022) , we remove words containing numbers or punctuation, words that are either the first or the last one in a line, as well as words whose previous words contain numbers or punctuation. On Natural Stories SPR corpus (Futrell et al., 2018) In this section, we explore the question of whether the benefit of temperature-scaled surprisal holds only for regression models already containing other predictors such as length and frequency. We conduct experiments similar to those detailed in Section 5.2 while setting base predictor variables v base to 0 and target predictor variables v tgt to only temperature-scaled surprisal s T (w t , T ) in Eq. 12. Fig. 7 shows that excluding base predictors decrease but not totally impact the effectiveness of temperature-scaled surprisal.In Table 5 , we demonstrate the calibration error (% ECE) for single-token and multiple-token words for GPT-2 small. Calibration evaluation is conducted at the token level as before. Results indicate that multiple-token words show larger calibration errors than single-token words.F PROBABILITY DISTRIBUTION BEFORE AND AFTER TEMPERATURE SCALING Fig. 8 shows actual-word probability distribution before and after temperature scaling for GPT-2 small on three corpora. Multiple-token words tend to have smaller probabilities than singletoken words, both before and after temperature scaling.We evaluate calibration error (% ECE and % CECE) in both equally-spaced and log binning schemes. In equally-spaced binning scheme, the samples are grouped into M ∈ N equally-spaced interval bins based on their confidences pi . Conversely, the log binning scheme operates under an empirical upper limit for -log 2 pi , denoted as max(-log 2 p). Table 6 shows ranges of p and -log 2 p for GPT2s on three corpora. For this scheme, we establish M ∈ N log-equally-spaced interval bins within the range of (0, max(-log 2 p)].We investigate scaling T ∈ [1, 10], considering both densely and sparsely distributed points. The values examined are detailed as follows: [1.0, 1.1, ..., 1.9] for dense intervals, [2. 0, 2.25, ..., 3.25] for moderately spaced intervals, and [3.5, 4.0, ..., 10.0] for sparse intervals. Code in this paper will be released upon paper acceptance.Link: https://huggingface.co/dslim/bert-base-NER 



Reviews: review: title: title: Word representations, reading time, and language model calibration
main_review: main_review: Why do humans spend longer reading some words than others? Work in psycholinguistics has long emphasized the importance of word surprisal as a driving factor; yet, such measures require an estimate of the underlying probability of a word. Language models are revising our collective understanding of word probabilities; while the authors note that one may expect that more faithful estimates of the probabilities of words in our world would lead to better fits to human surprisal judgments, it is not clear that they do. The authors try to figure out why in this work. 

In particular, the authors study whether the calibration of language models' probability distributions underlies such discrepancies. Through a series of experiments with GPT-2 variants, the authors find that indeed, modulating language models' distributions through temperature scaling (specifically, flattening the distributions) yields better fits to human reading times. They conduct a more granular investigation to identify differences at individual word levels. This work nicely informs a better understanding of human and machine representations of words, and uncertainty over these representations; the work is a good topical fit to this workshop. 

While I appreciated that the authors provided background on the metrics they considered (Section 3 was well-written), unfortunately I found the bulk of the paper to be quite clunky to read. The graphs were also quite small and not easily interpretable; there was also no sense of error bars in either tables or figures (which I would strongly encourage the authors to add in a future version). Coupled with my confusion from the text, this made it hard to deeply understand and assess the results. 

Additionally, I take some issue with the way the authors discuss humans throughout the piece. I would encourage the authors to avoid the phrase "human language model" or clarify a bit more what they mean there... I assume this refers to humans' internal model(s) of language processing? The passage: “Temperature tuning inherently shifts the model’s output in the generation’s quality/diversity spectrum (Caccia et al., 2018), with higher temperature decreasing the quality of generation while improving its diversity. This also aligns with our intuition: increasing the predictive diversity of surprisal provided by LLMs to be more human-like" implies that humans have poor quality language generation...? Yet, humans are great language generators overall... and can be much better than the GPT-2 model considered by the authors here. I think more nuance here would be beneficial. 

For future experiments, I would be keen to see how the authors' results stand up for larger language models, like LLaMA or Mistral. I believe the experiments should be doable with the models' logprobs? 

I will caveat my review that I am not well-versed in either psycholinguistics literature or the relationship between language model and human surprisal (which seems to be an active area of study), so I have a hard time assessing novelty. However, I did find the paper interesting and imagine others at the workshop would as well. I think the paper could spark discussion, and as such, I recommend to accept.
strength_weakness: strength_weakness: I believe the work will nicely appeal to the representation alignment community, straddling both ML and psycholinguistics. The work appears timely, and I think can spark conversation.
title: title: Well-crafted paper with opportunity to promote discussion
main_review: main_review: ### Summary
- It has been described in the literature that surprisal estimates from larger LLMs begin to reduce in fit to human RTs. The authors propose that this is due to the lowest perplexity models becoming the most "peaky" and propose using temperature rescaling to smooth out the resulting distributions. This has the intended effect of improving RT fit for large models. The authors investigate precisely on what subset of RT data this improvement is most pronounced and highlight interesting pathologies of subword tokenization.

### Clarity
Strengths:
- This is a clearly written manuscript. The authors make ample effort to discuss existing literature, formulate the research question and contribution, and outline the technical background.

Weaknesses:
- While the writing is clear, several of the figures could use work. Also, please label the y-axes in figs 1 and 2. 
- Please include the full specification of the LMEs fixed and random effects structure in 4.3

### Correctness
Strengths:
- The authors are careful to state their assumptions and the implications of their results. The investigation into exactly which subset of the RT data these fit improvements are most pronounced is excellent.

### Novelty & Interest to the community
Strength:
- This is a clear investigation of how temperature rescaling can be used to adjust model probability estimates toward human distributions. I think the workshop will find this interesting both within psycholinguistics and more broadly. The results on some of these subword tokenization pathologies are particularly worthy of discussion.

Weakness:
- While an extremely well-constructed paper, the core idea of temperature rescaling to improve calibration between two distributions is quite trivial (and has been used in other domains although not for this particular LM surprisal - human RT calibration problem).
strength_weakness: strength_weakness: This is a clear and well-constructed paper whose results are impactful in the subfield of sentence processing in psycholinguistics, but whose broader themes are relevant for numerous discussions comparing measurements from humans and probabilistic models.
title: title: Interesting to see a new method to improve LLM fit to human reading times
main_review: main_review: Summary: 
- This paper studies using AI large language models (LLMs) to predict human reading times (RTs). This is motivated by investigating the next-word prediction probability estimates of LLMs compared to humans. The paper shows that human RT prediction can be improved using temperature scaling, i.e., making the probability distribution more uniform.
- The paper runs further experiments, showing that this improvement is primarily driven by words composed of several subword tokens and also, named entities and open-class words, e.g., nouns and adjectives.

Strengths:
1. Tested on multiple sizes of LLM (gpt-2: small, medium, large, xl)
2. Tested on multiple human RT datasets (Natural Stories, Brown, Dundee)

Weaknesses:
1. As I was reading the paper, I kept thinking of one hypothesis for why temperature-scaling improves LLM prediction of human RTs: LLM surprisals in normal conditions poorly match human RTs, but temperature-scaling makes the LLM surprisals more uniform, hence matching human RTs better. This is supported by some of their other results, including the result that larger LLMs generally have worse delta-llh, and also require a higher value of T to reach their best performance (Table 1). This is somewhat related to Paragraph 2 of the paper's Discussion section, but I don't think they address this hypothesis directly.
2. I hope the paper can explain a bit more about why it matters to find new methods (e.g., temperature-scaling) to improve LLM prediction of human RT, i.e., what would achieving this mean? Also, why the paper's results are important -- knowing that temperature-scaling surprisal improvement is driven by multi-subword words and open-class words, how does this change our understanding of LLMs or human reading?

Minor notes:
1. The paper cites a paper saying that LLM probability estimates are often overconfident compared to ground-truth. I wondered if humans are also overconfident compared to ground-truth. Knowing this would help me understand why temperature scaling improves LLM fit to human RTs.
2. The paper describes "LLMs". However, GPT-2 is not really "large" (except the xl version they tested, maybe) compared to much newer LLM models.
3. Can the paper provide an intuitive visualization and explanation of how well each/any of these LLMs predict RTs? I think it would aid understanding. How close are LLMs to perfectly predicting human RT?
4. The paper title asks: BUT DOES IT DO SO FOR THE “RIGHT REASONS”? I wish the paper discusses more about what they think are the "right reasons", and also answers the question (whether temperature-scaling estimates improve fit to human RTs for the right reasons).
strength_weakness: strength_weakness: N/A