On the Entropy Calibration of Language Models
Authors: Steven Cao
Gregory Valiant
Percy Liang
Venue: NeurIPS 2025 poster ()
Keywords: language models
language generation
calibration
entropy
error accumulation
scaling laws
language model theory
rl theory
Primary area: theory

Abstract: We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text. Past work found that models are miscalibrated, with entropy per step increasing as generations grow longer, due to error accumulation. To calibrate the model and improve text quality, it has become standard practice to truncate the distribution, but this approach reduces output diversity, which we would like to avoid. Therefore, in this paper, we ask: does miscalibration improve automatically with scale, and if not, is it theoretically possible to calibrate without tradeoffs? To build intuition, we first study a simplified theoretical setting to characterize the scaling behavior of miscalibration with respect to dataset size. We find that the rate of scaling depends on the power law exponent of the data distribution --- in particular, for a power law exponent close to 1, the scaling exponent is close to 0, meaning that miscalibration improves very slowly with scale. Next, we measure miscalibration empirically in language models ranging from 0.5B to 70B parameters. We find that the observed scaling behavior is similar to what is predicted theoretically: our fitted scaling exponents for text are close to 0, meaning that larger models accumulate error at a similar rate as smaller ones. This scaling (or, lack thereof) provides one explanation for why we sample from larger models with similar amounts of truncation as smaller models, even though the larger models are of higher quality. However, truncation is not a satisfying solution because it comes at the cost of increased log loss. In theory, is it even possible to reduce entropy while preserving log loss? We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.

On the Entropy Calibration of Language Models
Steven Cao Stanford University shcao@stanford.c edu
Gregory Valiant Stanford University valiann@@stanfrd.ccmm
Percy Liang
Stanford University
pliangêcs. stanford.edu
Abstract
We study the problem entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text. Past work found that models are miscalibrated, with entropy per step increasing as generations grow longer, due error accumulation. To calibrate the model and improve text quality, it has become standard practice to truncate the distribution, but this approach reduces output diversity, which we would like to avoid. Therefore, in this paper, we ask: does miscalibration improve automatically with seale, and if not, is it theoretically possible to calibrate without tradeoffs? To build intuition, we first study a simplified theoretical setting to characterize the scaling behavior of miscalibration with respect dataset size. We find that the rate of scaling depends on the power law exponent of the data distribution - in particular, for a power law exponent close I, the scaling exponent is close to meaning that miscalibration improves very slowly with scale. Next, we measure miscalibration empirically in language models ranging from 0.5B to 70B parameters. We find that the observed scaling behavior is similar to what is predicted theoretically: our fitted scaling exponents for text are close to O, meaning that larger models accumulate еrror at similar rate as smaller ones. This scaling (or, lack thereof) provides one explanation for why we sample from larger models with similar amounts of truncation as smaller models, even though the larger models are of higher quality. However, truncation is not a satisfying solution because it comes the cost increased log loss. In theory, is it even possible to reduce entropy while preserving log loss? We prove that it is possible, if we assume access to black box which can fit models to predict the future entropy of text.
Introduction
We study entropy calibration, which asks whether a language model's entropy over generations matches its log loss human text. This definition is natural notion of calibration for generative tasks, and is more challenging than calibration for classification tasks because the output space is exponentially large. Braverman al. (2020) were the first to study entropy calibration language models, and they found that models are miscalibrated: entropy per step for model generations increases with the length of the document, in contrast with log loss on human flat over the length of the document (Genzel & Charniak, 2002; Verma et al., 2023). Entropy calibration can be thought of quantitative formalization of the well-known "error accumulation" or "teacher forcing" problem: entropy rises when the model generates erroneous tokens which are fed back into the context, derailing the generation (Williams & Zipser, 1989; Ranzato al., 2016; Basu et al., 2021; Hewitt et al., 2022, also see Appendix E). Therefore, we study entropy calibration to gain fundamental insights into improving generation quality.
which roughly
To correct error accumulation and calibrate the model, it has become standard practice to truncate the next token distribution (Fan al., 2018; Holtzman et al., 2020; Hewitt al., 2022), suppressing low
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

probability tokens to improve quality at the cost of diversity (Hashimoto et al.. 2019; Zhang et al., 2021). This solution is not satisfying: diversity especially important for difficult tasks where we must aggregate multiple answers (Wang et al., 2024; Brown et al., 2024), as well as for synthetic data generation, which has seen a resurgence of interest as the community has begun worrying about running out of internet data (Wang et al., 2023; Gunasekar et al., 2023; Maini et al., 2024). Therefore, it is natural to ask: do we expect miscalibration to improve with scale? If not, is it at least theoretically possible to calibrate without sacrificing diversity?
To build intuition, we first study simplified theoretical setting, where instability comes from the fact that the model might generate a token that it saw only a few times during training. This unfamiliar token then derails subsequent steps when is fed back into the context autoregressively. Drawing on classic results, we calculate scaling exponent capturing how quickly the probability of generating a rare token decreases with the number of training examples (Good, 1953; Karlin, 1967). We find that this exponent depends on how heavy-tailed the data distribution is: in particular, for power law exponents close to 1, as typical for human text (Zipf, 1936, 1949), the scaling exponent close to 0. Therefore, this setup predicts that stability in generation improves very slowly with scale.
Next, we measure miscalibration empirically in large language models with up to 70B parameters, on three datasets. We find that the observed scaling behavior is similar to what is predicted by the simplified setting: fitting scaling exponents relating calibration error to model size, we find that the exponent for the two text datasets is around -0.05, meaning that larger models are similarly miscalibrated as smaller ones. On the other hand, for the code dataset, the scaling exponent is around -0.3, meaning that miscalibration improves moderately with scale. We measure the power law exponent to be around 1 for the two text datasets, and 1.5 for the code dataset. Therefore, these findings are consistent with the theory: the code dataset has more quickly decaying tails, so the scaling should indeed be faster. However, further work on more datasets is needed to more strongly establish this relationship between the power law and scaling exponents.
If even large models suffer from error accumulation, why are reasoning and instruction-tuned models able to produce long, coherent outputs? We find that much like distribution truncation, instruction tuning reduces entropy at the cost of increased log loss, with the largest models now having entropy too low. This tradeoff relates to past work which found that aligmment degrades model capabilities, a phenomenon known as the alignment tax (Ouyang et al., 2022; Bai al., 2022; Lin et al., 2024).
Given that all known mitigations increase the model's 's log loss, is it even possible in theory to calibrate without this tradeoff? Drawing on ideas from reinforcement learning theory, we prove that it is possible, if we assume access to a black box which can fit models on the future entropy of text prefixes and attain low test error. Specifically, we describe a polynomial-time calibration procedure that adjusts each candidate token's 's probability based on the expected entropy of its continuations. While the resulting procedure impractical implement, we prove that it calibrates while preserving log loss, suggesting that generation stability and diversity might be possible to attain simultaneously.
2 Preliminaries
We first review key definitions and properties for entropy calibration, introduced in Braverman et al. (2020). Our setup is as follows: we are given prompts X E X' drawn from some prompt distribution X q. and responses Y e y drawn from the true conditional distribution Y ~ ~ppx. For example, X might contain a description of a coding task, while Y contains a solution to the task. We then train a language model P X' - A to fit the true conditional distribution p". We say that ý is entropy calibrated if its entropy over generations is equal to its log loss:
(p) L(p" P),
(I)
where the total/per-step entropy and total/per-step log loss are given by
H(p) = Ex~ E |-
H((p) = Ex~ Ex l- (Y: &<a)) (2)
C(p" p) = ExpxxxxE--pt/ooogg (Y)). C,(p" I1)
l- (% Y20).. (3)
To build intuition for this definition, entropy can be thought of as a measure of the model's uncertainty, which should be calibrated to match the actual loss it incurs on real data. This definition mirrors that of binary calibration, and we derive this connection more formally in Appendix B. Qualitatively, if model is undereonfident, then its generations have too much entropy and appear incoherent;

---

if it is overconfident, then its generations have too little entropy and appear repetitive (Braverman et al.. 2020; Basu et al., 2021); see Appendix E for a replication of this finding and Appendix D for examples. Entropy calibration is then the problem of adjusting the entropy to be just right. Empirically, Braverman et al. (2020) found that neural autoregressive language models have entropy too high: entropy per step matches the log loss at earlier steps but increases as the generation grows.
Why does entropy per step grow with the length of the generation? The main problem, as has been observed in empirical work, is that autoregressive language models accumulate error during generation. At training time, models are given input from the true distribution and asked to produce only single additional token. In contrast, models must generate multiple tokens at deployment time, which they do by producing one token at a time and taking their own production as subsequent input. Therefore, even models with very low single-step enror can degrade over multiple steps as they take their own slightly erroneous outputs as input and accumulate errors (see, e.g., Ranzato al. (2016), Welleck et al. (2020), Holtzman et al. (2020) for error accumulation in language modeling; and Daumé et al. (2009), Ross& & Bagnell (2010), Ross et al. (2011) for imitation learning). This intuition is formalized in the context of entropy calibration in Corollary 4.2 of Braverman et al. (2020), which states that for a model with e KL divergence to the true distribution, the entropy at step can deviate as much as E + Vet from the log loss, growing with t.
How does one calibrate the entropy? Unlike binary and multiclass calibration, entropy calibration is challenging because the models have exponentially large output space. In practice, practitioners use a number of distribution truncation methods, each of which uses a different heuristic to suppress low probability tokens in each generation step. Some standard methods include temperature reduction, top-k sampling (Fan et al., 2018), top-p sampling (Holtzman et al., 2020), and min-p sampling (Hewitt al., 2022). These methods improve text quality at the cost of diversity (Hashimoto et al., 2019; Zhang et al., 2021; Pillutla et al., 2021; Welleck et al., 2024). Following Hashimoto et al. (2019), we define a model's 's diversity to be its log loss on reference documents. The intuition behind this definition is that log loss (also known as cross entropy or forward KL) is coverage metric: to attain low log loss, the model must "cover" as much as the reference distribution as possible. Our goal, then, is to calibrate entropy to match log loss without also causing the log loss to increase.
In theory, Braverman et al. (2020) show that one calibrate entropy while preserving log loss via globally normalized temperature scaling, where the adjusted model is given by P. Oy.......,, ox p(y1,..., )/r. Unfortunately, this adjustment is intractable to compute because it involves normal- izing over the entire output space. It remains unclear, then, whether this goal is possible in polynomial time. Specifically, we wish to take in model á and produce calibrated model P with at most entropy calibration error per step, as well as log loss at most that of the original model p.:
1 EnnCE(P
[1]] E,
(4)
T
C(p" p) S L(p" || p),
(5)
where the entropy calibration error is defined as the difference between the entropy and the log loss: |EntCE(y" [] = HΦ))- - L(p" || ()
3 Intuition: Singleton Mass in Power Law Data
Before putting in the work to develop better calibration algorithms, it is natural to first ask whether we expect miscalibration to automatically improve with scale, as we train larger models on more data. To gain intuition, we first explore this question in simplified theoretical setting. We define the setup to capture the following hypothesis regarding error accumulation (see, c.g., Hewitt et al. (2022)): because the language distribution is heavy-tailed the model must assign non-zero probability to large number of rare tokens when fitting the data to achieve low log loss. However, if it happens to generate one such rare token, the model derails when that token is fed back into the context autoregressively, leading to a jump in entropy. Over many generation steps, then, the model will eventually derail. The degree of instability then depends on the probability of producing rare token.
Accordingly, our setup is as follows: at training time, the model stores the counts for m tokens drawn i.i.d. from an a-power-law distribution p over vocabulary of size v. defined as p; 1/7at for i= 1,.,,, v. The model then generates a sequence token-by-toker as follows: if all tokens in context were seen at least twice at training time, the model samples a random token seen during training.
3

---

Llama
Llama 3
Pythia
Qwen2.5
-0.2
-1.3
-0.
-05-
logtEnttDD
-1.3
-10
23-
20
312
14l
Y=
-23
Wottaghoepes Y-a S0r
-2.3
Vertnghoeppt Y-0a0x
-50
Meztighheptt 1-0722
-20-
Mrstngrrepp: F-CADX
Can Deesoin Y 61
DateCemoate 7-a728
25
log[Model Size)
Figure Log calibration error versus log model size for four model families and three datasets. We find that the scaling laws fit relatively well, suggesting that the relationship between calibration and scale predictable. Furthermore, while there is variation between model families, the scaling exponents for each dataset are somewhat close to those predicted by theory (WikiTeet: 0.089, WritingPrompts: -0100, CodeContests: -0.33), with heavier-tailed datasets having slower scaling.
But if any token in context was seen only once, the model samples from high entropy "derailed" distribution instead. This simple stylized setting captures our intuition about error accumulation and lets us study the effect of a, representing the teavy-tailedness of the data distribution.
In this setting, the expected entropy per step grows with slope proportional to the probability of emitting rare token (see Appendix B). Computing the rare token mass in power law data is classic problem, and we can compute the asymptotic scaling exponent with respect to the number of training examples m as follows (Good, 1953; Karlin, 1967):
Proposition 3.1 (informal). For v infinite and m large, the per-step probability of generating singleton, in expectation over draws of the training set, is given by
Km,1 C.ml/a_!
where Ca is constant depending only on a, and Km,1 is random variable denoting the number of items seen exactly once in a set of m samples.
We provide the derivation in the appendix. The key takeaway from this proposition is that the derailing probability scales as m 1/a-1, which is very slow the power law exponent a is close to 1, as is typical for text (Zipf, 1936, 1949). The reason for this slow scaling is that as increases, there always more rare items to be sampled from the tail of the distribution. In practice, of course, we not training unigram models, but the same intuition holds if we posit that semantic concepts in text are similarly heavy tailed: as larger models are trained on more data, there will always be new rare phenomena that they see during training only once. These phenomena are then memorized, and potentially generated at deployment, derailing the model.
While asymptotic analysis gives us clean expression, we can also estimate the scaling exponent in simulation for finite values of m and v. We find that the non-asymptotic simulated slopes are close the asymptotic expression as long m is smaller than u/3 (see Appendix E). We also calculate the power law exponent for our three datasets, finding that is around for WikiText and WritingPrompts and 1.5 for CodeContests, which predicts slow scaling for the first two datasets and slow-to-moderate scaling for the third.
Experiments: Miscalibration in Large Language Models
Next, we measure miscalibration empirically in large language models. We study four model families (Qwen2.5 (0.5B. 1.5B, 3B, 7B, 14B, 32B, 72B) (Qwen et al., 2025), Llama 3 (1B, 3B, 8B, 70B) (Grattafiori et al., 2024). Llama 2 (7B, 13B, 70B) (Touvron et al., 2023), and Pythia (410M, 1.4B, 2.8B. 6.9B, 12B) (Biderman et al., 2023)) applied to the three datasets listed below. In each setting, we use 5000 examples and limit samples to 1024 tokens; see Appendix C for more experimental details. We primarily study base models because we are interested in the problem of modeling human text; study the effect of instruction tuning in Section 4.3.
(a) WikiText- 103 (Merity et al., 2017): given 128 tokens of context from a Wikipedia passage, the model is tasked with completing the passage.

---

Writingrompps
Staii
Larop
Lib D16
Lacer
1773
38
379
375
234
Derd L0T
Deec
Step
Figure 2: Entropy for each generation step (solid) and log loss for each token in the ground truth (dashed), for each dataset (columns) and each model family (rows), with models colored by size. Models have entropy much higher than their log loss, with the gap growing with the number of generation steps, a sign of error accumulation. For the text datasets, models of different sizes seem to be similarly miscalibrated, while for code the degree of miscalibration seems improve with size.
(b) WritingPrompts (Fan et al., 2018): given a prompt from r/writingprompts along with 128 tokens of context from buman-written story, the model is tasked with completing the story. (c) CodeContests (Li et al., 2022): given coding problem from one of five websites and 128 tokens of context from a human-written solution, the model is tasked with completing the solution.
4.1 Miscalibration scaling in base models
Past work has found that many model capabilities improve predictably with model size, with task loss and model size following linear relationship when plotted on log scale (Kaplan et al., 2020; Hoffmann et al., 2022). We use a similar methodology to study the relationship between entropy miscalibration and model size. If model size and dataset size are scaled proportionally, Proposition 3.1 suggests a scaling law of log EnICE (1/a 1) log m + C, where the data distribution and m is the parameter count. Does the actual data also follow a clean scaling law, and how close the scaling exponent to that predicted by the simplified setting?
the power law exponent of
For each madel-dataset combination, we compute the model's calibration error as the difference between its average entropy per generation step and its average log loss ground truth data. We then plot log calibration error versus log model size, as shown in Figure 1.
First, we find that the linear fit is accurate, suggesting that the relationship between calibration and scale is predictable. Next, we find that the scaling exponents are dataset-dependent: for the older model families (Llama 2 and Pythia), the exponents around 0.0 for WikiText and WritingPrompts
5

---

Qwen2: 3-1.58
To
Insruct
insroct
02
T- 2005
-2.
0.25
0.85
226
3.2
3.34
238
130
286
208
3:00
3.3
254
208
0680
046
C95
addo
oxes
0370 CiT3
Owen2.5-7R
1.00
02
2.0
0.3
0.5
1-005
ingruct
-c.0 0.0
0.0
1.84
1.02
740
3:0
3:41
3/0
8.0
Owen2,5-728
1.0
1:501
2.05
Tw00 T=050
0.9
4050
0.9
-324
-8.2
0.8
Instrct
-35
0.5
01
instruct
1.50
loss
Figure 3: Entropy calibration error versus log loss for base Qwen2.5 (1.5B, 7B, 72B) compared to the instruction-tuned versions, along with various temperature settings (please see Appendix E for all model sizes). Positive calibration error means that the model's entropy is higher than its log loss, while negative means that its entropy lower than its log loss. We find that each modification of the base model reduces entropy while increasing log loss, calibrating at the cost of diversity.
and -0.2 for CodeContests, while for the newer model familics (Llama 3 and Qwen2.5), the exponents are around -0.13 for WikiText and WritingPrompts and -0.35 for CodeContests. Notably, these exponents somewhat close to what is predicted theoretically (Figure 4): WikiText and WritingPrompts, with power law exponents of 0.918 and 1.114, are predicted to have slow scaling exponents of 0.089 and -0.10, while CodeContests, with a power law exponent of 1.5, is predicted to have a moderate scaling exponent of 0..3. However, future work on more datasets would be needed to more strongly establish the relationship between these exponents empirically.
We speculate that recent model families have better scaling due to changes in their pretraining data mixtures, and especially the addition of a midtraining step with higher quality and less diverse data. However, training details for three out the four model families (all but Pythia) are not public, and future work with controlled data mixtures would be useful disentangle the effects of model size, dataset size, and dataset composition.
Overall, these plots suggest that miscalibration in text generation improves very slowly with scale: scaling exponent of -0.10 means that to reduce calibration error by factor of 10, dataset size must increase by factor of 10
4.2 Entropy over time
To gain a more fine-graince understanding entropy blowup, we also produce entropy over time plots for each model and each dataset, as shown in Figure 2. Specifically, we plot each model's entropy at each generation step f, averaged over 5000 generated samples. We then compare this curve to the model's log loss on each token t of ground truth example, averaged over 5000 examples. Recall from Section 2 that theoretically, for an accurate model, entropy is initially close to log loss, but can deviate as much as vi at the /-th step. A calibrated model which does not experience error accumulation should have entropy close to the log loss for all generation steps.

---

First, we find that for each model and dataset, the log loss is mostly constant or slightly decreasing over time. Past papers use model's log loss estimate the actual entropy of the underlying text, as the former is an upper bound for the latter that grows tighter if the model is more accurate. This part of the plot replicates past findings that the entropy per step of human text stable over time, also known as the entropy rate constancy principle (Genzel & Charniak, 2002; Verma et al., 2023).
On the other hand, unlike human text, the entropy per step of model generations is not stable and instead increases over time, The lack of scaling shown quantitatively in the previous subsection is reflected visually in Figure 2, with larger models having entropy growing at similar rates as smaller models for WikiText and WritingPrompp (the left and middle columns). For CodeContests, the slopes decrease with model size, visually confirming that there is hllw-to-moderate scaling.
4.3 Calibration-diversit tradeoffs
In this subsection, we seek to better understand how distribution truncation and post-training affect entropy. For each model-dataset combination (excluding Pythia, which has Instruction-tuned version), we compare the model with temperature 1.0 to that with temperature 0.95, 0.9, 0.85, or 0.8, well to the instruction-tumee version of the model. We then plot entropy calibration error against log loss, where each setting of the model is one point on the plot, as shown in Figure 3.
First, we find that reducing temperature below 1 reduces entropy but increases log loss, replicating similar findings in past work (Hashimoto et al., 2019). Furthermore, the temperature attaining zero calibration error is similar across model sizes, which makes sense given that they are similarly miscalibrated. We find that instruction tuning also reduces entropy while increasing log loss, which is consistent with past work showing that instruction tuning harms diversity (Ghosh et al., 2024). Unlike temperature scaling, the magnitude the effect varies across model sizes, with larger models experiencing both a larger reduction in entropy and larger increase in log loss. However, this pattern is not robust across model families (see Appendix E). Further work with more controlled instruction tuming would be necessary explore this relationship further. These experiments reconeile previous finding, that even large models accumulate errors, with the fact that in practice, one can use truncation post-training generate long, coherent pieces of text. The tradeoff is that each of these mitigations comes at the cost of diversity.
5 Theory: Future Entropy Scaling
If all known mitigations increase log loss, is it even possible in theory to calibrate without this tradeoff? In this section, we provide evidence that this tradeoff is not inevitable: given the assumption that there exists procedure to fit regression models that generalize to i..d. test data, we show that there exists a tractable, albeit impractical, procedure that calibrates while preserving log loss.
5.1 Definitions
For a model (Y........ Y7) mapping a prompt X to distribution A over the output space y = VT. let the future entropy of the prefix (X, Ys)) be given by
Hp (Y> Yst)= = Σ (Ysi logxx (Yst Yg.).
(7)
Yzt
Given prefix Ys.. this expression computes the model's entropy over the remaining generation Yst. We can then define the future entropy adjusted model, for parameters (a1, ..., (T)) E RT, as
.......))) (Ys, Yg.)
plonk (Y, Yex) exp (11 + a,) log (Y, Yee) H Parx
(8)
This expression adjusts each candidate token's probability based on what the future entropy would be if that token were chosen. The calibration procedure then involves fitting models to predict the future entropy of prefixes, and choosing the weights a, to calibrate the model (Algorithm 1).
5.2 Assumptions
For a distribution ý that can be tractably sampled from, we can take Monte Carlo estimate to compute the future entropy, which concentrates because entropy is bounded (Algorithm 2). However, cannot
7

---

at = argmin Le (p" parm.) Algorithm Future entropy scaling
Inputs: model p. length T, vocab V, future entropy fitting algorithm A, future entropy dataset size sample size m, prompt distribution true conditional distribution p". optimization tolerance g
0r 0, f2 2: Fort = T...., 1:
1: Initialize Og
fr+1 O.
3: Choose dg to minimize expected log loss at step t, until the gradient is at most e:
o
where a' (O,..., ..., 0, o 01+1 ... aT). (Ce: Equation 3, p Equation 9), o'
4: Fit the future entropy predictor fi as follows:
5: Sample prefixes (x()), v(i) <1-1 i=1 (1,2").
For each token V, compute labels (h(o))) by passing each prefix (x10, [x (2)-11)))
into Algorithm 2, along with inputs phopp. T, m.
7: Fit one future entropy predictor for each token v, setting f( (Yc)-1,0)) = fr, ((,,)))),,,,, where each fix is the output A { (x0, y <1-1 (((m))
8: Return (a1...., ar), (fe..., fr+1).
Algorithm 2 Future entropy estimation via sampling Inputs: model p. length T, prefix (X, Ysi), samples
1: Sample m trajectories from the model: (F:1).... :.., p00 i=1 px(2 Yss).
ltd
2: Return
t Σ 1= -e= - log 10gxx(1'' (9 v-(1)
assume that Parn can be tractably sampled from, cannot compute its future entropy naively. Instead, we will use assumedd model fitting procedure to iteratively replace each intractable future entropy term H pla (Y>t Ysa) with tractable fitted model f(X, Yst), leading the following approximate future entropy adjustment:
Fonny
fixi (Ye Yee) exp (1 + a.)) loggx aa
Then, we can initialize a = (0, ..., 0) and first fit oT for the last generation step. Next, now that the last generation step is calibrated, we can fit future entropy model fr, taking in length T prefixes and predicting the entropy at step T. Given fr. we can then fit (r-11, calibrating the second to last step. This procedure proceeds from T, ...... 1, resulting in calibrated model.
The future entropy model fitting relies on the following assumption, which states that we can fit regression models that attain good i.i.d. test error:
Assumption 5.1. Let (x(), y(())
"),,,)) " ;=1 be dataset with inputs (+) lid g and y(f
Px(. along with noisy labels (00,, Furthermore, suppose each noisy label is given by h(0
(x Y(i)) for the true label (*) Y!)) E R and noise e; sid. E, where & is mean-zero noise distribution bounded by a. Then, there exists polynomial time algorithm A which takes in
dataset of size poly(o, d) and outputs fitted model f attaining at most 8 test error:
Ex~olEyeurpy 11 Yee) (X, 8.
8

---

Empirically, neural networks can fit almost anything while attaining low test error in distribution, and the future entropy prediction problem described here not particularly complex, involving mapping set of tokens to single bounded scalar. However, future empirical work is needed to determine how accurately large neural model can predict the future entropy.
5.3 Main theorem
With this assumption, we now state the main result:
Theorem 5.2. Suppose that Assumption 5.1 holds, where each future entropy predictor attains test error 8. Also, let (a, f) be the output of Algorithm 1, where each ar is an e-stationary point. Then,
T
EmCCE (p" (1
- 2T8
E (1+
1=1
C (p"
I ( S Lip" P)
This theorem tells that if each future entropy predictor has error ō and choose each dy be an e-stationary point with respect to the log loss, the calibrated model will have entropy within O + + of its log loss each time step, and its log loss will be better than that of the original model.
Why does future entropy preserve log loss? Future entropy adjustment can be derived as a first- order approximation of globally normalized temperature adjustment; we provide this derivation in Appendix B, along with a derivation in the MaxEnt RL framework (Ziebart et al., 2008). Global temperature adjustment attains calibration long as the gradient of the log loss with respect to temperature is small (Braverman et al., 2020), which a first-order condition. Then, intuitively, first-order approximation of global temperature scaling should preserve this property.
The procedure described in Algorithm 1 is not practical to implement, as one would need to a fit separate future entropy predictor for each generation step and each candidate token, each of which involves slow data collection process based on a repeated sampling. Nonetheless, the existence of such an algorithm provides evidence that log loss tradeoff's are not inevitable in entropy calibration, despite the output space being exponentially large. One other point to note is that our analysis holds for any approximation of the future entropy that attains error d, with worse approximations just weakening the calibration error guarantee. For example, one could use the one-step future entropy (Braverman et al., 2020), or truncate to k steps instead. We hope that our theory, which establishes future entropy as the target to approximate, guides future work to achieve better quality- diversity tradcoffss than existing approaches.
6 Additional Related Work
Calibration is most commonly studied in binary and multiclass classification, with some classic algorithms including binning, Platt scaling, and isotonic regression (Platr, 1999; Zadrozny & Ellan,, 2002; Guo et al., 2017; ; Kumar et al., 2019). In the language modeling setting, Liang et al. (2023) evaluate the calibration of language models prompted to perform wide range of classification tasks, finding that models are almost always miscalibrated and overconfident. simply apply standard calibration techniques to adjust the model's outputted probabilities. More challenging is linguistic calibration, where models appear overconfident in the language they use to answer question. To address this problem, past works propose techniques based on controllable generation and reinforcement leamning (Mielke et al., 2022; Band et al., 2024). Finally, the term "calibration" is also used describe the procedure of eliminating the model's innate bias toward certain tokens when doing in-context learning, to improve task performance (Zhao et al., 2021). All of these settings are distinct from our setting, which studies the calibration of a model's 's entropy over an entire generation, and whose related work we discuss in Section 2.
such a setting, one can
7 Conclusion
We find both theoretically and experimentally that entropy miscalibration improves very slowly with scale. Furthermore, while all current methods calibrate at the cost of diversity, we provide theoretical

---

evidence that this tradeoff can be avoided. Therefore, given recent community interest in test-time scaling and synthetic data, both for which diversity is centrally important, we are excited about work which seeks to attain both generation stability and diversity simultaneously.
Acknowledgements
We would like to thank Rishi Bommasani, Sarah Cen, Irena Gao, Konwoo Kim, Suhas Kotha, John Thickstun, and anonymous reviewers for useful conversations about the paper. GV is currently affiliated with OpenAI but did this work while at Stanford. GV and SC were supported by NSF Award AF-234189) and the Simons Foundation Investigator Award, PL and SC were supported by the Open Philanthropy Project Award, and SC was supported by the NSF Graduate Research Fellowship Program.
References
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXivv2204.05862 2022. URL https: org abs/ 05862.
Band, N., Li, X., Ma, T., and Hashimoto, T. Linguistic calibration of long-form generations. In Forty-first International Conference on Machine Learning, 2024. URL cestps://openreee Jo
Basu, S., Ramachandran, G. S., Keskar, N. S., and Varshney, L. R. Mirostat: A neural text decoding algorithm that directly controls perplexity. In Proceedings of the International Conference on Learning Representations: ICLR 2021, 2021.
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https: //arxiv. org/ abs/
Braverman, M., Chen, X., Kakade, S., Narasimhan, K., Zhang, C., and Zhang, Y. Calibration, entropy rates, and memory in language models. In Proceedings of the 37th International Conference Machine Learning, ICML' 20. JMLR.org. 2020.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini. A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https: //arxiv.org org/ abs/2407.211 21787.
Daumé, H., Langford, J., and Marcu, D. Search-based structured prediction. Mach. Learn., 75 (3):9973325, June 2009. ISSN 0885-6125. doi: 10. 10.10077/10999440009951100 //doi //doi.org/10.
URL https:
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int80: 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30318-333 1-3322..... Cur- ran Associates, Inc., 2022. URL /tttps::/pproooeeiiiiii neurips cwtpccccpppccceeeeeeeeee files/iee//eeeo hales//pper////// 44222//1144..2400220666644640660 Pww.rapppppaaaaaaaeeeeeeeeeeeeee pdf.
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Gurevych, and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 889-898. Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 18653/v 1/P18- 1082. URL https: /acaa///oolnnnnogggoo org/P18- 1082.
Genzel, and Charniak, E. Entropy rate constancy in text. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 199-206, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3 1033115/0073033310331111 URL htttp:://accaatholooyy wrg/po2- 1026.
10

---

Ghosh, S., Evuru, C. K. R., Kumar, S., S, R., Aneja, D., Jin, Z.. Duraiswami, R., and Manocha, D. A closer look at the limitations of instruction tuning. In Salakhutdinov, R., Kolter, Z., Heller, K.. Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15559-15589. PMLR, 21-27 Jul 2024. URL https://ppooette httpse//proceeddiggs mlr www.:mees./////mmmm..t..mmm pweps//essss /v25// ghosh24a. html.
Good, J. The population frequencies of species and the estimation of population parameters. Biometrika, 40(3/4):237-264, 1953. ISSN 00063444, 14643510. htRp:htp:// //wp::///wwww jstor org/ (cable/2333344
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang. A., Fan, A., Goyal, A., Hartshorn, A., Yang. A., Mitra, A., ..Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodrigue7, A., Gregerson,
A., J.Saataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConelll,, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song. D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D. Garcia-Olano, D., Perino. D., Hupkes, D., Lakomkin, E., AlBadawy. E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F, Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang. J., Copet, J., Lee, J., Geffert, J.. Vranes, J., Park. J., Mahadeokar, J.. Shah, J.. van der Linde, J., Billock. J., Hong, J., Lee, J.. Fu, J., Chi. J., Huang. J., Liu, J.. Wang, J., Yu, J., Bitton, J., Spisak, J.. Park. J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K. Stone,, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala- Yeary, L., van der Maaten, L., Chen, L., Tan, L.,, Jenkins, L.., Martin, L.., Madaan, L.., Malo, L.., Blecher, L., Landzatt,, L., de Oliveira, L., Muzzi, M.. Pasupuleti, M.. Singh, M., Paluri, M., Kardas, M.. Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov. N., Bogoychev, N., Chatterji. N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P.,, Zhang, P., Li, P., Vasic, P., Weng. P., Bhargava, P., Dubal, P.. Krishnan, P., Koura, P. S., Xu. P.. He. Q. Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S.. Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R.. Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor. R., Silva, R., Hou, R., Wang, R., Hosseini. S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy. S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom. T., Speckbacher, T. Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V. Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang. X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A.. Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichennhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D.,, Li, D., Adkins, D.,, Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D.,, Dowling, E., Jamil, F., Montgomery, E., Presani, F., Hahn, E., Wood, E., Le, E.- Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk. F., Tian, Kokkinos, F,, Ozgenel, F. Caggioni. F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G. Sizov, G., Guangyi. Zhang, Lakshminarayanan, G., Inan, H., Shojanaazzrri H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H. Zhan, H., Damlaj. I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E.. Gat, I.. Weissman, J.. Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J. McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A. L., Silva, L.. Bell, L., Zhang, L., Guo, L.,, Yu, L..
11

---

Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M.. Bhatt, M., Mankus. M., Hasson, M., Lennie, M., Reso, M.. Groshev, M., Naumov, M.. Lathi, M., Keneally, M., Liu, M., Seltzer. M. L.. Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P. Bontrager, P., Roux, P.. Dollar, P., Zvyagina, P.,, Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R.. Wang. R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S.,, Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S.. Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng. S., Lin, S., Zha, S. C., Patil, S., Shankar, S.. Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang. T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W.. Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang. Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The Ilama 3 herd of models, 2024. URL https::////rat
org/abs 2407 21783.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kaufffman, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. URL https://arxie org/ abs//aas////// 11644.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL https: / http://poooeeeennn.... mlr
the
httpp/innmm......//eeeeeeeeeeeee www.ppressvvvii v00 guol7a.html
Hashimoto, T. B., Zhang, H., and Liang, P. Unifying human and statistical evaluation for natural language generation. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1689-1701, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/113/////11-11111 1169. www. 1169.
URL
Hewitt, J., Manning, C., and Liang, P. Truncation sampling as language model desmoothing. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1414-3427, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v 1/t.12/ www.://.iiiiinnn-em..22444444 URL wtg/ indings-emmll 249.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T. Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche,
G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E.,, Rae, J. W., Vinyals, O., and Sifre,
L. Training compute-optima large language models, 2022. URL https: / /ttpx/axivv//////////////aaaalla org/ abs/ 2203. 15556.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations: ICLR 2020, 2020.
Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9()) 90-95, 2007. doi: 10. 10110000MMEE.000.555
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https: //arxiv. wwerviiorg//ae abs///as/200.... 08361.
Karlin, S. Central limit theorems for certain infinite urn schemes. Journal of Mathematics and Mechanics, 17(4):333-411 1967. ISSN 00959057, 19435274. URI. http:://t:w/ jstor org/
12

---

Kumar, A., Liang, P. S.S and Ma, T. Verified uncertainty calibration. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Bus Buc. F., Fox, E., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. https: // //wgoeeeeeed.gs... neurips. cc/ httpppper//01//f1eeee. http:///e////oo06222222213311110
.df.
Kwon, W. Li, Z.. Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Lefaudeux, B., Massa, F., Liskovich, D., Xiong, W., Caggiano, V., Naren, S., Xu, M., Hu, J., Tintore, M., Zhang, S., Labatut, P., Haziza, D., Wehrstedt, L.. Reizenstein, J., and Sizov, G. xformers: A modular and hackable transformer modelling library. https://githi: thub. com/ FacebookresearchhSee ormers, 2022.
Li, Y., Choi, D.. Chung. J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling. J., Gimeno. F., Lago, A. D., Hubert, T., Choy, P., de Masson d' Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K. and Vinyals, O. Competition-level code generation with alphacode. Science, 178(66624) https: atttp::/www... htwp:wwwwwww.........w.w.ww.wwwt org doi abs 10. 1126/ sww.scciencccccccce.eeeeeeeeeccc abq1
2022. doi: 10.1126/cience.abgg 158. URL
Liang, P., Bommasani, R., Lee, T., Tsipras. D., Soylu, D.. Yasunaga, M.. Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B.. Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, A., Zelikman, E., Durmus, E., Ladhak, F, Rong, F. Ren, H., Yao, H., Wang, J., Santhanam, K. Orr, L., Zheng. L.., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q. Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang. W. Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models, 2023. URL https: :/tp:://aaree //aax//aavvv..//ee www..org/aaao h/s///bs/220 /2111.22111111 09110.
Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang. J., Pan, R., Wang, H., Hu, W., Zhang, H., Dong, H., Pi, R., Zhao, H.,, Jiang, N., Ji, H., Yao, Y., and Zhang, T. Mitigating the alignment tax of RLHF. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 580-606, Miami, Florida, USA, November 2024. Association for Computational URL atttps://aclantheet
Linguistics. doi: 11118653///222..mmllpp/ emnlp main. 35.
org/ org/2024.
Maini, P., Seto, S., Bai, R., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Ku, L.-W., Martins, A., and Srikumar, (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14044-14072, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/w//0244.acll 757. URL https: h/al///thllllllggc/cc.o
acl- org/2024.acl-long 757/.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In Proceedings of the International Conference on Learning Representations: ICLR 2017, 2017.
Mielke, S. J., Szlam, A., Dinan, E., and Boureau, Y.-L. Reducing conversational agents' over- confidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022. doi: 10. 10.1162/aac_a 004994. URL 2022. t 50/.
org/
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P. Christiano, F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, Cur- ran Associates, Inc., 2022. URL httpt: nttpss/////p neurips cctppper_riiiiiieeeee www.ffilespppppe /aper/ 2012://1ieeeeiaeeee00s3332111112 Patpaaaaoo011-PPPP---Paaa-e Seapper-oonffeenceeeemm pdff.
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf. A., Yang, E., Vito, Z., Raison, M., Tejani, A., Chilamkurthy,
13

---

S., Steiner, B., Fang, L.. Bai, J., and Chintala, S. Pytorch: An imperative style, reghhperformance deep learning library. In Wallach, H., Larochelle, H.. Beygelzimer, A., d'Alché- Buché-Buc, F., Fox, E.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: httpps://proceedinggs wttp://oowcc.nnnnnn.nneerreeeeee neurips...... http://ss..uuususseppeeeeeeeeeee f
pdf.
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Measuring the gap between neural text and human text using divergence fron- tiers. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4816-4828. Curran As- sociates, Inc., 2021. URL https: httttp:://ppoceeediinii wtwpoeeeeediinnnnnniiiieeee neurips. httppppprrrffeeeee / pdd. pdf.
Platt, J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances Large Margin Classifiers, 1999.
Qwen. :, Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang. F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang. J., Yang. J.. Yang. J., Zhou, J., Lin, J., Dang. K., Lu, K., Bao, K., Yang, K. Yu, L.., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z.,, Zhang, Z.,, and Qiu, Z. Qwen2.5 technical report, 2025. URL https: aarxivivv..././///aa//aaaaavvaa/
15115.
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. In Proceedings of the International Conference on Learning Representations: ICLR 2016, 2016.
Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Teh, Y. W. and Titterington, M. (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, volume of Proceedings of Machine Learning Research, pp. 661-668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: httpa////ooocooeeee/e mml....innnmm.........mmmm.. www.mm.rr.////....mm. press/ v9/ross10a. html.
Ross, S., Gordon, G., and Bagnell, D. reduction of imitation learning and structured prediction to no-regret online learning. In Gordon, G., Dunson, D., and Dudik, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statisties, volume 15 of Proceedings of Machine Learning Research, pp. 227-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: h/tp:////ccccceees/seeoe mlr
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.., Ferrer, C. C., Chen, M., Cueurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T.. Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang. Y.. Fan, A., Kambadur, M., Narang, S., Rodriguez, A.. Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:// //apxiiv/axxvv... org/ abs abss/300 2200/ 09288.
Verma, V., Tomlin, N., and Klein, D. Revisiting entropy rate constancy in text. Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 15537-15549, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10.18853/v1/2022 windeiiinninnnnnni... 1039. URL https://accanthoolll org/ 2023. findings-eniinnn
1039.
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agents enhances large language model capabilities, 2024. URL https: : htt///axivvv..////i//eee/e/aaev/ org. harrg//bs///0e wwwpaa//......////// .02.200004469..
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, A., Khashabi, D., and Hajishirzi, H. Self- instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 13484-1355 13508, Toronto, Canada, July
14

---

2023. Association for Computational Linguistics. doi: 10. 10.18653w1/2022..cloongg URL attps://aclanthologg org/2023. acl long. 754.
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. Neural text generation with unlikelihood training. In Proceedings of the International Conference on Learning Representations: ICLR 2020, 2020.
Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. From decoding to meta-generation: Inference-time algorithms for large language models, 2024. URL https://aaarre
16838.
Williams, R. J. and Zipser, D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):700-280, 1989. doi: 10.1162/neco. 19116///eco.999..111 1989.1.2.277
Wolf, T., Debut, I.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J.. Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In Liu, Q. and Schlangen, D. (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. (1.11663/w11222.mmmllpp.pm owg.oog////2/// 2020. emnlp-den demos. 6.
URL https: /aaalacaaanahoooe
Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings the Eighth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '02. pp. 694-699, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581135674 doi: 10.1145/77504..751551 URL https: //doi. //doi.org//11 10. 1ee111157777444 775111.
org/10
Zhang, H., Duckworth, D., Ippolito. D., and Neelakantan, A. Trading off diversity and quality in natural language generation. In Belz, A., Agarwal, S., Graham, Y., Reiter, E., and Shimorina, A. (eds.), Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pp. 25-33, Online, April 2021. Association for Computational Linguistics. URL https://aallanthollggy org/2021. htg/2221.humeval111.. hww.hal-m........
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-28 24 Jul 2021. http htppttts//prroceednggg mlr. presss/ v139/zhan 20039//ha22c..... .tmml
Ziebart, B. D., Maas, A., Bagnell, A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI'08, pp. 1438-1438. AAAI Press, 2008. ISBN 9781577353683.
Zipf, G. K. The syycho-biology of language: An introduction to dynamic philology. Routledge, 1936.
Zipf, G. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio books, 1949.
15

---

Algorithm 3 Future entropy scaling
Inputs: model p. length T, vocab V, future entropy fitting algorithm A, future entropy dataset size sample size m, prompt distribution true conditional distribution p". optimization tolerate e
1: Initialize Og
0r 0, f2
fr+1 0.
2: Fort = 1:
3: Choose dg to minimize expected log loss at step t, until the gradient is at most e:
at = argmin Le (p" || po.m.)
4
where a' (0,..., O, og 04 1, (T))
(Le: Equation Pa Equation 9)
4: Fit the future entropy predictor Se as follows:
5: Sample prefixes (x(),
(1) <t- :=1 with X Lid, q, y(i)
Px
6: For each token E V, compute labels (h(t))) by passing each prefix (x1o) [r
into Algorithm 2, along with inputs p/... T, m.
7: Fit one future entropy predictor for each token v, setting fi(X, (X21-1,0) = fe. Yer--),, where each fi, is the output A (x(),, x(1 st-1 h(r)))
8: Return (ai...., aT), (rt).,,,,t))) fr+1).
A Proofs
Recall notation: we are given prompts X E X' drawn from some prompt distribution X g, and responses Y E y drawn from the true conditional distribution Y ~ p'x for p\ E A For simplicity, let y be the set yd of length 7' strings over vocabulary V. We then train a language model p: X' + A fit the true conditional distribution p".
We say that ý is entropy calibrated entropy over generations equal to its log loss, in expectation over the prompt:
(p)) L(p" |I p),
(10)
where the total entropy and total log loss are given by
H()) Ex~gEe |- log px (Y)).
(11)
[(p") p) = Ex upx log px (Y)).
(12)
We can also write the per-step entropy and per-step log loss as
H, (f) = Ex Ev [- log (Yr &g(),
(13)
(p || p) = Ex~g Ex ~px
1 log (Y: Ye.).
(14)
Let the total entropy calibration error be given by
EntCE(p" p) H (p) C(p p*)l
T
Hz - Li( 11
(15)
Our goal will be to calibrate the model while preserving its log loss, which we will do by the following adjustment:
(em)) fix (Ye Yee) exp (1 a.) log Yet) - ox
(16)
16

---

where 011 ..... denote the adjustment parameters, and f2...., fr+1 denote future entropy predictors (with fr+1 0), whose goal is to approximate the future entropy. Using Algorithm 3 (copied from Algorithm for convenience) fit each o4,, ft. show the following result:
Theorem A.1. Suppose that Assumption 5.1 holds, where each future entropy predictor attains test error 8. Also, let (a, f) be the output of Algorithu 3, where each at is an e-stationary point. Then, we have
T
EniCE (p' 11 ((a..)
- 278 E oa)e,
t=
C <0' (1a.)))) S L(p" ()
The proof proceeds as follows: first, recall that for each step t, we choose a to minimize L. (p" pa..)) The first lemma will show that if the future entropy predictor fe+1 fitted in the previous iteration has at most å error (in expectation over Y and uniformly over Y₁ E V), then this choice of a, produces calibration- like guarantee.
Lemma A.2. Suppose that an an E-stationary point with respect Ci:
d do:
Li (p"
1 SE.
and that the future entropy predictor fit1 attains at most ō error, in expectation over Yet and uniformly over Yg V:
(e))(X, H Horn pa.
max E
(Ysa Yst) 58.
Y,EV
Then, we have the following calibration guarantee:
Ex E Simpprn
log loggd Damny (Y P50]]
ma (1))))
-ExinEyer E Evermpmy f.x (11)))
- log p (onnt a. (Yes: 820]
S (1+ 28.
This bound can be thought of as partial calibration guarantee in the sense that it allows us to swap Y, p" and V, 1/2.. in the expectation.
To show that Algorithm improves log loss, note that each a, is initialized O, so the initial model pa..) is equal to p. Then, suffices to show that each iteration of the algorithm improves the log loss, relative to the previous iteration. This statement is true by the following lemma, which states that at each step in the algorithm, optimizing C, is equivalent optimizing the overall log loss L:
Lemma A.3. Let 0tt1, Orr be set arbitrarily, and let on = at-1 = O. Also, let f be set arbitrarily. Then,
argmin C, (p" || poon))
argmin L (p" plany)
of
of
where a' = (0,.., O, d4, 0441 .. ar).
The final lemma involves showing that each future entropy predictor outputted by the algorithm attains low error and satisfies the condition in Lemma A.2. This lemma relies on the fact that the future entropy H Hgm. g (Yzt Yg+) only depends on a++1, or and fe42...., fr+1. because it only and onward. Therefore, after ax+ is chosen, the generation process is fixed for steps /. + and onward, so we can fit future entropy predictor over those steps despite not having yet chosen @1, .... 04, These facts, along with the black box fitting procedure provided in Assumption 5. lead to the following lemma:
involves generation steps
17

---

Lemma A.4. For any a = (a1...., or) and f = fr-1). and for some fixed t, let a' (O, O at.... .. OT) and f' (O....,0) 0, fit1,... ... fr-1) be the results of zeroing out the first t - 1 entries of and f. Then, we have that
H P
(Yst-1 ( H Ho P (Yst-1 Yg(-1)
for all Ys--1- (x(), r( (x()r()) 21-1.
Furthermore, suppose that Assumption 5.1 holds, and let
be a dataset with
h( gom Pe
8/x (Yse-1 [x (x + Ei,
x(0) ltd. y(i)
<t-1 ~ Ex &
for some token E V, datasel size poly(Tlog, V. 8), and some mean-zero noise distribution & bounded by log Then, letting A denote the black box fating procedure Assumption 5. 1, we have that fw, = A(D) satisfies
ExngExar (x,
H Hyon (Ys1-1 1 (( 86.
We use these lemmas to prove Theorem A. as follows:
Proof of Theorem A.I. Let a (ai,,..,,r) and f (f2, fT+1) denote the outputs of the algorithm. It suffices to show the following three inequalities for all l:
(a) Prediction error bound: the predictor f++1 satisfies
max Exerpyerrp fx, H Meen (Yse Yse) 8. V.VV
(b) Calibration bound: after iteration t of the algorithm, we have
Exwglyygrpps paa andix (115s))
ofg fenn log fxx (KsnBs))
-FxxngExcupp F Egr ai (11ia)))
log log (fenn)
axx (Yet, Pz4)
$ (1 ax)e 25.
(c) Log loss improvement: letting (1)) (0, 0, ..., aT) and f(0) = (0, ..., 0, fut1 fr+1).
we have
L (p" pl ( s C (p' plan (annfff
The theorem follows from combining these inequalities for t: first, to show that log loss improves, (fen) suffices to apply inequality (c) (log loss improvement) for all i, where Pa0i_f = plenp and
Pat++ a, p. Similarly, the calibration result follows from applying inequality (b) (calibration bound) for with triangle inequality:
EntC (p" (( = Exng [- log glen glomp (Y)]
Ey a.tix (V) | log glems (X)
T
E ExngEEsrnn Eg
Ensepplan aalix (-18gs)
log p slnut) a.fix
(Y 85e)
t=1
Exnq E
[-
log p plan
f.ff (Yees Y22)
x (11ax)
(1 ++)++20) +0].
18

---

Then, showing inequalities (a), (b), and (c) for all t completes the proof. First, note that if inequality (a) (prediction error bound) holds for all f. then the other two inequalities follow directly from the lemmas: inequality (a) ensures the condition in Lemma A.2 is satisfied, directly proving inequality (b) (calibration bound). Inequality (c) (log loss improvement) follows from the fact that orr is chosen via argmina;, Lr (p" par..) which by Lemma A.3 equivalent minimizing the overall log loss L
<(p' (p- || p2...)
To show inequality (a) (prediction error bound), first note that for t = T. it holds trivially because the future entropy is 0. For t 1, ..., T the prediction error bound follows directly from applying Lemma A.4 for each f(+1.e for e V, where each noisy future entropy label computed via parallel sampling (Algorithm 2) has mean equal to the future entropy and is bounded by (T' t) log V. O
The proofs of the three lemmas proceed as follows:
Proof of Lemma A.2. Taking the derivative of the log loss La with respect to or we have
d Le (p" || (loa)))))
da
d F Ex Ex Exmux doa
(.)" 10 softmax + as) log YYa) - oafet1(X, (Yer, 1D)]
Ex~olErrpii (1x(())- (: Yei)) (logâx( Y2d - fx41 m1
where we use f() E R denote the vector [f]))lv.. the indicator function is given by 1y, (v) = 1 iff Y, = v. and softmax (R Vl + RP applies the softmax operation, which exponentiates each entry and then normalizes the vector by its sum. Splitting this term into two expectations results in the expression
Ex Expg Exgerpx [(log (Yc (22) ((2((X,
(log px Yea) fx,1(X, (.)))
- -Ex=nExeerrp E (11a))
where the two terms differ in whether Y, ~ p² Y, plomp Multiplying both sides by (1 az).
we have
[- (log Y2)) - f1+1(X,
(1 + a1)= 2 Exepprer
-Ex=pEvarny Erinptn (1Dad) [1 + (r)+oxxxxxxx((1 (Y) (xx YaD)]
Next, noticing that both expressions include unnormalized logits for the distribution Paffxx applied to either Y, or Y,, subtract the same normalizing constant from both expressions, resulting in
(en))
|- (log Di plenn pe (Yr (Ya) f(41(XX))
[- [lo pfonn (8x Yee) (e (.))
Ex-gEverny Exmpinn
fix
19

---

Next, to turn each
conditional probability into a joint probability, log Pa (Y=x)] to both expressions:
we
can add
Exng Ey Egrup
[- (log a.fix (V,, Ke) fet1(X,
Ex ExmglEysen
[- (og (loggamm p f (Ye,, (e
Exxg Eypppay a. (1r2))
At this point, we can use the fact that f4+1 is within of the future entropy (in expectation over X q, Yu px and uniformly over Yi) to produce the bound
[- [oggarfix (V,, Yet)
(Yzt) Ys:)) Ys:)) }
(1 on)e + 23
Ex
[- log plomy
(Yse (Y Fxll )]
-Ex Ey Eyaggen oon (1Mc))
(Y Yet) - Bom parfix
a
Finally, note that the future entropy is defined as
| log log a. (Ys²) YE.)]
Pa,fix (Yst Ys.) 2
which we can substitute into the previous equation produce the desired result:
(1 + at) + +2
Ex E Sumppn
( (11ggs)
| log log (lo))) a. (8ses Ye, Yer)
[- log pa (P Se Yo, Yet)
-ExmlEverns Eeeepgan fixix ((1re))
Proof of Lemma A.3. Let denote the time step of interest. Writing the full log loss as sum over 8,
we have
T
C <(" || pa.t)
Σ C. (p"
||
1 p(
By the definition of plom) the 1-th parameter oa, has no effect on summands # t. Therefore, optimizing the entire sum equivalent to optimizing only the summand corresponding to s t. proving the desired result.
Proof of Lemma A.4. First, to show that
H Jan (Yst-1 Yst-))
gaa P
(Yst-1 Ys--1)
where a', f' are the results of zeroing out the first entries of a, f. we can simply write out the definition of the future entropy:
H gon fix
(Ys--1 Y E
(115 [-
log log (ee)) a.fix (8se-1 (S(-1)
where we can write out the probability as
T
flom (8se-1 YSL-1)= = II (ent)
(Y, Y P
f.fxx
smt
T
T softmax (1 a1) 1 Pim.a-1)
s
o.fet (Y 1 P1....:::))
20

---

the model's This expression has no dependence on the first t - 1 entries d1,.... 0t-1 of a, and no dependence on the first 1 entries f.., fo of f. proving the first half of the lemma.
The second half of the lemma follows directly from applying Assumption 5.1, where a', f and a, can be interchanged by the fact that their future entropies over steps l,..., T are the same.
B Derivations
B.1 Entropy calibration from binary calibration
Recall that for binary classifier f X' - [0, 1]. where I* - [0, denotes the true conditional distribution, binary calibration asks whether the model's probability corresponds to the actual fraction of ones in reality:
Ex~gTE (Y] fx pl p-
First, note that the right hand side can replaced by
Ex~gEy [Y fx Ex [8 fx pl.
Next, we can weaken this requirement by making the expectation non-conditional, or E Ex~gFr Y Ex~gFy-jx
which simply asks whether the overall rate ones under is the same as the overall rate of ones in reality. The most natural extension of this definition to multiclass calibration is top-class calibration,
1 [u{r= argmax fx (x)) max fx((') = p.
Ex
-p
which states that across all instances where the model assigns p probability to the top class, the actual label should be equal to the top class fraction of the time on average. Like before, we can replace the right hand side by
1{v argmax fx(o1)} max fx(e/) =p
ExngEYmNts
=ExmolEs 1 2 argmax fx(o)} max fx()') =
where Y 1& and Y fx are interchanged. In this expression, the top class probabil- ity max, fx(1') be thought of as the confidence of fx, while the zero-one loss function { = argmax,, =rgmaxx,,, fx') defines the metric the confidence should be calibrated confidence should correspond to the loss it incurs in reality. For language models, it is natural to replace the zero-one loss with the log loss, which produces the definition
Ex~gEEwws [ log (x)( | = 1 =
= =ExngEy [- log fx H(xx) = =
which asks whether the model's entropy corresponds to the log loss it incurs in reality. We study the unconditional version of this definition
ExngEy [- log fx)) = ExngEy [-
log fx)
which simply asks whether the model's entropy matches its log loss average. We study uncondi- tional calibration for simplicity, but the same techniques to calibrate unconditionally would likely work for conditional calibration well if one buckets the inputs X by their entropy H(xx).
21

---

B.2 Future entropy adjustment from global temperature adjustment
To derive future entropy adjustment from global temperature adjustment, recall that the global temperature adjustment with respect to inverse temperature a (where T = 1/ + a)) is given by
p(YI...,,YY) (لalobaا (Y YT) Ey'evr p(Y Yf)
Factoring this joint distribution into conditional distribution for each we have
Ers, Sys,P(Yst, Y1, Yst) log log D (Y, Y=x)= log Ex,ys, P(Y Vl, (xt)+ta"
(global)
Taking the gradient the log probability with respect to a, we have
d
tlgloolbbll (Y, Yet)= softmax {(1 a) p(Y Y, Y, = -)3 logp(Y<r Y, =) da
softmax {(1 + a:) pY <t. [Y, Y = ))}" log (ogp(Yen,)) en [Y, Ysr] = =):
where the first softmax is over Yst and the second softmax is over both Y, and Ys.. Simplifying this expression results in
log pY, Yxt) + E Y Sa ((
(1)g.) log p(Y YYg))
E Ygr Sund youn (.12at) (11C)) log (Y2) Yet)-
Then, the first-order approximation of log (glohal)
(Ye Yzr) centered around a is given by
d
log Pa
(global) (Yr Yea) = log (global) (Yr Yee) + da
log Pa
(global) (Y, Yet)
a=0
= logp(Yi Yea)
+ a logp(Y( Yea) + E palao ( log logp(Ys> Ysa)
Ey (-1 logp(Yzz Ye()
= (1+ log Yzι) - - Yge)] + Cy.e
where the final term is constant with respect to Y,
B.3
Future entropy adjustment from MaxEnt RL
The future entropy adjustment can also be derived in the MaxEnt RL framework (Ziebart et al., 2008), where the reward function is given by r(x, y) log with ý denoting the base model. Specifically, we can write the MaxEnt RL objective as
max Ex=g Tx (Y) aKL(p 1)
Then, the value function for this objective is given by
Vx(Ys)) F (yseopx(Sss) logíx 10g (
Ysr),
and the Q function is given by
Dx(Ye,, rx (Yr Yee) + Vx(5s))
llog (Y, Yer) Eys logíx (Ys Yge).
Using this function define the KI.-regularized policy then results in Pa:x (Ye Yee) exp (log Px (Y; Yet) (Xxx
exp {(1 a) log px (Y/ Yee) a alEy (Ys log -og [Ys)]).
which is the future entropy adjustment.
22

---

B.4 Scaling in the simplified setting
Recall our simplified setup: the model sees m tokens drawn i.id. from an a power law distribution over a vocabulary of size e, and it stores the count of each token it sees. At generation time, the model generates sequence of length L as follows: if the context contains only tokens the model has seen more than once, it behaves normally and produces the next token according to its fitted unigram distribution. But if the context contains at least one token that the model saw only once, then it instead produces the next tokens according to some derailed distribution with entropy larger by some constant Cи-
First, if the per-step derailing probability is small, we can compute expected entropy at time as follows using the binomial approximation:
(T (1 a)² Ho + (1 - -1- (1-(1))'(H CC)
2 (1 - qt) Ho + (1-
+ Cn)
Ho + q(CH,
so the overall miscalibration is given by
L
Σ
H.(p) Ho = Σ qtCH
1=1
L(L
qCH
2
Next, to characterize the scaling of the expected per-step derailing probability q with respect to dataset size m, first note that
Km,1 m
where Kmn, is a random variable denoting the number of items seen exactly once in the training set of size m. Taking the expectation with respect to random draws of the training set, we have
EKm,1
(count, (1) = 1}
E1 (count, (i) =
Σ mp:(1 -p:)m=1
where p; Z/ is the power law probability of the ith item, with Z Ei= 1/1a denoting the normalizing constant. Next, taking - oo following the infinite urn setup in Good (1953); Karlin (1967), compute
mp:(1
di
mZZi mZ - Zi Zi-am-- di
1=1
1=1
7 (m 1);g(1 1/a, (m 1)2),
a
where
Ha x) 1" fo-le-'d
the lower incomplete gamma function. Taking m + co and using the fact that 7(a, x) + I'(a) for I - 0C, we have that
K Km,1 1 - T[1 1/a),
m
as desired. This expression can also be found in Equation 23 of Karlin (1967).
23

---

C Experimental details
We study four model families (Qwen2.5 (0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B) (Qwen al., 2025), Llama 3 (1B, 3B, 8B, 70B) (Grattafiori et al., 2024), Llama (7B, 13B, 70B) (Touvron et al., 2023), and Pythia (410M, 1.4B, 2.8B, 6.9B, 12B) (Biderman al., 2023)) applied the following three datasets:
(a) WikiText-103 (Merity et al., 2017): given 128 tokens of context from a Wikipedia passage, the model is tasked with completing the passage.
(b) W WritingPrompts (Fan et al., 2018): given writing prompt from the writingproppts subreddit along with 128 tokens context from a human- written story, the model tasked with completing the story.
(c) CodeContests (Li et al., 2022): given coding problem from one of five websites (e.g. Code- forces) and 128 tokens of context from human-written solution, the model is tasked with completing the solution.
In each setting, we use 5000 examples and limit the generation to at most 1024 tokens. For generation vLLMM (Kwon al., 2023) with the xFormers attention kernel (Lefaudeux et al., 2022) and no quantization, and we use HuggingFace (Wolf et al., 2020) with 4-bit quantization (Dettmers et al., 2022) compute logprobs. All experiments plots are produced using Matplotlib (Hunter, 2007). For better readability, the entropy over time plots (Figure 2) are produced with exponential smoothing (a 0.2). All experiments are rum on 1-4 NVIDIA-A100-SXMEE
run using PyTorch (Paszke et al., 2019), and all
GPUs, or 1-4 NVIDIA RTX 6000 Ada Generation 49.1GB GBB GPUs.
D Example generations
In this section, we print excerpts from generations of Qwen2.5-14B applied to WikiText, where choose three random excerpts each from high, medium, and low entropy buckets (i.e. randomly chosen from the first, 16th, and 32nd entropy buckets). Qualitatively, low entropy generations are either repetitive or contain verbatim copies of the training set, medium entropy generations are high quality, and high entropy generations are incoherent.
Low entropy:
Generation (Entropy=0.548);
-3.15 Tropical Storm Mischa
-3.16 Tropical Storm Nigel
-3.17 Tropical Depression Seventeen
-3.18 Tropical Storm Patty
-3.19 Hurricane Rupert
3.22 Tropical Storm Sarah
33.21 Tropical Storm Tory
3.22 Tropical Storm Whitney
-3.23 Tropical Depression Twenty-two
3.24 Tropical Storm Vince
-3.25 Tropical Storm Wiloma
4 Impact
Season effects
Generation (Entropy=0.068);
Performance Review
"Performance Review" is the eighth episode of the second season of the American comedy television
series The Office, and the show's 's fourteenth episode overall. was written by Larry Wilmore and directed by Paul Feig. It first aired on November 15, 2005 NBC. The episode guest stars Melora
24

---

Hardin as Jan Levinson.
The series depicts the everyday lives of office employees in the Scranton, Pennsylvania branch the fictional Dunder Mifflin Paper Company. In this episode, Michael Scott (Steve Carell) conducts job performance reviews with his employees, and struggles
Generation (E11):)y=0.817): rebuilt superstructure in pagoda mast style, displacing her to 32,000 1. These modifications brought her speed down to 21.5 kn (39.8 km/h; 24.7 mph), causing her to be assigned second-ling line duties, conducting training operations through 1939.
Following the outbreak of World War in 1941, Yamashiro took part in the Indochina Incident in late 1940 and early 1941. Shortly before the attack on Pearl Harbor and the Japanese entrance into the war, she conducted
Medium entropy:
Generation (Entropy=2.574)) season, any confrontation between contestants or Gleib during stunt will lead girl screaming briefly anguish before leaving the set for the rest of the game. While only few teams have reached this period of the game while the game transitioned to a conclusive period, which concluded with Gleib instigating one team to perform another stunt during the bonus time. The winnings range from $500 to $5,000 for each round.
Idiotest debuted on August 13, 2014, airing on GSN. official showcase occurred the following evening on August 19, featuring the
Generation (Entropy=2.557): measured only across the glacier, not along the PIG's length, and the cross-x data interpolated.)
In May 2006, scientists found an increase of liq Celsius over warm ocean currents surrounding Antarctica average of almost 1 iqC warming over the last 100 years.
In 2005, University of Bristol (UK) researchers report, "Recent changes in Antarctic ice streams" and found that "This slowing was likely driven by piece of ice shelf breaking away from Pine Island Glacier. However, the slow down was only temporary and the effect seemed only to have been temporary."
Generation (Entropy=2.522): premiere of the thirteenth season and then departed permanently, as part of major overhaul of the cast. She returned in guest-acting role for the show's 's series finale. Abby first appeared on television in June 1979, two years after Jacobs created Dallas, series about Texas oilmen whose motivations were less virtuous than its male and female leads. He used the same theme for Knots Landing, however, the series was more regulated and politically correct. Whereas Chester's antagonists were generally viewed as brutish or psychologically ill, Abby was by definition the rich, glamorous and cunning oil tycoon's 's daughter;
High entropy:
Generation (Entropy=4.922): the Common who reads them. It made reading easy to carry the inflection marks to comwith al-Fa[] pratient al-QayTan pensal bearing 'the Mariacheron of every native" "alchemy" D the knowledge of formation through the transformation of macroscopic matter in molten liquid but usually precipitated by boiling at low temperatures for fluorine is not involved in the mineral as clays" is the quality word of decoration have a heart of rock. UThen he commits the inflection marks to a reasonable argument about the diiocean O mugeatun; represent letter D then
25

---

Generation (Entropy=7.606): J.CDIDD..SOwehppiCSsssttmmmmmm 201 iiiimaattooo//iiieviiiisssse
D w0w...Dee.eee..e.. Jer.bbe JYonnShowrksccidd..2oojomm [coordPt="]11
fs=1 telel
Generation (Entropy=5.031): varieties like Lemon Pop, Canadian Grape, Peruvian Peach, Kiwi tossed with autoimmune syrups and served on the rocks. 16. Maple Syrup recipe
When on the cuban lemons growing in the homemade garden and the layout of the lemons personalized with the heart graphics are some of the items around the quarters. When it comes to the Lijoy mosquit[were magazines
which are not available anywhere. just offing Nashville You need them find their way. Polymorphous wonders of the Cross Shades Align by universal rights and bound package for commerciality.
You may be happy Be fit
26

---

E Additional Experiments
Expectod total mass of singlitions given in samgles from power law distribattin
Zipf curves
1.0
1.25
1.5
V-100000 (low-d0 125
233 diegn-
MidTed lokgow 00031
VaTraCIEE A.w. (1ER7
BI opp. 2-11
Aap 3307
Conki qpee 1001
Figure 4: Left: the expected total mass of tokens seen exactly once, given m samples from power law distribution over a vocabulary of size v, for three settings of the power law exponent = 1.0, 1.25, 1.5. Their relationship is roughly log-log linear up to m s v/3, with slope slightly steeper than the asymptotic expression of 1/a 1. Right: log frequency versus log rank of the top 5000 unigrams in three datasets. The power law exponent a, given by the slope of each curve, is close to for WikiText and WritingPrompis, while it 1.5 for CodeContests, suggesting that text has heavier tails than code. Together, these plots suggest that the singleton mass should decay more slowly with ma for WikiText and TritingPromphl than for CodeContests.
27

---

WikiTexl
WrilingProppls
CodeContests
09
000
008
6A6
6.00
335
2.4
0
233
Laze 5 123bm
13
2-12--7
22
Jimro 2028-7
3.4
02
Liae 52-35
Utaa 22.23
3.2
1.0
0.5
0.
1.4
1/38-112 120
63
2010 130
26720-22
1.
28
35
30
60
2.0
27
33
45
8.5
0.0
1.5
0.-
Owee1.0.3
Ou192
Ovee25.00 2.30
Ove92.500
Owan2.520
Oweed 118
23
Oveed 201
Uwea 2228
Unietir 228
Entropy
Figure 5: MAUVE for excerpts of model generations plotted against the entropy (in nats) of the excerpt, with models colored by size (see Appendix E for the full plots containing all model families). These plots show that sample quality drops when entropy is too high or low.
28

---

wikitext
writingprompts
codecontests
Llama-2-7b-ht
1.0
0.05
Instruct
0.30
0.9
-122
0.0
0.00
Entropy calibration uvver
-133
-0.75
0.0.8
Inst
-185
-1.39
Llema-2-13b-44
0.75
1.0
22
0.21
0.55
Instruct
0.95
0.9
1 0.9
0.05
0.8
0.8
-C4
0.8
unstgut
-1.31
1.0
6.75 8:000
Llama-2-70b-hf
1.0
1.0
0.0
1.0
1.4
1.05
0.5
0.05
.95
0.9
-3.22
T 0.9
-0.1
00.05
-0.59
0.8
0.8
Instract
Instruct
1.30
0.8
-64
115
1.50
1.25
333
135
1.40
1,45
2335
235
2.20
222
2.30
2.35
240
2330 0005
8350
0075
8800
0622 3.50 0.55
Log Lowu
Figure Entropy calibration error versus log loss for all Llama 2 models: each plot contains per-step-averaged calibration eтror versus log loss for the base model (T = 1.0) compared to the Instruction-tuned version, along with various temperature settings.
29

---

wikitext
writingprompts Llama-3.2-18
codecontests
1.
1- 1.9.5
1.0
1.
1:95
0.00
0.85
0.85
0.8
-2.4
2.35
2.30
2335
2,40
245
1.50
225
230
2:35
3000
315
310
882
0.03
0.64
0:05
0.7
011
LJama-3.2-38
41
Instuct
Entropy calibration
T 0.95
0.0
1.8
0.85
-02
0.8
Llama-3,1-a0
Instgact
Instruck
L-sstract
0.95
1.0
-03
0.9
0.95
T- 0.85
0.8
0.8
126
1.90
2003
2200
2505
2430
2005 2ade 2012
0.00
0608 was 01d
Llama-3.1-700
isstrumt
Instruct
Instruct
0.-
1.41
0.95
0.85
010
0.85
1.20
135
1.30
1.50 2.32
234
2.30
2:33
2.90
2/2
244
245
0,00 0,30 0,500 8355
0250
0355
0268
Log
Figure Entropy calibration error versus log loss for all Llama 3 models: each plot contains per-step-averaged calibration eгror versus log loss for the base model (7 = 1.0) compared to the instruction-tune -uuned version, along with various temperature settings. Unlike the other model families. instruction tuning on Llama 3 seems to increase calibration error instead of decreasing it. Based on issues that others have also had with these models, we suspect that there might be unresolved issues with the tokenizer configuration. We use the same standard code for all models, and hope to recreate these plots when the issues with the model are resolved.
30

---

vildtext
Covenz.3 0.53
Iequt
ULD
-10
-02
114
3.0
330
3.4
Owee2.3
1.53
laggect
203
0.
355
3.
245
208
3
5033 6605 <de
coe coe
Qwen2.55 33
satrut
332
3.1
23
176
355
400
340
Qwen2,5-79
vetnvc
-10
-02
60
300
350
323
34
26.
180
350
Owen2.5:1 5-143
20
-22
2L00
-10
-02
240
120
15
2.55
440
649
Owen2.5-2 5-323
T-
50
Y2253
-22
4.5
045
2.0.
2.55
1
240
028
040
Owen2.5-27 5-723
1-1
0:35
-0.7
0.2
2.02
-44
Intruc
Figure Entropy calibration error versus log loss for all Qwen2.5 models: each plot contains per-step-averaged calibration error versus log loss for the base model (7 1.0) compared to the instruction-tuned version, along with various temperature settings.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are tied to specific experiments and theoretical results in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed throughout the paper (e.g., when discussing the power law and scaling exponents, the practical feasibility of the algorithm, etc.).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification:
The assumption is stated formally and the appendix contains full proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Appendix C for experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code will be provided upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides// htte://oooSSottiiiiooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wwtpcccppppppeeeeececceccccceeee http//udddes////eeeee
ssi.sssicccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please see Appendix C for experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Error bars provided where they would make sense (see, e.g., Figure 5) but omitted when they would not make sense or would clutter the plots visually.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етrоr rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttps//neeurr neurips....../ httpaiiicehhhcccccieeeeeeeeeee
Answer: [Yes]
Justification: The paper only uses public datasets and no human subjects. The work is mostly theoretical, and societal implications discussed below.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While our work primarily involves analysis and theory, has implications for downstream tasks like creative writing and code generation. The advancement of language model capabilities in these domains would lead to useful tools, but would also disrupt online communities and people's 's livelihoods. We hope that language models can be deployed responsibly, in ways that maintain the health and well- well-being of the communities they are trained
Guidelines:
35

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper uses publicly available code packages and datasets and cites them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/c datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pettpa:/nneurjge http://p//.//oooeeeeee////////// for what should or should not be described.
38


Reviews:
[{"review_id": "az25fheQMA", "reviewer": "Reviewer_CdPG", "summary": "This paper investigates entropy calibration in autoregressive language models, namely whether a model’s entropy over generations matches its log loss on human text. The authors first provide theoretical insights under a simplified setting, showing that miscalibration improves very slowly with scale, particularly for power-law distributions with heavy tails, which are common in human text. Empirical analysis across a range of models (from 0.5B to 70B parameters) and datasets (WikiText, WritingPrompts, CodeContests) confirms this slow scaling behavior. In contrast, code datasets show moderately better improvement due to their lighter-tailed distributions. The paper also highlights that temperature sampling and instruction tuning can reduce entropy but at the cost of increased log loss, reinforcing a tradeoff between calibration and diversity. Finally, the authors propose a theoretical algorithm that adjusts token probabilities based on predicted future entropy. They prove that it is, in principle, possible to achieve calibration without increasing log loss, although the method remains impractical due to high computational costs.", "questions": "See weakness above.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The paper provides a rigorous and intuitive theoretical analysis showing that entropy miscalibration improves extremely slowly with scale, especially under heavy-tailed data distributions typical of natural language.\n3. By linking the power-law exponent of the data to the scaling behavior of miscalibration, the work highlights a fundamental limitation of current model scaling approaches.\n\n\n\nWeaknesses:\n1. Given the authors’ definition, it is not evident that entropy miscalibration always negatively impacts model performance. In fact, in some creative generation scenarios, higher entropy could potentially enhance diversity and be desirable.\n2. While the theoretical algorithm for future entropy scaling is elegant, it remains purely conceptual. No empirical experiments are conducted to validate even small-scale or at least approximated versions of the method, leaving its practical effectiveness unclear.", "was_revised": true, "final_justification": "The clarification of the relationship between miscalibration and entropy, as well as the additional experiments, help to address some of the concerns I raised. While I still have reservations about the practical applicability of the proposed methods in certain settings, the response has provided a clearer understanding of the theoretical contributions.\n\nGiven these considerations, I will maintain my positive recommendation.", "rebuttal": "Thanks for the helpful review! We appreciate that you found the paper well-written and recognized its core theoretical and empirical contributions. From our understanding, the main weakness you highlight is that the practical implications are unclear, which we address below.\n\nBefore addressing each point individually, we first provide a high-level discussion of the paper’s impact. In the field, many take for granted that to improve generation quality, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate. This is in addition to the paper’s other main contribution, that of studying miscalibration scaling, which has direct implications for practice in that it predicts which modalities will see reduced error accumulation as models scale.\n\nNext, we answer each question individually.\n\nThe first question is whether miscalibration always hurts downstream performance, given that some tasks like creative writing may benefit from higher entropy. The answer depends on the type of miscalibration: if a model is miscalibrated in that its entropy grows over the length of a generation, that is almost always detrimental for any long-form generation task, as it is indicative of error accumulation (see, e.g., Holtzmann et al., 2020, Basu et al., 2021). This is the form of miscalibration that even the largest base LLMs suffer from: entropy initially matches log loss but increases as more tokens are generated (Figure 3). Reducing sampling temperature until the entropy curve is flat then leads to entropy matching log loss; in other words, the log loss is roughly the highest entropy we can attain while not having error accumulation. This analysis suggests that base LLMs should at least be calibrated down to the log loss to make generations stable.\n\nIf by miscalibration we mean the exact entropy target to maximize downstream performance, the specific calibration target likely depends on the setting. For settings where both quality and diversity matter, our paper provides evidence that the log loss is a reasonable target: in Figure 5 of Appendix E of our paper, we plot MAUVE (an automatic text quality and diversity metric) for excerpts of model generations against the entropy of those excerpts. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. On the other hand, for settings where we don’t care about diversity, like one-shot question answering, it might be better to drop entropy even lower, which is often done in practice.\n\nThe second question is about the practical takeaways from the proposed theoretical algorithm, future entropy scaling. First, as discussed above, the theory proves that one can avoid the quality-diversity tradeoff, which has immense practical implications for settings like test time compute scaling and synthetic data. To provide empirical support for entropy lookahead, we also run additional experiments where we evaluate the quality-diversity tradeoff for one-step entropy lookahead (Braverman et al., 2020) and compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step to adjust their logits. This procedure is impractical in that it involves processing 32 additional tokens for every single token produced, making it roughly 33x slower than vanilla sampling. Nonetheless, we find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).\n\n| Entropy Target | Log Loss (zero-step)| Log Loss (one-step) |\n|:-------------:|:--------:|:--------:|\n| 1.25 | 0.983 | **0.978** |\n| 1.0 | 0.991 | **0.985** |\n| 0.75 | 1.006 | **0.997** |\n| 0.5 | 1.035 | **1.020** |"}, {"review_id": "NAMx8IXGqT", "reviewer": "Reviewer_XzYA", "summary": "The authors study the problem of entropy calibration in LLMs, finding that miscalibration improves very slowly with model scale, especially for text data. They provide a theoretical model linking the scaling behaviour to the power law exponent of the data distribution. They propose a novel calibration algorithm, 'future entropy scaling', and prove that it can theoretically calibrate a model without increasing its log loss.", "questions": "1. Since Algorithm 1 suffers from exponential explosion, what if we use $\\hat{p}$ instead of $\\hat{p}_{\\alpha', \\hat{f}}^{(\\text {ent})}$ to estimate the future entropy? The theory would not apply but the experiments would be feasible?\n2. In 4.1, why do the newer model families (Llama 3, Qwen2.5) seem to have slightly better scaling exponents than the older ones (Llama 2, Pythia) on the same datasets? Does this suggest architectural improvements can partially mitigate this issue?", "limitations": "A comparison against Algorithm 2 from [1] is missing, as mentioned in weaknesses.\n\n[1] Braverman, Mark, et al. \"Calibration, entropy rates, and memory in language models.\" 2020.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength: The empirical finding that entropy miscalibration follows a scaling law, and the elegant theoretical connection of this law's exponent to the data's power-law distribution, is an interesting contribution. The analysis is thorough, spanning multiple model families and datasets.\n\nWeaknesses:\n1. The paper does not compare its Algorithm 1 to Algorithm 2 from [1] which is a practical one-step lookahead version of Algorithm 1. A formal analysis comparing the new approach to this baseline is missing.\n2. The main theoretical result relies on Assumption 5.1, but the test error of the future entropy predictor A might be difficult to control because the future entropy is recursively defined.", "was_revised": true, "final_justification": "The additional experiments addressed my concern although I think the practicality is still limited. However the paper is technically solid and could inspire empirical studies in the future.", "rebuttal": "Thanks for the helpful review! We appreciate that you find the theoretical contribution elegant and interesting, and the empirical analysis thorough. The main weaknesses mentioned involve specific questions about the theory, which we believe result from minor misunderstandings that we will clarify in the revision.\n\nThe first question asks about a formal analysis of one-step lookahead. This a great question which we will devote space to addressing in the revision. One-step lookahead does not attain full sequence calibration, but our analysis can be easily adapted to this case because we can simply replace the future entropy predictor with the one-step entropy, and treat one-step entropy as an alternative approximator for the full future entropy. The $\\delta$ term in Theorem 5.2 then becomes the expected error between the one-step and full future entropies. This analysis suggests that as we increase the number of lookahead steps, the approximation error should improve, making $\\delta$ smaller and smaller.\n\nThe second question asks whether Assumption 5.1 is reasonable, if it involves controlling the test error of a recursively defined quantity. Actually, one key contribution of our algorithm is that it avoids this recursion, so Assumption 5.1 does not involve any recursively defined quantity. Predicting the future entropy of $\\hat p^{(ent)}_{\\alpha}$ would indeed involve predicting a recursively defined quantity. Instead, our algorithm makes two key changes: (1) it replaces each future entropy term with a fixed function $f_t$ defined in previous iterations of the algorithm, and (2) it iterates backward from $T$ to $1$, so all subsequent entropy terms have already been fixed. As a result, each future entropy predictor $f_t$ just needs to fit a fixed target function, whose values are bounded $[0, (T-t) \\log V]$, with no recursion required. We will clarify this point in the revision.\n\nOther questions:\n\n1. On using $\\hat p$ to approximate recursive future entropy: this would be interesting to try. However, it doesn’t improve much upon Algorithm 1 in terms of practicality, which is already polynomial time, because it still involves sampling a huge number of sequences when performing calibration. One would need to sample roughly $T \\log V$ continuations per candidate token per step to approximate future entropy with constant error, so we would need to process $V T^2 \\log V$ additional tokens for every 1 token generated. Instead, we run an additional experiment where we replace $\\hat p^{(ent)}_{\\alpha}$ with $\\hat p$ as you suggested, and we also limit the lookahead to just one step so that the experiment is feasible (i.e. Algorithm 2 in Braverman et al., 2020). Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step, which is 33x slower than vanilla sampling. The resulting algorithm attains a better quality-diversity tradeoff than temperature scaling, which is equivalent to zero-step entropy lookahead, providing empirical support for the effectiveness entropy lookahead. However, is still insufficient to avoid the tradeoff entirely, suggesting that more steps of lookahead are likely necessary to further improve the tradeoff. Thanks for the suggestion!\n\n| Entropy Target | Log Loss (zero-step)| Log Loss (one-step) |\n|:-------------:|:--------:|:--------:|\n| 1.25 | 0.983 | **0.978** |\n| 1.0 | 0.991 | **0.985** |\n| 0.75 | 1.006 | **0.997** |\n| 0.5 | 1.035 | **1.020** |\n\n2. On why the newer model families attain better scaling exponents: this is a very interesting question. Unfortunately, the specific training details of three of the four model families (all but Pythia) are not publicly available. If we were to speculate, one of the main changes from past to current training practices is that newer models are pre-trained in multiple stages, with later stages (“midtraining”) involving higher quality data and specific domains of interest (like code). The pretraining data mixtures are also likely very different. Based on our analysis in Section 3, the data distribution has a large effect on how quickly miscalibration improves with scale, so changes in the pre-training data and data schedule are very likely to alter the calibration scaling curves. On the other hand, the architectures for these models are publicly available, and are very similar across model generations, suggesting that they are unlikely to be the cause."}, {"review_id": "7ply09iNHD", "reviewer": "Reviewer_vtdE", "summary": "The paper investigates whether a language model's entropy matches log loss on real text (calibration). The focus is on whether calibration improves with scale of dataset. The intuition is that calibration is poor because autoregressive models are trained on single step prediction, but what is desired is sequence level calibration, and error can accumulate. Typical measures to ensure calibration, like truncation, involve loss of token diversity. The paper shows that, under some assumptions, one can (in theory) reduce entropy while preserving log loss. The paper explores both a simplified dataset, and real ones (text and code), and find qualitatively similar scaling laws. As noted in the conclusion, one takeaway is that entropy miscalibration has a small scaling factor, i.e. improves only slowly with scale.", "questions": "Something that wasn't clear from the introduction is why calibration, and specifically entropy calibration, matters. For an outsider to this subfield, it's hard to have an intuition about why this is relevant. This is a minor point, since this issue is mostly addressed in section 2.\n\nI'd have liked to have got more motivation for the toy model in section 3. Is the point that you hand engineer the model to produce a high entropy distribution for words only seen once, which roughly resembles what happens in practice?\n\nHigh and low entropy generations are qualitatively described as being respectively incoherent and repetitive. But is this high and low relative to the log loss? I.e. do we actually know that the best results are from perfect calibration? You reference some papers, but it's mostly taken for granted in the paper that calibration is desirable for optimal qualitative performance.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The framing of the paper is clear, and the research question makes sense. Intuitions developed on an idealized model are then transferred to real text and code data. In particular, the scaling law from the toy dataset roughly matches what is fit for real data. The fact that scaling laws are dataset dependent is maybe not surprising, but useful to see. \n\nA reasonably large set of models and datasets are included in the study, both of code and human text.\n\nSection 5, as the authors admit, doesn't provide a practical algorithm. Of course, the point is just to show that calibration is possible in theory, but it wasn't clear to me whether this has practical significance in any way.\n\nIn general, I wasn't clear on what the practical takeaways from this paper were (and hence the impact). We see evidence that calibration improves only slowly with size, but what to do about this isn't clear.", "was_revised": true, "final_justification": "The authors provided a useful response to my questions about the significance of the main results. I have made a change to the significance score in light of this, and kept my overall score. My rating and review seems to be consistent with other reviewers.", "rebuttal": "Thanks for the useful review! We appreciate that you found the paper to be well-framed, the research questions to be well-posed, and the experiments to be useful. From our understanding, the main weakness you highlight is that the practical takeaways are unclear. To address this concern, we describe some practical implications of the paper below.\n\n(a) On the practical significance of entropy calibration: the main challenge of entropy calibration is that one must control sequence-level entropy for a per-step autoregressive model that can accumulate errors. This mismatch, also known as “teacher forcing” or “exposure bias,” has been the subject of a long line of empirical papers, but remains unresolved and continues to harm generation quality (see, e.g., Williams & Zipser (1989), Ranzato et al. (2016), Welleck et al. (2020), and Welleck et al. (2024)). Framing this empirical question in terms of entropy and log loss gives us a quantitative handle on the problem, letting us make principled statements about a long-standing problem in language modeling.\n\n(b) Practical implications of future entropy scaling: in the field, many take for granted that to improve generation quality, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate.\n\nTo provide empirical support for entropy lookahead, we also run additional experiments where we evaluate the quality-diversity tradeoff for one-step entropy lookahead (Braverman et al., 2020) and compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step. This procedure is impractical in that it involves processing 32 additional tokens for every single token produced, making it roughly 33x slower than vanilla sampling. Nonetheless, we find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).\n\n| Entropy Target | Log Loss (zero-step)| Log Loss (one-step) |\n|:-------------:|:--------:|:--------:|\n| 1.25 | 0.983 | **0.978** |\n| 1.0 | 0.991 | **0.985** |\n| 0.75 | 1.006 | **0.997** |\n| 0.5 | 1.035 | **1.020** |\n\n(c) Practical implications of the analysis: it is a priori unclear whether error accumulation is fundamental, especially as models continue to scale and saturate previously difficult benchmarks. Should we continue searching for an algorithm which addresses exposure bias, or will it go away with scale? Our principled framing lets us make the following statements, supported by both theory and experiments: (1) miscalibration improves very slowly with scale, and (2) the scaling rate depends on the heavy-tailedness of the data. These statements provide practical guidance: for example, they suggest that addressing exposure bias for creative writing is more pressing than addressing it for code, as the latter has faster miscalibration scaling.\n\nOther questions:\n\n(1) Additional intuition for the toy model: this is a great question which we will devote more space to addressing in the revision. A related setup which might provide additional intuition is that of n-gram models with backup: when generation begins, the model has low entropy because the prefixes are familiar and its n-gram statistics are well-populated. However, once it generates a rare token, it must back up to its (n-1)-gram statistics because its n-gram statistics are no longer well-populated, leading to higher entropy and a worse outputs. LLMs may have similar behavior, where they must back up to coarser representations when a rare token or concept is generated, increasing entropy. The toy model studied in the paper is then just a simplified / stylized version of this setup.\n\n(2) On whether log loss is always the best calibration target: this is also a great question which we will address in the revision. Recall that for base LLMs, entropy initially matches log loss but increases as more tokens are generated (Figure 3). This entropy growth is almost always detrimental for any long-form generation task, as it is indicative of error accumulation (see, e.g., Holtzmann et al., 2020, Basu et al., 2021). Reducing sampling temperature until the entropy curve is flat then leads to entropy matching log loss; in other words, the log loss is roughly the highest entropy we can attain while not having error accumulation. This analysis suggests that base LLMs should at least be calibrated down to the log loss to make generations stable.\n\nIn terms of calibrating past the log loss, the specific entropy target likely depends on the setting. For settings where both quality and diversity matter, we provide evidence that the log loss is a reasonable target: in Figure 5 of Appendix E of our paper, we plot MAUVE (an automatic text quality and diversity metric) for excerpts of model generations against the entropy of those excerpts. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. On the other hand, for settings where we don’t care about diversity, like one-shot question answering, it might be better to drop entropy even lower, which is often done in practice."}, {"review_id": "A2zRKYCviH", "reviewer": "Reviewer_xNGs", "summary": "This work studies the gap between entropy and and loss on human-written text in language model generations, and shows empirically that this issue persists across scale (and follows predictable Zipfian trends). This work also shows that is it theoretically possible to calibrate entropy and loss without sacrificing diversity or accuracy.", "questions": "1. How does scaling dataset size or diversity impact the gap between log loss and entropy? \n\n2. Could your future-entropy scaling approach be approximated with something similar to a modified beam search procedure at generated time? \n\n3. What do well-calibrated generations in text actually look like, and what are the implications for downstream model performance (e.g., what does better \"diversity\" mean? Better coherence or controllability during generation?) \n\n4. What do memorized examples (>1 token ahead) look like in terms of entropy calibration?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: (quality, originality) \n* This work is well written and empirically demonstrates that miscalibration between log loss and entropy persists even with scale, and that the scaling behaviour aligns with predictions from Zipfian data distributions. \n* The authors also show that instruction-tuned models reduce entropy but increase log-loss, which is an interesting finding that confirms empirical observations in other work showing a decrease in diversity in instruction-tuned models. \n\nWeaknesses: (significance) My main concern with this work is the lack of empirical experiments grounding the practical implications of the calibration gap, and the proposed theoretical solution in Section 5. \n\n* For the scaling experiments, diversity is not clearly defined, and it's difficult to assess whether the trends are influenced more by model or dataset size. It would be helpful to disentangle these two factors a bit more cleanly or explain how they're controlled for. \n* For Section 5, while clearly presented, is difficult to interpret in relation to existing decoding techniques. For example, would it be possible to approximate Algorithm 1 using a modified beam search that re-weights token scores based on estimated future entropy? It would be helpful to have a toy experiment here. \n* The connection between memorization is mentioned (L145-6) but raises further questions (see Q4. below). The set up of the experiments (prompting with prefixes from the dataset) is similar to how memorization is tested in pre-trained LLMs. An analysis here would help clarify how calibration interacts with memorization of passages.", "was_revised": true, "final_justification": "The author response has clarified the practical implications of their work, which was my primary concern.", "rebuttal": "Thanks for the useful review! We appreciate that you highlight quality and originality as strengths of the work. From our understanding, the main weakness you highlight is that the practical implications are unclear, and you also raise specific clarification questions about diversity, calibration, and memorization. We address these points below, starting with conceptual clarifications, then addressing practical implications and additional experiments, and finally discussing other questions.\n\n**Conceptual clarifications:**\n\n(a) The first question is about what diversity means, and whether the paper uses a precise definition. Thanks for bringing up this point, which we will clarify in the revision. The first point to note is that we take diversity to be a property of the model, not individual generations. We use the definition put forth in Hashimoto et al. (2019), which is that a model’s diversity is its log loss on reference generations. The intuition behind this definition is that log loss (also known as cross entropy or forward KL) is a coverage metric: to attain low log loss, the model must “cover” as much as the reference distribution as possible. Please see Hashimoto et al. (2019) for more intuition and experiments for this definition.\n\n(b) The review also asks about what calibration means qualitatively. The answer to this question depends on the type of miscalibration that the original model suffered from. In our paper, we show that base LLMs suffer from entropy blowup, a type of miscalibration where the entropy grows over the length of a generation. Calibrating the model then means that the generations remain coherent rather than becoming incoherent as more tokens are generated. We can see this in Appendix D, which contains randomly chosen high, medium, and low entropy excerpts of model generations. Qualitatively, we find that low entropy excerpts are either repetitive or contain verbatim copies of the training set, medium entropy excerpts are high quality, and high entropy excerpts are incoherent.\n\n(c) Finally, the review asks about the connection between memorization and calibration. The review touches on two distinct memorization-related phenomena, which we disentangle here. First, in Appendix D, we find that model outputs containing memorized training examples have low entropy, which makes sense. Nonetheless, the tendency of models to memorize examples verbatim is outweighed by their tendency to accumulate errors, so on average they are still miscalibrated upward, with entropy too high. The second connection to memorization, as mentioned in Section 3 of the paper, is that models have the capacity to remember rare tokens seen during training. This notion of memorization is less about memorizing long excerpts verbatim, and more about the model putting nontrivial mass on rare tokens. In contrast with long memorized excerpts, which drive entropy down, outputting rare tokens instead causes the entropy to increase in subsequent steps, where the model must take in these rare tokens as context.\n\n**Practical implications:**\n\nBefore addressing each point individually, we first provide a high-level discussion of the paper’s impact. The main challenge of entropy calibration is that one must control sequence-level entropy for a per-step autoregressive model that can accumulate errors. This mismatch, also known as “teacher forcing” or “exposure bias,” has been the subject of a long line of empirical papers, but remains unresolved and continues to harm generation quality (see, e.g., Williams & Zipser (1989), Ranzato et al. (2016), Welleck et al. (2020), and Welleck et al. (2024)). Framing this empirical question in terms of entropy and log loss gives us a quantitative handle on the problem, letting us make principled statements about a long-standing problem in language modeling.\n\nIn particular, in the field, many take for granted that to improve generation quality and mitigate exposure bias, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate. This is in addition to the paper’s other main contribution, that of studying miscalibration scaling, which has direct implications for practice in that it predicts which modalities will see reduced error accumulation as models scale.\n\nNext, we address specific questions individually.\n\n(a) First, the review asks for experiments which ground the calibration gap. In Figure 5 of Appendix E, we plot the MAUVE for excerpts of model generations against the entropy of those excerpts, where MAUVE is an automatic text quality and diversity metric. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. This experiment corroborates findings in past works that entropy miscalibration leads to worse outputs (see, e.g., Basu et al. (2021)).\n\n(b) Next, the review asks about empirical experiments for the proposed theoretical algorithm of future entropy scaling. To answer this question, we run an additional experiment to provide empirical support for entropy lookahead. To make the experiment feasible, we limit the entropy lookahead to one step (Braverman et al., 2020), and we compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step. We find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).\n\n| Entropy Target | Log Loss (zero-step)| Log Loss (one-step) |\n|:-------------:|:--------:|:--------:|\n| 1.25 | 0.983 | **0.978** |\n| 1.0 | 0.991 | **0.985** |\n| 0.75 | 1.006 | **0.997** |\n| 0.5 | 1.035 | **1.020** |\n\n**Other questions:**\n\nQ1: How were model and dataset size were controlled? A1: This is a great question which we will clarify in the revision. For three out of the four model families (all but Pythia), the training details are not public. On the other hand, the Pythia models were all trained on the exact same data. If the other model families were trained to be Chinchilla-optimal, then model and dataset size should be scaled proportionally. Notably, in all four model families, the scaling behavior of miscalibration is predictable, despite the fact that their training details differ. Nonetheless, they do have different scaling exponents, perhaps due to the fact that recent models are trained with higher quality data.\n\nQ2: Could future entropy scaling be approximated by a beam-search-like procedure? A2: Regarding the specific suggestion in the review, to reweight beam candidates by estimated future entropy, if we are given a future entropy estimator then we can just do future entropy scaling and beam search is unnecessary. One other reasonable suggestion is to reweight beam candidates by the entropy so far. This is an interesting approach, and can be thought of as trying to approximate global temperature scaling with importance reweighting. For the approximation to be close, the beam would have to be exponentially large, but smaller beams might still offer improvements over vanilla sampling (albeit with increased cost).\n\nQ3: How do pretraining dataset size and diversity affect the calibration gap? A3: This question is interesting and certainly worth exploring empirically. Section 3 of the paper provides theoretical predictions for this exact question, if we define diversity to be the dataset’s power law exponent. Specifically, the theory predicts that the calibration gap will scale as $m^{1/\\alpha - 1}$, where $m$ is dataset size and $\\alpha$ is the power law exponent. In other words, making the dataset larger or less diverse should decrease the calibration gap."}]

Author rebuttal:

Meta-review: This paper investigates "entropy calibration" in LLMs. Such a model is said to be well calibrated if it yields predictive entropy in accordance with its log loss on observed text. The paper offers a theoretical account of this phenomenon and characterizes it empirically. The main finding supported is calibration improves slowly with scale; this paper thus provides a "scaling law" for LLM calibration. The paper concludes with theoretical evidence that log loss improvements do not inherently come at the cost of entropy calibration. 

Overall, all reviewers agreed that this work provides an interesting new perspective on a practical problem in language modeling. Many issues were addressed in rebuttal. One reviewer quibbles that some of the results are strictly theoretical (namely the evidence that one need not necessarily sacrifice diversity for loss), but in my view this is a worthwhile contribution regardless.
Decision: Accept (poster)