Intrinsic Entropy of Context Length Scaling in LLMs
Authors: Jingzhe Shi
Qinwei Ma
Hongyi Liu
Hang Zhao
Jenq-Neng Hwang
Lei Li
Venue: ICLR 2026 Oral (2026)
Keywords: context length
intrinsic entropy
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose to use Intrinsic Entropy for understanding impact of context length on Language Modeling, and conduct experiments to validate theoretical assumptions and deductions with language and synthetic datasets.

Abstract: Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find that long irrelevant context could harm performance, while some experimentally summarize loss reduction by relevant long context as Scaling Laws. This calls for a more thorough understanding of how long context impacts Language Modeling. In this work, we (1) propose to use `Intrinsic Entropy' for explaining the impact of context length on language modeling; and (2) conduct experiments on natural language and synthetic data, validating our proposed theoretical assumptions and deductions. Our theoretical framework can provide practical insights such as establishing that training dataset size dictates an optimal context length and bounds context length scaling for certain cases.
We hope our work may inspire new long context Language Models, as well as future work studying the physics of Language Models.

Published as a conference paper at ICLR 2026
INTRINSIC ENTROPY OF CONTEXT LENGTH SCALING IN LLMS
Jingzhe Shi Qinwei Ma Hongyi Liu Hang Zhao' i Jeng Neng Hwang", Lei Li "Isttittee for Interdisciplinary Information Sciences, Tsinghua University
"Carnegie Mellon University, $CPHOS! "University of Washington "sjzworking@gmal com, {mqw21 @mailm is,, hangzhao@mail] tsiinghua, edu. en, Phongyi 12@andrew. cmu cmu.edu, 3 hwann@uu @uw @ww. lenny, lilei. ca ca@gmail com, "Equal Contribution "Equal Correspondence
ABSTRACT
Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find that long irrelevant context could harm performance, while some experimentally summarize loss reduction by relevant long context as Scaling Laws. This calls for a more thorough understanding of how long con- text impacts Language Modeling. In this work, we (1) propose to use 'Intrinsic Entropy' for explaining the impact of context length on language modeling; and (2) conduct experiments on natural language and synthetic data, validating our proposed theoretical assumptions and deductions. Our theoretical framework provide practical insights such as establishing that training dataset size dictates an optimal context length and bounds context length scaling for certain cases. We hope our work may inspire new long context Language Models, as well as future work studying the physies of Language Models
INTRODUCTION
As language-model capacity has rapidly increased and long context has become crucial for tasks such as reasoning and retrieval, recent work has focused on extending context length. A variety of methods have been proposed support long- mongccttext language models (Su al.. | Katharopou- los et al., 2020 Gu & Dao 2024 Peng et al.) 2023 Sun et al. 2024). At the same time, prior work reports mixed outcomes: some studies show that long irrelevant context worsens LM perfor- mance (Xu et al. 2024, Levy et al. 2024), some summarize gains from relevant long context as scaling laws (Xiong et al. 2024). and work in other domains such as time series shows that even relevant long context can hurt performance (Shi al. (024)). These observations call for more thorough understanding of how context length affects language-model performance.
Previous theories explain scaling laws with respect to dataset and model size (Bahri et al. 2024 Sharma & Kaplan 20200 Chen et al. (025). However, most of them do not study how context length impacts scaling laws for language modeling, thus they cannot contribute directly to the problem
In this work, we analyze the impact context length by decomposing the total loss into 2 com- ponents. As shown in Figure [1, the components optimal language model given certain context length, and the Approximation Loss, the loss caused by the gap between the optimal language model and the trained model. The total loss is the sum of these 2 components, the balance between which would possibly lead to seemingly counter-intuitive behaviors like longer context worsens the performance. Such analysis works for analyzing impact of context length on multiple scenarios, such as for pretraining targeting Cross-Entrr loss, or when evaluating down-stream metries, as shown in Figure] T
the Bayes Risk, representing the loss of an
CPHOS is academic non-profit organization.
Cooee
for experiments is available at:
https: wewggihub... wtt..ccom//ngggheeeeee
NI,PCt 1Scal ingandBounds)
We discuss about previous work Appendix

---

Published as a conference paper at ICLR 2026
Sheee Thanegiuue Son Opanee 2are
Ese fate bo Sule Soae Salaoo Martlale
Raxees Appeses
Cresul
Appper
Roour
Visible
Figure 1: Left: Total loss is decomposed into Bayes Risk (decreasing with context length) and Ap- proximation Loss (increasing with context length), so a critical point can emerge in some scenarios. Middle: Validation Loss Gap (Val Loss min (Val Loss) vs. Context Length), measured on sub- (Val
sets of TpenWeeText, where we subtract the minimum loss within each dataset-size curve (please refer to Figure [5] for the original figure). For each training dataset size, there exists an optimal context length that minimizes pretraining validation loss, and this optimum increases with dataset size. Right: Error rate of Qwen series models on the CWE task from RulerBench when a certain amount of context is masked. Critical points are visible. More results other RulerBench subsets are shown Figure
Building this decomposition, introduce Intrinsic Entropy, which measures how much in- formation available to an LLM at a given context length for a dataset. Starting from simple assumptions about Intrinsic Space and information entropy, derive the relationship among Cross Entropy Loss, Intrinsic Entropy, and Context Length. We then validate these assumptions and deductions on both natural language and synthetic data. Our main contributions are:
We analyze context-length effects through the trade-off between Bayes Risk and Approx- imation Loss, showing that longer context does not always improve performance.
We introduce Intrinsic Entropy and use it to explain language modeling behavior across different context lengths
We validate the theoretical assumptions and deductions with experiments on both real lan- guage data and synthetic data.
We hope our work may inspire future work when comes to explaining context impact and/or training new long context Language Models.
2 ASSUMPTIONS, DEDUCTIONS AND OBSERVATIONS FOR LANGUAGE MODELING
2.1 PRELIMINARIES
2.1.1 PRELIMINARY: LOSs DECOMPOSITION
It is common in MI. studies decompose the loss into Bayes Risk (the minimum loss possible, achieved by the theoretically optimal Bayesian Model), and Approximation Loss (the loss mea- suring the ability of trained model actually approximate the Bayesian Model). Specifically for Cross-Entropy loss H, we have (please refer to Appendix B.l for formal definitions and deriva- tion details):
H(P, Qi) -R Approzz =H(P, P.) DKL(Pr, Q.)
Where P. p( is the distribution of Natural Language (or our experimented dataset), Pt = p(o)(=-11)) is the Bayesian Model for context length and Q: = ((fol=--1)) is the
'Intrinsic Space", the basis of 'Intrinsic Entropy", is commonly defined the middle-layer feature repre- sentation of well-trained neural networks, and we follow this practice in the main paper. In Appendix D we also provide formal definitions for these assumptions.
2

---

Published as a conference paper at ICLR 2026
learned Language Model of context length 1. RBgges H.., Pi) is the Bayes Risk of opti- mal model (the assumed 'limit' when we have infinite data points and model parameters) and L Approz (Pt, Qt) is the Approximation Loss, which be affected by dataset size D, etc. The Bayes Risk model or data agnostic, only related to natural language itself and is limited only by visible context length.
2.1.2 PRELIMINARY: INTRINSIC SPACE
In previous work (Bahri et al. 2024 Cheng et al. 2023). common practice, the 'Data Manifold' often defined as the middle feature representation well-trained neural networks, and assump- tions are made on this kind of mid- reprreeeeennt,nnn, with experiments to validate these assumptions. (Intrinsic Space is defined as the space where the Data Manifold lies.) We follow such practice in main paper for clarity.
Meanwhile, the Data Manifold can be more formally defined by a mapping from input data to some Intrinsic Space which satisfies certain set of properties, and mid-representation of well-trained neural networks are assumed to have such properties, which can be experimentally validated. This an equivalent yet more formal perspective. In Appendix D we formally define the Intrinsic Space and derive related results in our work with such perspective for completeness.
2.1.3 PRELIMINARY: OUTLINES
In Section 2.2 we propose the definition of Intrinsic Entropy, and discuss how to bridge Bayes Risk with it, thus explaining how context length impacts Bayes Risk.
Approximation Loss, or how well the trained model learns Bayesian Model, is related to Intrin- sic Dimension previous work of Scaling Laws (Sharma Kaplan 2022 Shi et al. 2024 In
Section 2.3 we discuss more about how the context length impacts Approximation Loss from this perspective.
We further derive that the balance between Bayes Risk and Approximation Loss would lead to an optimal context length which increases with the size of the training dataset. Our theoretical deduction and experiments on language are presented in Section 3
2.2
BAYES RISK WITH CONTEXT LENGTH: AN INTRINSIC ENTROPY PERSPECTIVE
In this section we discuss to bridge context length and Bayes Risk with the concept of Intrinsic Entropy.
2.2.1 BAYES RISK AND ENTROPY IN INTRINSIC SPACE: DERIVED FROM FIRST PRINCIPLES
IInformation Entropy' is defined as the amount of information carried in the Intrinsic Space. Here are detailed assumption H as definitions:
Assumption 1. Information Entropy of Intrinsic Space for Bayes Model limz S(Pi) ((Poo) is finite, which the Information Entropy of next token prediction of language itself.
1I, 12 such that 4 < 12 S(PG) < S(Pi)). This is because a longer context contains more information.
Assumption 2.
Assumption 3. Linear Entropy Relationship: The Information Entropy w.r.t. Next Token Prediction, defined as (P)) = (P)) H (PP)). linear with the Entropy in the Intrinsic Space of the Bayes Model, i.e., Sntp definition can be found Appendix D
k S(P)) b, and
< 1. A formal
Sntp is smaller than S since the Intrinsic Space contains important information on previous tokens that are important for the prediction of future tokens, while Sntp is related only the next token. The hidden state in RNNs contain more information than only the next token to predict. For example, consider a character-level RNN that predicts the sentence *1 +
AAppendix D defines Intrinsic Entropy from a more formal perspective.
"oo avoid confusion, we 'H' for 'Cross Entropy Loss', and 'S" for 'Information Entropy".
3

---

Published as a conference paper at ICLR 2026
equal_', the next character to predict is 's', but the hidden state should contain information about answer '3' for the latter tokens.
With these assumptions, we can derive that the Bayes Risk is linear with respect to the Intrinsic Entropy:
Bayes HP) - S(P)) + Cons!
This linear relationship is observed in experiments for LMs in Section 2.2.2 and for synthetic data in Section 4.3
Note that by Assumptions and we derive: o ORggee
< 0, and limy-soo
RR 0. 8
81
2.2.2 BAYES RISK AND INTRINSIC ENTROPY: EXPERIMENT MEASUREMENT
We use well-trained Large Language Models to conduct experiments for approximating the Bayes Risk (Pi) on certain text corpora. We find that:
H(P,P)) Co + C/r
(3)
approximates the experimented behavior well on both Open WebText and other text corpora (please refer Appendix H and Appendix E for detailed figures and results different datasets).
Experimentally measure Intrinsic Entropy using Gaussian-KDE To validate the linear rela- tionship between Bayes Risk and Intrinsic Entropy (Equation 2 we measure the Intrinsic Entropy in the hidden representation space of well-trained Language Models. For a given context length, we gather the hidden state of the final layer for the last token across multiple (2 10000) samples, and use Gaussian Kernel Density Estimation (Gaussian-KDBF KDE) to estimate the Information Entropy of the distribution in this space. We also provide an alternative eigenvalue-based estimation method in Appendix
conduct experiments on multiple Language Models: Alama-3.1-8B. Qwen3-8B- RecurrenGemma--9. all evaluated subset of the OpenWebText dataset. As shown in Figure the linear relationship between Cross Entropy loss and Gaussian Bauussia-KDDE measured Intrinsic Entropy holds across different model architectures, validating theoretical assumptions:
and
Rayes ~ - S(P)) Const,
which aligns well with Equation 2 thus validating our entropy-based deduction. Note that for RecurrentGemma-PP several outlier points at very low context lengths exhibit significantly higher CE loss than other models, indicating that RecurrentGemma-99 is not good approximation of the Bayes Model those context lengths; excluding these outliers, the linear relationship still holds.
2.3 APPROXIMATION Loss WITH CONTEXT LENGTII: AN INTRINSIC DIMENSION PERSPECTIVE
Approximation Loss in the Training Scenario Previous work summarizes scaling laws (Kaplan
al.) 2020 Hoffmann al. 2022) LApproz (D) Co A/ D for dataset size D. This has
been explained from an intrinsic-space perspective in work such as (Bahri et al., [0024] Sharma &
Kaplan 2022), where a e/dim and dim is the manifold dimension of the data/model under
uniform-diie distribution assumption in intrinsic space. In Appendix D.2 we derive this rigorously from weaker assumptions (Theorem 1 Theorem 2).
'Generally speaking, the context length is an integer. Here, following previous work (Kaplan et al. 2020 Tao al. 2024). assume Rraye admits differentiable extension Raaye real-valued | and O.Baaeee (Ol to denote ABtayee (L)/Ot /t evaluated at integer I. In this sense, the derivative serves as continuous approximation to the discrete difference RBayee (I+1) 1)- RBayer (?). We use the same convention for expressions of the form (T) (I) throughout this work.
4

---

Published as a conference paper at ICLR 2026
Con Err Syteee
Llama-3.1 3.1-88
Qwen-3-8BB 3-8B-Base
RecurrentGemma..e 9B
Figure 2: Cross Entropy loss Gaussian BReussianKDDE measured Intrinsic Entropy (in nats) for three Language Models on a subset of Open Wexee:: Llama-3.1-8F (left, k =
= -0.0088, R -0.9888))) 0.9888), Qwen3-8B-Ban (middle, k 0.0026, R 0.9960), and RecurrentGemma-99 (right, -0.0174, Rt = -0.9967, with outlier points at high CE loss excluded from regression). The linear relationship between CE loss and Intrinsic Entropy holds across different model architectures.
As assumed in Section 1.2.1 the Intrinsic Dimension should increase with I. Combined with previ- results e/d,m(l) we have,
L Ayproz = Co A(1)
da Øl
(4)
< 0.
This shows that longer context makes harder for the model to approximate the Bayes model.
Approximation Loss in the Inference Scenario The analysis above considers the training sce- nario, where both training data size and context length jointly determine approximation loss. We now consider second case: a pre-trained model with fixed parameters evaluated on downstream tasks with varying visible context length losa at inference time.
In this scenario, model parameters fixed, and approximation loss Approz depends how well the fixed model approximates the Bayes model Pt... for each visible context length. As lois increases, Pi... lies in nigher-dimensions intrinsic space, making it harder for fixed-capacity model to approximate. Hence, ÖL Approw Mluis O: approximation loss increases with visible context length for fixed model.
Moreover, for a harder downstream task (e.g., one that requires information spread across a wider context range), the Bayes model is more complex and thus harder approximate, leading larger approximation loss overall. This implies that, for a fixed model, harder tasks have larger approximation loss at any given Liis.
DEDUCTION: OPTIMAL CONTEXT LENGTH
In this section, present deductions from the theory in Section In both training and inference scenarios, Bayes Risk decreases with context length / (Section 2.2 while Approximation Loss increases with (Section 2.3 The balance between these two opposing trends leads an optimal context length.
In general, let denote context length (either training context length or visible context length inference). let 6g denote task-specific parameters that affect Bayes Risk (e.g.. task difficulty y in Position-Weighted Weighted Ruler-QAI, or the distribution of relevant information across context), and let On denote modeefftat--ppecific parameters that affect Approximation Loss (e.g., training dataset size D, or model capacity). The total loss can written as:
Loss(1, 02 0m) = R Bayes (1, 02) + L Approa (1, 8m):
(5)
where Bayes f0l 0 with limy-soc Bayaee / O (Section 2.2 and Approz (ll (Section 2.3). Since Bayes is a decreasing convex function of | and L Apprea increasing in I, the derivative of total loss, Loss O RByes + Ō Approzz transitions from negative (Bayes
5

---

Published as a conference paper at ICLR 2026
MaxCti
8k tokens
Query Doc i
Doc N
Doc
Doc 2
Doc 1
distance
Query Probability P()) x (1
distance
Maxcti
Our modified Desition-Weighted Ruler-QAI (Hsieh et al. 2024) dataset. Multiple para- graphs are concatenated together with context length close to Max tt, and a question queries the "golden paragraph' (i.e. the doc paragraph with answer to that query). In the original Ruler-QAl dataset each doc has equal probability of being queried (i.e. y = 0); while in our experiments shown in Figure 4 measure LLM performance set of tasks with different hyper-parameter 7 each with different probability of querying far-away contexts.
Figure 3:
Nhe We-techeett PRLLER OAT Accuness Prreed vollee contect beugh conter brugh $5530
witreic Erireerrrreyyy Medited 2-
305
210
Figure 4: Measured results on Position- Weighted Ruler-QAI dataset. Left: QA accuracy vs. num- ber of tokens input to the Language Model, for different tasks with different y values. We observe that: (1) each curve shows a trend to increase and then decrease with context length; and (2) the critic point corresponds to smaller optimal context length for tasks with larger y (i.e. tasks requir- ing less long context abilities). Right: Intrinsic Entropy measured on samples truncated to certain context lengths. The Intrinsic Entropy shows increment of intrinsic information when increasing context length, and resembles acc-ctl curves for larger ?,.
Risk dominates) positive (Approximation Loss dominates), yielding optimal context length /" where ooLos = O. Note that Ragee depends on o.: in inference, different tasks (c.g.. different in Position- Weighted Buler-QAI) have different distributions of relevant information across context, leading to different rates at which Rayees decreases with l.
Moreover, when Am varies such that Approz decreases (e.g., more training data the training sce- nario, or stronger model in the inference scenario), the point where 0) Lpprooz balances 0 Ragesl shifts larger I, the optimal context length 1* increases. Conversely, when 0, varies such that Bayes decreases faster with (e.g.. tasks where relevant information is distributed across wider context range, corresponding to smaller 7), Bayes Risk keeps decreasing larger also leading to larger 1*
3.1 EXPERIMENTAL MEASUREMENT OF OPTIMAL CONTEXT LENGTH FOR TRAINING
We conduct experiments on a subset Open Text with sufficiently long context length. We use nanogpt (Karpathy) 2022 and train model with GPT-2 (Radford et al. 2019) architecture (GPT-2- 124M, 12-head transformers, 768-dim feature vector, with half the transformer layers (12 - 6) to reduce GPU memory for long contexts). We train GPT-2 different context lengths with different amounts of training data (200M, 250M, 300M, 350M, 500M, 750M tokens), until the validation loss increases.

---

Published as a conference paper at ICLR 2026
We show results in Figure 1 and Figure 15 As shown both theoretically and experimentally, there does exist an optimal context length, beyond which even relevant long context would in- crease validation loss of pretraining Language Models. Such optimal context length would increase with training dataset size. We also provide similar experiments to prove an optimal context length exists synthetic dataset, shown in Appendix F presented in Appendix [
More details for experiment settings
3.2 EXPERIMENTAL. MEASUREMENT OF OPTIMAL CONTEXT LENGTH ON DOWNSTREAM TASKS
The analysis above focuses training. We also study context-length effects on downstream tasks, where a trained model is evaluated with varying visible context length. We observe that optimal context length also exists for downstream tasks, and that the optimum increases with task context- length requirements. As shown in Figure & on the RULER benchmark, most Qwen3 series models show optimal context length for qa_ fwe, and ewe subtasks.
To study the impact of task properties on this "optimal context length' phenomenon, we propose a Position- Position-Weighted Ruler-QAI benchmark: instead of uniform query distribution, query probability depends on the distance of the golden paragraph to the end of input: P(x) xx (1 x L)7. Different y values correspond to tasks focused on different context ranges. As shown in Figure and and Figure an optimal context length exists for each and a smaller (i.e., task requiring more long-context ability) typically leads a larger optimal context length.
This result can be interpreted through Bayes Risk and Approximation Loss decomposition. In- tuitively, some tasks require larger context lengths to solve (i.e., the Bayes Risk for that metric decreases more slowly with context length than for metrics such as next-token Cross Entropy), so they benefit from more context. However, because model performance eventually degrades long context (i.e., Approximation Loss still increases with context length), the balance of these two terms still produces an optimal context length. More details and additional results on models and RULER subtasks are provided in Appendix A
4 PROOF OF CONCEPT WITH SYNTHETIC DATA
4.1 LIST OF POINTS TO PROVE
In this section, we conduct experiments on synthetic dataset, explaining the Bayes Risk and re- lated theories we proposed in Section 2.2 With this synthetic dataset, we would like to prove the following,
Point 1. Cross Entropy Loss is approximately linear with Intrinsic Entropy (Assumption 3 in Section 1.2.1). Shown in Section 4.3
Point By measuring Entropy in Intrinsic Space of well-trained models, one could obtain a valid
measurement that is linear with Cross Entropy Loss (Section 2.2.2) Shown in Section 4.4
4.2 CONSTRUCTION OF SYNTHETIC DATA: THE 'POSITION WEIGHTED MULTITASK SPARSE PARITY' DATASET
In previous work, common practice is to mask the leftmost tokens and leave tokens before the token-to-predict visible to Language Models, as shown in Figure 5 Although this may not show the impact of important tokens to final answer perplexity (e.g.. it fails to show the importance of the second key info in Figure 5) this method aligns well with our setting increasing context length.
Although the next token to predict might depend several pieces of key information, see from Figure 5 the first key token would raise model perplexity.
Inspired by this concept in Figure 5 and the 'multitask sparse parity dataset previously studied in (Michaud et al. 2024| Barak et al. 2022), we propose the dataset. In detail, each input consists of L 'context bits, each bit lies in {0, Each subtask takes xor on two certain bits the context bits, and the answer to some sample is the answer of the only activated subtask, shown in Figure 5 We use 60 context bits and 200 tasks. From 1 11th to
multitask sparse parity

---

Published as a conference paper at ICLR 2026
Cross Entropy
h.jeuareee lntevddon
only Subtask33 active Subtask1Subtask Subtask2 $ubtask33
inttre senw, Venatil Info
Ther seree secind
the onginal Moddl Sul
0010..0
100101101.0011
Oundpm Varsatie
Kay Info
fituiiiee 'iiversa
Ountayy "Soelbe and lokkey
pll
00t0-0, bumbor
Mooked loft nost
control bits
Visible context
for /
context
jolcc years after the ensroo
Quastion
bits
onggnial sacves Arsonn by
requerng
Key
Figure 5: Left: An example of the 'two needles in a haystack' task, similar to those in (Levy et al,, 2024). The text part is the input to the Language Model, with key information and question visual- ized in blue; the figure part shows perplexity of the answer token (8) of LLaMa-3.1-8P (horizontal) number of masked leftmost tokens (vertical). Although seeing both pieces of information necessary to answer the question, perplexity rises dramatically only when the first piece of informa- tion masked. Right: An example of our synthetic data. Each sub-task corresponds to 2 context bits of fixed position. At each time, exactly one sub-task is activate, and the ground truth output is calculated by taking XOR over the 2 context bits of the activate task. As shown in the example, the answer for Subtask 1,2,3 is O ④ 0 = 0, 0 1 = and + 1 = 0 respectively, but since the thrid bit for control bits, only Subtask 3 activated and the final answer However, for a model of context length 7, it cannot see the 9/h bit required by subtask 3, making it unable to predict the answer correctly.
the 60th bit, each bit corresponds to the max bit of two tasks: #Task (en bitt)) 2, Vi {11, 12, .......00 60}.
We assign different frequencies to different tasks, approximating the real-world situation where tasks requiring nearer bits are more often. In all, Bayes Risk, or the minimum Cross Entropy Loss, is:
Rayes (ctl)
=(
Σ
freg(task) 2)/
Σ freg(task) task
task s.t. (axt bitz)
xA+ + B/(cll C7)0
More details are shown in Table 1
4.3 TRANSFORMER-BASE SYNTHETIC MODEL WITH ENTROPY MEASUREMENTS
We use a 3-layer causal Transformer, with embedding dimension 208 and FFN dimension 832, RoPE embedding with base frequency 4000; input sequence length is always 60+1, with 60 context tokens (cither 0, 1 or ?) and task tokens (chosen from task tokens of vocab size 200).
We use 100 tasks and 60 task bits. From 11th to the 60th bit, each bit corresponds to the max bit of two tasks: that is, #Taski mar(bit) bitg) = 2, Vi E {11, 12, .......... 60).
During training. 50% of the samples are unmasked, while for the other 50% samples, we mask the last X task bits to be 0.5, where X is random int from 60 10 to 60 60. This ensures our model to be able to handle mask bits, and also ensures can learn uncommon tasks (relying on context bits that are at the end of the context bits) well. We train the model large enough dataset that it approximates the Bayes Model well (please refer to Table in Appendix for more details).
After the model has been trained, we measure its eigen values, shown in Figure 7 is shown that: (1) Larger context length contains more information, hence eigen values in Intrinsic Space degrades slowlier (left figure): (2) the model approximates the theoretical Bayes Model well (as the green points in the middle figure is very close to the orange ones) (middle figure); (3) CE Loss follows a very good linear relationship with sum of log eigenvalues of the first N' dimensions for N 2 70 in
8

---

Published as a conference paper at ICLR 2026
MLF
Wfrilsic estimation
Causal Transformer Decoder Layer
Causal Transformer Decoder Layer
RoFE Entoedding
? ?
2
Tabk
takers,
[-tasid,,, <taskk>, StaskTs)
Figure 6: Transformer and Rope-based based model for the synthetic task. Here, we use one task token to encode the task information.
Eggnwaluus of Mid Features
tsum JsN |logireistive eigenvolue IJ]I for different Ns
Correlation between CE Loss and Measured Entropy
1.7
17ca
Exp cE-
Thoo
30
Byotekee Max Egpondae
20-1
30cl
20-4
100
130
035
120
130
180
18
140 100
100
200
150
0.93
Index of Ligenetie
Figure Eigen value and CE results measured on trained model for Synthetic Dataset in this section. Left: Eigen Value vs. Index of Eigen Value; Right: Correlation between Cross Entropy Loss and Measured Entropy. We see good linear relationship between CE Losses and Measured Intrinsic Entropy from Lower figures.
the Intrinsic Space (right figure), where the case 200 (all eigen values) are also shown in the middle figure.
This validates Point 1: Cross Entropy Loss is approximately linear with Intrinsic Entropy mea- sured by the sum of log eigenvalues.
4.4 ENTROPY IN INTRINSIC SPACE: SYNTHETIC DATASET VALIDATION
Figure shows the measured results of Intrinsic Entropy on synthetic dataset, which follows linear relationship with the Cross Entropy Calculated (Theo CE) and Cross Entropy loss measured (Exp CE).
This provides evidence for Point 2 in Section 4.1
measure entropy in the intrinsic space using eigenvalue-bass methods or density- based methods, and both show linear relationships with Cross Entropy Loss, validating entropy- based theoretical framework.
CONCLUSION AND DISCUSSIONS
5.1 CONCLUSION
In this work, we study the impact of context length on language modeling, especially through Bayes Risk and Approximation Loss, from both theoretical and experimental perspectives.
In Section 2 we propose assumptions on the relationship among CE Loss, Intrinsic Entropy, and context length. We derive linear relation between CE loss and Intrinsic Entropy, and analyze

---

Published as a conference paper at ICLR 2026
how context length affects Intrinsic Entropy. We further investigate the relationship among Intrinsic Entropy, context length, and Intrinsic Dimension in Appendix K]
from an Intrinsic Dimension perspective. We also provide formal definitions of assumptions and derivations of key theorems in Appendix
We also conduct experiments both real data (Section 2 Section B) and synthetic data (Section
measuring Intrinsic Entropy and validating the relationship among Cross Entropy loss (Bayes Risk Approximation Loss), context length, and Intrinsic Entropy.
As a corollary of our theory, an optimal context length exists and increases with dataset size in pretraining; this is validated in Section 3 For downstream tasks such document QA over long documents, we also observe an optimal context length that increases with task context-length re- quirements for a fixed model, as shown in Section 3 and Appendix AA] We hope our work provides useful insight for future work on long-context language models and on the broader physics of lan- guage models.
5.2 LIMITATIONS AND FUTURE WORK
Our theory starting from Intrinsic Entropy only holds with assumptions in Section 2 and in Ap- pendix K we use the perspective of Intrinsic Dimension to (partially) explain our assumptions and measurements w.rt. Intrinsie Entropy. We hope future work may try to propose even more funda- mental theories to explain our Intrinsic Entropy measurements.
In our work, similar several previous work (Bahri et al. 2024] Aghajanyan et al.) 2021 we explain the impact of context length scaling from the perspective of Intrinsic Space (or Data Manifold), the neural network (that maps the data into
which is related not only data, but also potentially such Intrinsic Space) and the prediction task (Bahri et al. [022]. Our explanation leans toward how the model represents the data its intrinsic space and hence more related real language models, meanwhile other types of more model-agnostic explanations might also be proposed.
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning, 2020. URL
13255
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 7319-7328, Online, August 2021. Association for Computational Linguis- tics. doi: 10.1865..1 00.18653/v1/2222 lcl- URL https: / //ccl//ccc/ ant org/2021.
ael 8el-1ong.500 oog. 5687
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27), June 2024. ISSN 1091-6490. doi: (0.1073/pnas.2311888822 2311878121
URL/ttp::/ //dp dx. oww.oo///...173/////
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham M. Kakade, eran malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL htRlstttp::// httpptss///ppeeeeiiieeeeeee /wtp/epeeevvveeeeeeee net feet/foruret forum?id 8XWP 2ewx im
Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. The role of deductive and inductive reasoning in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume| 1: Long Papers). pp. 16780-16790, 2025.
10

---

Published as a conference paper at ICLR 2026
Zhuo Chen, Oriol Mayné i Comas, Zhuotao Jin, Di Luo, and Marin Soljacic. LS-2$m: Mutual information scaling law for long-context language modeling. In The Thirty-ninth Annual Confer- ence on Neural Information Processing Systems, 2025. URL forum? i d=s.dmm-.--mmmmmm. 5M
Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging thformation-theoreticc and ge- ometric compression in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 12397-12420, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v 1tt5533//12222mmmmmmmml main.762. @mm.p main. 762/
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang. Ruizhe Pan. Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li. Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2024. URL wwwp://wwwrrggp/oop/poooo abs/ 2412. 19437
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aicsha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes. Egor Lakomkin, Ehab AlBadawy, Elina Lobanova. Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid
11

---

Published as a conference paper at ICLR 2026
El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan. Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marein Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov. Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero. Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers. Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Haho, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanann Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill. Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar. Ozlem Kalinli, Parkin Kent, Parth Parckh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy,
12

---

Published as a conference paper at ICLR 2026
Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang. Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li. Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL /ttps://arxiin wwwpxxvvv..ggaaaa. httg/abb/24407 21783
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.
URL htttps://aaxiv..oto org/ abs. 2375.007522
Alexander Havrilla and Wenjing Liao. Understanding scaling laws with statistical and approxima-
tion theory for transformer neural networks on intrinsically low-dimensional data. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
7/ openzeviiew wwt.:oermm..d-2222tr..ee.. N2wYddw22
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen. Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL
org/ /abbbbs//////oooooo//.///////b/b
#203.15552
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's s the real context size of your long-context language models? In First Conference on Language Modeling, 2024.
opw.revieeeeeeeeeeeeeee net forrforrm?
1d=kIoBbc768y
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL /ttt::///ariiv.oorr/////0.....1.
08361
Andrej Karpathy. NanoGPT.
2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention, 2020. URL https::/arxiv.
org/ hkg/abs/2006. 16236
Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, and Lei Li. Attention consistency for Ilms explanation. arXiv preprini arXiv:2509. 17178, 2025.
L.D. Landau and E.M. Lifshitz. Statistical Physics, volume 5 Course of Theoretical Physics. Hntterworth-Heinemmnn Oxford, 3 edition, 1980. ISBN 978-0-7506-3372-7.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length the reasoning performance of large language models, 2024. URL abs/24022 14848
Lei Li, Sen Jia, Jianhao Wang, Zhongyu Jiang, Feng Zhou, Ju Dai, Tianfang Zhang, Zongkai Wu, and Jenq-Neng Hwang. Human motion instruction tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17582-17591, 2025.
13

---

Published as a conference paper at ICLR 2026
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2024.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Chung. Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozmiak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu Rwkv: Reinventing rnss for the transformer era, 2023. URL https: /tttpp://axivv /arxiv.vvvv. httg/abs/22n 2305 13048
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https:: attta::/aaxxv..o///aaa... wwtpor/gg//b/lr.eeeeeeeeeee 10683.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL
05250
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold, 2020. URL
Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022. URL http::/
10-111.1hmm
Jingzhe Shi, Qinwei Ma, Huan Ma, and Lei Li. Scaling law for time series forecasting. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:///////
openreview www/tetteeeee wtr/mooruummmm faneruummiii-e - (1 EHJB9q
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. 1I04.09864
org/abs/
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn test time): Rons with expressive hidden states, 2024. URL https:://axxvv.org
abs/ 00te/2407..0 04620
Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong. Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads, 2024. URL https:
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong, Scaling laws with vocabulary: Larger models deserve larger vocabularies, 2024.
URL httts:://tattttt
org/ abs
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang. Prajjwal Bhargava, Rui Hou, Louis Mar- tin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation mod- els. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 4643-4663, Mexico City. Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 003663//1/2222/ 1/2024 nawcl- long..260.. URL
https: ///s:///cl antho wow.ooloyy...ooo...o...o wrwp://gg224444 naacl m 260
14

---

Published as a conference paper at ICLR 2026
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models, 2024. URL https::tt////....... http://axxivv.oooo att///bbs//210./ 03025
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: wwwpt:ttttt/ttttte htap:s://ariiie org/ abs. 2505 09388
15

---

Published as a conference paper at ICLR 2026
A DOWNSTREAM TASK
In the main paper, we experimentally discover and theoretically analyze how context length impact Cross Entropy loss for next token prediction. Previous studies (Hsieh al. 2024) show that Cross Entropy Loss might not be highly correlated with important downstream tasks.
In this section, we study the impact of context length on downstream document QA tasks, similar to those proposed in Ruler-QA1 (Hsieh et al. 2024). The conclusions we observe in this section are:
For downstream tasks studied (i.e. similar to Ruler-QA document QA tasks), optimal context length still exists, and this phenomenon can be analyzed from the perspective of Bayes Risk and Approximation Loss.
2. For these tasks, Intrinsic Entropy can still act as a proxy of information learned, and when Language Model is not deviating from Bayes Model by large margin, is still positively-coorrlateed with QA accuracy.
A.1 OPTIMAL CONTEXT LENGTH CILER-QA: A CASE STUDY
Ruler-QAI (Hiih et al. 2024) is representative among series of doc-QA tasks in the sense that (1) it is composed of woal-world documents and QA pairs from Doc-QA vac-QA tasks like SQuAD (Rajpurkar ltt al. 2016); (2) its samples are generated by inserting 'golden paragraph' (i.c. the paragraph con- taining answer to a specific question in SQuAD) into other paragraphs sampled also from SQuAD, hence one can control the total length of tested samples. This provides with great testbed for experimenting the impact of visible context length to models, and to test the Intrinse Entropy. The task shown in Figure 3 (with 7 0 setting) aligns with the Ruler-QAI dataset.
The measured results are shown in Figure in the main paper. We study the performance of Llama- 3.1-8B. For the original Ruler-QAI dataset, the 'golden paragraph" is inserted randomly and uni- formly across the sample. We first generate samples with maz_ctl = 16k, i.c. each sample has length close to 16k tokens and is composed of multiple paragraphs sampled from SQuAD, and we test the Language Model to answer question related to some certain paragraph uniformly dis- tributed across the context. During test, we only allow the Language Model to see the closest 'ctl' tokens (and hence it cannot see previous paragraphs), and measure accuracy varying this 'ctl'. The result shown in the line labeled as 'uniform distributed' in Figure 4 As shown, though the accu- racy first increases when increasing context length, but the context length drops after ctl 6k. This proves the existence of an optimal context length, for Llama- 3.18-BBBBBBBBBBB on Ruler- Ruler-QAI.
To further study how this optimal context length depends the property of tasks, propose the Position-Weighteed Weighted Ruler-QA: QAI datasct. We test the Language Model to answer a question related to certain paragraph sampled by: P(x) (1 2/ L)7, where is the distance of the paragraph to end of input (in tokens), L is the maximum context length (i.e. 8k), and 7y is a hyper-parameter fixed for certain task. 7 degrades to uniform distribution (i.e. the standard Ruler-QAI task), while a larger 7 means the task focuses less on far-away tokens. Similarly, for fixed 7, we adjust the number of tokens visible to Language Models ('etl') and measure its accuracy; results are also shown in Figure 4 From the figure, we have two observations: (1) an optimal context length exists for each 7; and (2) a smaller 7 (i.e. task requires more long context) typically leads to a larger optimal context length.
More detailed analysis of this result from the Bayes Risk and Approximation Loss decomposition perspective can be found in the main paper (Section 3
A.2 INTRINSIC ENTROPY: A PROXY OF INFORMATION LEARNED BY LANGUAGE MODEL
We measure Intrinsic Entropy of different context length the Doc paragraphs samples we con- struct. In our experiment, we take the closest ctl tokens of the concatenated samples as input to Language Model (Llama-3.1-8B) and take the hidden state of the final layer of close-to-the-crt token. After obtaining N such vectors, we conduct Gaussian- -DE to measure the Intrinsic Entropy. The result is shown in the right figure of Figure
16

---

too simple for sota Published as a conference paper at ICLR 2026
Pertermance sadtinkk MaxCt
Performancs sastase
Deec 10 Ovesl-6
S1612-00 30
612
022
900
713
Vide Iac0c
Pertermence LAs
Petermanto sobaakk $132
Swech
100
Qvent 1ea Ovetl-0
O1e13-4
V1000 F1312
Figure 8: Ace vs. Visible Context Length of Qwen-3 series models toon-thinking chat models) on 4 representative subsets of the RULER dataset: qa (document qa, upper-left), fwe (frequent word extraction, upper-right), cwe (common words extraction, lower-left), and vt (variable tracking, lower-right), for a fixed max context length and varying visible fraction of the input context. Most models show an optimal context length for qa_l, fwe and ewe subtask, while the subtask shows increased performance with respect to context length. Moreover, larger model tends to perform better and have a larger optimal context length, represented by the performance comparison between Qwen3-4B and Qwen3-8B cwe subtask (lower-left).
We observe from Figure 4 that, the Intrinsic Entropy still increases the input context length in- creases. Moreover, though measured Intrinsic Entropy does not always follow a linear relationship with QA accuracy (notice that the Intrinsic Entropy calculated does not depends on y while QA curacy is related to the task setting 7.), we still see positive correlation between Intrinsic Entropy and QA accuracy when context length not very long.
In principle, drop in accuracy when increasing context length actually implies that the model is no longer good approximation Bayes Model for certain task that context length. For relatively larger tasks (i.e. tasks focusing more on nearer tokens), we see more aligned trend in increment of QA accuracy and Intrinsic Entropy; while for smaller y tasks (those focusing on farther tokens), the QA accuracy might increase a lot when Intrinsic Entropy increases little. This potentially implies that Language Models are memorizing and keeping only the information likely to be useful from farther-awar away tokens, and these pieces of information sufficient for the QA task.
A.3 MORE EXPERIMENTS ON RULER BENCHMARK
To see how different tasks might have different behaviors with respect to context length, we further conduct experiments on three RULER subtasks: the qal subtask (document qa), the ewe subtask (i.e. common words extraction), the vt subtask (i.e. variable tracking), and the fwe subtask (i.e. frequent words extraction). Other subtasks like single-needle-in-haystee LLMs hence we did not perform experiments on them.
To study the impact of model size, we utilize the Qwen3 series models. We use the non-thinking mode of the chat models of Qwen3 series Yang et al. (2025), including Qwen3-4B, Qwen3-B -BB, DWen3-14B and Qwen3-32B for experiments. We use codebase modified from RULER et al. (2024). The maximum context length is set to 16k for owe and 8k for other subtasks. These results shown in Figure
17

---

Published as a conference paper at ICLR 2026
Owen3-48 performance. MaxCC 9192
Qwend-48 parformancs.
50-
04
1.0 0.2
0.
10
Figure A.3-2. Acc vs. Visible Context Length of Qwen-3 4B, on cwe (common words extraction) subtask, with different Max context length (left: MarCti 8k, right: MarCtl 16k). As shown, though optimal context length is hard to observe for the task requiring 8k as max context length, it is easy observe for that requiring 16k as max context length.
As shown in Figure 8 (1) for subtasks resembling fwe, document qa, variable tracing, etc., there exists an optimal context length for most models tested; and (2) for vt (variable tracking), in the experiment we conducted models' performance improve with respect to visible context fraction. This could be caused by the fact that variable tracking is relatively easy for current LLMs, thus their approximation loss is low; while given its distribution of variables the Bayes Risk would constantly decrease with more context length, thus it shows a trend of improving even for long visible fraction.
Comparing the results of Qwen3-4B, Qwen3-8B and Qwen3-14B the owe subtask in Figure we see that: (1) optimal context length is larger and harder to observe for larger models, this can be attributed larger models lead less Approximation Loss; comparing results of the same ewe subtask on different max context lengths in Figure AA3-2] we see that: (2) optimal context length is easier to observe for longer task, which can also be attributed a larger Approximation Loss (i.e. language models fall short to deal with longer contexts).
These results could potentially argue that the subtasks defined in RULER are of different difficulty for current LLMs. That is, if one can observe optimal context length, this proves that the model gets distracted by more context beyond the optimal context length, and hence performance gets worse even if these context contains more information. Potentially, our work provides a new perspective: Consider a case where some specific LM achieves 95% accuracy on certain subtask with 0.8 visible context fraction and 90% accuracy on it with 1.0 visible context fraction. Even though the absolute accuracy numbers are high, the existence of optimal context length and degraded performance also implies an ineffectiveness of the language model when handling long contexts with respect to that specific task.
B DEFINITION AND PROPERTIES OF CROSS ENTROPY LOSS
B. DEFINITION CROSS ENTROPY Loss DISCUSSED THIS WORK
well-known that the original definition of Cross Entropy between two sequential distributions P and Q: Horg (P, Q) should be:
Horg (P,Q)= = Σ - P(x) log ((x))
Σ P(x P(zol
where Tazb denotes Tai Po+1..... Tb-1; it is common practice calculate perplexity of Language Models with its input as GT labels (e.g. in technical report of LLaMa-3 3 et al., (024)), in
other words, the experimentally measured Cross Entropy Heap (P Q) is actually:
18

---

Published as a conference paper at ICLR 2026
(eap(P,Q)
Σ
x
=Contt Ex Σ
P-P log Q(zo|z-cool
20
Therefore, in this work we use:
H(P.Q)
Herp ()
(6)
=E, 2000 Σ (( log
jr-
as the definition Cross-Entropy loss, and P(zol Q - se:0) as the definition of Natural Language distribution and Language Model distribution, respectively.
B.2
CROSS ENTROPY LOSS FOR LANGUAGE MODEL WITH CONTEXT LENGTH
Q1(00 (0000) is a language model with limited context length Qi -o:00) (2 -10),, we have:
Equation
B.
(PP, Qt)
E 0c:0
- P log Qı [10)]
Σ P(x 1(x- log (1(20(=-11)
P(0-o0--1--111 log Qi(zolx-co)
T oe:-l 1-111
Σ P(=-1a) log (Fol
#-i1
Ex ( log
H (H, Qi).
Note that P(zg|z 1:0) is exactly the Bayes Model with context length I. Hence, we have:
(kI(P,Qi) = H(P,Q) - H Q1)
-- + (RI Q1).
Specially, if we calculating the KI. Divergence between Nature Language and Bayes Model with
context length 1, thus Qi = Pt. we have:
= DKI(P,P) = -()
C EXPERIMENTALLY MEASURE NEXT-TOOKEN-PREDICTION INFORMATION ENTROPY Snop
C.I PCA-BASED INFORMATION ENTROPY ESTIMATION
Though related. Entropy in Intrinsic Space does not equal to Entropy in the next token prediction task. From the probability perspective, let dec(x) be the next decoded token for some point in the
19

---

Published as a conference paper at ICLR 2026
intrinsic space, have: S - -P(x) log P(z), while Sntp -
- Evocad P(u) log P(o)
where P(v)= = de P(x). Sntp is a coarse-grained Entropy compared to S. S contains important information on previous tokens that are important for the prediction of future tokens, while Snop is related only to the next token.
Experiments in Figure 14 show that, matter what subspace we use, the Cross Entropy Loss usually follows a linear relationship with the Entropy we measured in the subspace, supporting the claim that the next token prediction task likely lies in some subspace of the Intrinsic Space, or (statistically) its Entropy should be some weighted average of Entropy of several subspaces of similar dimension. This also suggests that Hntp is approximately linear with His, which validates our previous assumptions and claims.
C.2 GAUSSIAN- KDE BASED INFORMATION ENTROPY ESTIMATION
In this sub-subsection Figure / this estimation also aligns well with PCA-based estimation; moreover, such estimated entropy is also linear with respect to Intrinsic Dimension and Cross Entropy Loss.
another method for Information Entropy Estimation. As shown in
CALSS mastare LOROO FSATMMUNN
JALLEM Libiea rassuneaatt
Figure 9: Gaussian KDE measured Entropy (10000 samples, auto bandwidth 0.997756) vs. PCA- measured Entropy (left), Measrued ID (middle) and Cross Entropy Loss (right).
C.3 SYNTHETIC DATASET: ENTROPY INTRINSIC SPACE, AND ENTROPY FOR OUTPUT LAYER
For our synthetic dataset, if we view the Context Feature Vector shown in Figure 17 as the feature in the Intrinsic Space, then the best strategy for the context encoder is to generate the answer for all subtasks (it can see) in the Intrinsic Space (since it cannot see the task bits). This would lead to an Entropy of 5 log 2 in the Intrinsic Space.
The entropy of the output layer is, however, Soupput log 2 since the answer bits 0, 1 have the same probability. In this way, the answer of the output layer actually corresponds to one dimension in the Intrinsic Space, which should be the exact dimension at which the answer of the current task is stored. Therefore, Soutput = 1/T * Sis, which explains why the Entropy for output logits is linear to the Entropy for Intrinsic Space.
C.4 DETAILS FOR SYNTHETIC DATA
Here we present details for synthetic dataset and model training.
We train the model on training set of 10000000 and validation set of size 1000000, for 125 epochs (and an early stopping setting 25 epochs, though the training process did not trigger early stopping).
To make sure that the trained model can be used to approximate the Bayes Model, we compare the model's loss on validation set with context cl/ with the calculated minimum possible CE Loss for the task. As shown in Table [ that the model is not too different from the Bayes Model: the BCE Loss only differs by around 0.001. Thus, context feature in Figure 6) as the feature in Intrinsic Space to approximate the Bayes Model for 17 < ell 50.
use the middle-representatio (shown as
20

---

Published as a conference paper at ICLR 2026
MinCELoss(ct))= =
Σ
freg(task) log 2)
Σ
freq(task)
task s.t. man(bit, bitz) > cll
task
Context Length
Model CE Loss
Minimum CE Loss Calculated
17
0.4648
0.4643
20
0.3988
0.3988
23
0.3438
0.3437
25
0.3119
0.3116
28
0.2687
0.2686
30
0.2429
0,2429
35
0.1867
0.1864
40
0. 0.1390
0.1387
50
0.0613
0.0612
Table 1: Comparison between trained model and Bayes Model (minimum CE Loss) for Synthetic
Data
D DEFINITIONS OF INTRINSIC SPACE AND DERIVED PROPERTIES
As mentioned in Section 2.1.2 in previous work (Bahri et al. 2024 Cheng al. 2023), as a common practice, the 'Data Manifold is often defined as the middle feature representation of well-trained neural networks, and assumptions are made on this kind of mid-representation, with experiments to validate these assumptions. (Intrinsic Space is defined as the space where the Data Manifold lies.)
Meanwhile, the Data Manifold can more formally defined by mapping from input data to some Intrinsic Space which satisfies a certain set of properties, and mid-repressenattion of well-trained neural networks are assumed to have such properties, which can be experimentally validated. These two perspectives are actually, equivalent to each other:
Experiments show miirrepresentttions of neural networks have certain properties - Data Manifold in Intrinsic Space satisfies such properties.
Perspective 1:
Perspective 2: Experiments show mid-representations of neural networks have certain properties > such redreeerrennnttion can be viewed as Data Manifold of Intrinsic Space that is defined to have such properties.
These two perspectives are equivalent to each other, and Perspective used in some previous work (Bahri et al., 2024 Sharma & Kaplan 2022).
In this section, we formally define the Intrinsic Space and formally derive related results, following Perspective 2.
D.1 FORMAL. DEFINITIONS INTRINSIC SPACE
We define an intrinsic space to formalize the latent structure underlying natural language sequences. This space is independent of surface forms and aims to capture the semantic and syntactic essence of language contexts across different sequence lengths.
Setup. Let be a finite vocabulary X' the set of all finite sequences over V. Let X denote the original data manifold of natural language, i.e., the support of the data distribution p(r.).
Definition. An intrinsic space is latent representation space defined by a mapping
T
X'e + 2,
where Xx: = Ulzo D is the set of all language contexts of length t. The image of the original data manifold under this map is denoted M3 = o(Mg)) C. Z, or the data manifold (in Intrinsic Space). We require the following properties:
21

---

Equivalently, in the idealized zero- Published as a conference paper at ICLR 2026
Predictive Consistency; There exists decoder 2 A(V) such that
a <++(x==)) p(2; set))
i.e., the intrinsic representation enables accurate next-token prediction.
Moreover, there are some other properties assumed (separately) in our work, for which we give formal definition here.
Uniform Information Gain: This assumption assumes the following linear relationship between predictive divergence and intrinsic dimension dim(!):
(R Pi) (dim((o) dim(())
(10)
for some constant > O, which we interpret as the average number of bits of predictive information contributed by each intrinsic dimension. This empirically observed in ex- periments.
Linear Entropy Relationship This assumption assumes that there exists constant < 8 < 1, b and sequence of tolerances {Er)e20 with Et such that for every context length t,
H [9)) + H Hpp xx))
Et.
(11)
Where (+(-) denotes probability density function. tolerance limit:
-s H [q(()] + b= H [p(ra [<)]]]]]]]]]]]]
At.
(12)
It is worth mentioning that, we can easily derive the linear entropy relationship from the uniform information gain assumption, but not vice versa. Hence, linear entropy relationship is weaker assumption compared to uniform information gain.
Lipschitz Differentiable Density This assumption assumes the density of data distribution is smooth in the intrinsic space:
Vg(=)] s L
(13)
for some constant L > 0
Finite c-negative moment: This assumption means the integral of the c-negative moment the data distribution is finite:
J ((()) dz = C. < 0o.
(14)
Remark. When Z is bounded (ʃz q(2) dz Vz oo), and if there exists a constant Gmin 0 s.t. q(=) 2 Jmin > then this assumption is satisfied. Hence this is weaker assumption compared to boundedness and non-zero density, which is even weaker than uniform distribution assumption.
To conclude: if some space satisfies these properties, then the data representation is referred as 'Data Manifold' in this space, and such properties would lead to further derivations in these work (including this work). In experiments, Middle represetrptennntaiionn of neural networks are assumed (and shown) to have these kind of properties, hence explain some of the scaling behaviors they have.
D.2 DERIVATION FOR DATA SCALING FOR APPROXIMATION LOsS
Theorem (Expected capped nearest-neighhour distance), Let 2 C RRd (d 2 1) and there exists non-empty open set U E Rd such that U C Z (i.e., a d-dimensional region). Let q > (0) be a probability density satisfying
1. Lipschitz Differentiable:
[V((=)]| & for all 2 E Z;
2. Finite c-negative moment: for some fixed > 1/d, / q(2) d(z)~-- dz:= = =Ca <0.
22

---

Published as a conference paper at ICLR 2026
Key Properties Assumed
Derived Results
Bahri et al.
2024 Thcorem
Bounded, Uniform Distribution, Lipschitz Differentiable
Data Scaling for Approx. Loss
Thcorem
in Appendix
D.2
(Bounded,) Finite Negative Moment, Lipschitz Differentiable
Data Scaling for Approx. Loss
(cort. 10 Section 2.3
Theorem in Appendix
D.3
Predictive Consistency, Uniform Information Gain
Bayes Risk for Nip of varied (Intrinsic Dimension perspective)
(corr. to Scction 2.2.1
Theorem in Appendix D.4
Predictive Consistency,
Bayes Risk for Ntp of varied Ctl
(corr. to Section 2.2.
Linear Entropy Relationship
(Information Entropy perspective)
Table 2: In this Section (Appendix D we formulate results from previous sections with The- orems derived with defined assumptions and properties intrinsic space this section. Nip refers to dext-token-predictin tion loss with weaker assumptions compared (Bahri al. 2024), please refer Theorem Appendix D.2 for more details.
Ctl refers to Context Length. We derive data scaling for approxima-
Draw i..dd samples Zp = (Zi....,Zz) ~ n and define the capped nearest nearest-neighhour distance distane
RM (Z:) = minf M, min Z - Z J
M > 0.
(15)
Then, there exists constant Do:
C(d, L, e, M, C.) and Do Dold, L, c, M, C.) such that VD >
d+1
C (log D) D- (=((11)) ife d
Ez (Z))] $
(16)
d+1
CD-1/d
ife 2
d
Thus, there exists constant e c(L, C, M, Cc). such that:
Ez, (MM s C D-
(17)
Proof. We write Zi for the distinguished point and R(Z)) min 1 Zμ for its exact near- est-neighbour distance, always capping by M at the very end. Throughout the proof the expectation [||] is taken over the whole sample ZD ,Z1....,24 LD) DD
Step Notation.
F5 :o: () (0))
Ca
Vd
ro(2)
q(2)
2
2L
va the volume of unit ball. Moreover, since ||V < L, whenever ro(2) one has g(u) 2
2 (2) for every u E (2).
Step 1. Exponential hole probability inside the Lipschitz ball.
Fix and r S ro(z). The mass inside B, (x) satisfies
(+(=) / Bx(z)
1 2((2) vard cag(z)rd.
q(u) du 2
Conditioned on Z1 = 2, the (D- - 1) other points are i..dd q, so the conditional probability that all other points sampled outside the ball (2) is:
PT(((2) > r 2i 2) = (x)) D-1 < expl-co(D - 1)9(2))
(18)
Step 2. Density threshold and spatial split.
23

---

Published as a conference paper at ICLR 2026
Define data dependent threshold 2d+ Lnd D
a
Ap (-
(D2
Ho (2:9(2) 2 dD)
=o: Rd HD.
CaD
(The power 1/(d + is tuned to balance two error terms below.)
Step 3.1. Distribution in Hp (moderate or high density region).
For every E HDp set
plz, = 2d log 1/d caD q(z)
(._Dg(z)
Bound on p. Since g(z) 2 AD. (z, S ro(z) and therefore equation 18 is valid for all 0 < r S p(z, D). Tail probability at With r = p(z, D),
Pr(R(z) p(2, D) Z1 =) 2) 1 expl-2dlop D D-2d
Because (M( min(n(22 Z1), M) M.
(Z)) 11= = 2| $ MD 24
(19)
Integral of R up Using equation 18
B[R(Z)) A p(z, D) Zi = / Pr(R > r r = 2) dr
< / expl-ca(DD ca(D - 1)1(z)) dr.
Make the change of variable t ca(D - 1)g(2)r"; then (t/calD 1)g(z)) and dr
r dt/t. The upper limit r = maps to f = 2dloo D. Hence
! expl-ca(D- - 1)g(z)r= dr ['(1 + 1/d) (Dq(z)) 1/d
d'acc 1/d
Absorbing constants:
E E[R(Z)) A p(z, D) 21 = < Ca.L (Dq(z))-11 1/d
(20)
Average over HD. Taking expectation equation 20 with equation 19
Zı first restricted to HD and then combining
E [Z) (Z1) s CCD-1// + MD-24 24 C1 = CaL 1/d < co.
(21)
Step 3.2. Distríbution in Ln (ultra-low density region).
On LD one has 9(2)* & XD. so by Hölder's inequality
Pr(Z e Cp) = / g(z) dz & AD / dz
Since RM S M.
(ZRu(Z))cn)) (Z)) & MC = MC. D) D- D-e/(d+1),
(22)
Step Global bound.
Adding equation 21 and equation 22 For large enough D, the term MD-20 2d is higher-order small quantity compared to - or D (d+1), Therefore,
E RM (Zi) < 1/D11/4 MC. (log D (((((+1)) ommin 1D/ B. ((d+tt))
Finally, compare the two powers of D. e < d+1 then e/(d 1) < 1/d and the second term dominates; otherwise the first dominates. This yields the two-case estimate claimed.
&
24

---

Published as a conference paper at ICLR 2026
How large can be, for unbounded and bounded 2?
is usually assumed (Bahri et al. 2024; Sharma & Kaplan, 2022) that bounded: this assumption makes sense since in usual cases we approximate Intrinsic Space with mid- dle feature representation of neural networks, which can indeed be bounded. For bounded 2, it is possible for e to be larger than + 1/d. We would like mention that in (Bahri eet al., (2044), a constant distribution g(z) = Const is assumed, where e can be arhi- trarily large and dominant rate D D-1/d derived: this a much stronger assumption compared to Finite e-negative moment assumed in our work.
If Z is bounded and 3mmi 0 such that Vz, q()) 2 Qmin > 0, then Vz > 1, [z (2(22)=dd> dz I [g Umsin dz I-e Jg dz, and the final term is finite for any That is, in this case can be arbitrarily large.
For unbounded 2,
is the usual case; = 1 the condition becomes f Leb(supp < oc, i.e. compact support of finite measure. For most unbounded densi- ties (Gaussians, sub-exponential, power-law) one only has 1.
The comparison threshold d+1 is always when d 2 1; hence the dominant rate is d
D D-</(d+t1)
for every admissible 1/d < <<< 1, only if the support is compact and 1 + 1/d.
(23)
D-1/a 1/d
unless q is essentially bounded below Thus can never "reach" the critical value d its support.
From Nearest-Neighbour Distance to Approximation Loss
Capped nearest- neighbu distance can be derived natually if one assume the maximum distance of neighboring points to be bounded by constant, or if one assume the Intrinsic Space 2 is bounded.
Restate of Theorem 2 in (Bahri et al., 2024):
Assuming I(f). be Lipschitz with
contants KG, Ky, Kr and ((F) = O, D) be training dataset of size D) sampled i.i.d from D. Then, for each training point x, let å be the nearest 2|].
Ma- Let f(x) = (V
neighboring training data point, we have L(D)= < ( + FF)
Combining Theorem 2 in (Bahri et al.) 2024) and previous results (nearest neighbour distance in this Appendix D.2 since Approximation loss of context length / is D (2 Q) which can be 0 when Q = Pt. thus satisfying the assumption of Theorem 2 in (Bahri et al.,
2024 Thus, Approzx = Cu + A(1)/ Do/dim = Co- + A(1)
Therefore, have:
Theorem 2 (Data Scaling for Approximation Loss). Let 2 C Rd (d > 1) be a d-dimensional region (exists non-empty open set U E Rd such that U C Z). q Z [0, oo) be probability density function satisfying Lipschitz Differentiable and Finite e-negative moment. Let g Z + Pv be decoding mapping from intrinsic space
distribution of tokens in vocabulary V, and 1(P11, be KL divergence loss function (thus I is zero for identical distributions). Assume ((g(21).((22)) differentiable and Lipschitz smooth for 21 and 22 with Lipschitz coefficient L1.
Then, draw Li.dd samples Zp {Z....., Zn} ~ 8DD if min | Iz; Z,1l is bounded by M, then there exists constant C(d, L,c, M, Ce, Lı) and e cL,, c, M, C.) such that for D > Dold, L,C, M. M.C.):
min (in((((Z)).)(()))) 3#1
(24)
Proof.
min minl(g(Z).gg(,)) < min La 11 Zyバl jfz
3#1
L min [M, min jti
||z 2,1l)
(25)
Li RM (Z:) by the definition of RM (2:) in Theorem
]
25

---

Published as a conference paper at ICLR 2026
By applying Theorem 1 we have:
Ezn Imi mmnn((((2)) C.C o/d
(26)
37
for constant C C(d, L, e, M, c= c(,, M, C.) and large enough > Dold, L, e, M, C.). thus completing the proof.
Meaning of finite e-negative moment
Lebesque-measure view
Write Fiy == {= (()) s 1). Chebyshev gives
Leb(F.) S1 g-= = C
(27)
Hence Assumption (A2) controls how large the very-low- region can be; the smaller the larger that region may grow.
Rényi entropy view
For order a > 0, the Rényi entropy is
1 log qe.
(a
(28)
Setting a - E (0, 1) (the Tsallis regime) and re-arranging,
f --- -e
(29)
finiteness of the e-negative moment is equivalent to finite sub-Rényi entropy of order 1. Smaller (order closer to 1) corresponds to heavier low-density tails, which precisely slows the nearest-neighbour rate as captured in Theorem
D.3 DERIVATION FOR BAYES RISK WITH INTRINSIC DIMENSION ASSUMPTION
Theorem 3 (Bayes Risk and Context Length with Intrinsic Dimension Assumption). Let Z be an intrinsic space satisfying Predictive Consistency and Uniform Information Gain, then the Bayes Risk (PP, Pi) of context length is Linear with respect to Intrinsic Dimension dim(l). That is,
H(P, Pt) - dim(() + Const
(30)
Proof.
H(P,P)) = H(P) + (P,(P,P))
H(P) + s (dim(co) - dim(())
(31)
= s dim(!) + Const
An intuitive example for the 's-bits per dimension' assumption: assuming that the vocab is an integer from to 2dim(co)=s 1. assuming P(zo|==>c:0) = 8 that is, the next token given 2-oo:00 is sure to be y. y. For P((zolx-oo:n). the first dim(l) digits of the integer (in binary representation) are known, but the remaining (dim(oo) - dim(()) * digits are un- Thus, D...... (P(.o (P( (z -c.oo)) = 1 log (012/ (dimm (dddmm())))d((())))))))) = s (dim(oo)
known, making a guess in these numbers yield P1 = (12=)((m(())dmmmm))
dim()))-
26

---

Published as a conference paper at ICLR 2026
D.4 DERIVATION FOR BAYES RISK WITH INFORMATION ENTROPY ASSUMPTION
Theorem 4 (Bayes Risk and Context Length with Information Entropy Assumption). Let 2 be an intrinsic space satisfying Predictive Consistency and Linear Entropy Relationship zero- tolerance limit, then the Bayes Risk H(P Pt) of context length is Linear with respect to Intrinsic Entropy H [g [2)] where q-)-) denotes probability density function. That is, H(P,P) -8 (( Const
(32)
Proof.
H(P, Pi) = (P)) (from Appendix B.2
Hpp(sa12 < ())
(33)
8 H (Z)) +
H ( + Const
E MORE EXPERIMENTS OF LLAMA ON ANOTHER DATASET
According to the technical report of LLaMa 3.1 (Grattafiori et al. 2024), the text corpora with num- ber of ddirty words' beyond certain threshold would be filtered out, as proposed (Raffel et al. 2023 We collect some text corpora online which include forbidden words defined in (Raffel et al., 2023 as text corpora unseen by LLaMa 3.1. By conducting experiments it we obtain results similar to Openwebtext subset.
LLaMA Cross Entropy
Context Length
collected dataset, fitted with: y=C_0+C/+" D+Clsty
3.75
88 (data)
708 (data)
88 fit (Cam2.029, C=4.574, y=0.500, R.-0.99973)
70B [Co-1.919, C-5 C-5.256, y-0.546 R=00.99952)
3.25
Cross Entropy Loss
2.75
2.50
2.25
2,00
10
102
10º
Figure 10: Cross Entropy Loss vs. Context Length, with log scale. We see that y Co + C/2" fits this curve well.
According to Figure 10 sure not to be seen by the model.
that CE Co C/r² approximates well for text corpora that
OPTIMAL CONTEXT LENGTH FOR TWO-NEEDLEENN-HAASSIACC TRAINING: STUDY ON SYNTHETIC DATASET
Here, we utilize our proposed synthetic dataset as a proxy to study the coo-neeleein--haysteee ex- periment (as mentioned in Figure 5)
27

---

Published as a conference paper at ICLR 2026
Derpen dey M 808
dor 303/10
bease: can 2,0000
Centell
Figure 11: Validation set Cross Entropy Loss of the output token (resembling the 'answer' token of two-needle-in coo-needle-in-haystaee hn-haystack tasks) vs. context length, for MLPs trained with different training dataset sizes.
As mentioned in previous studies, the Cross Entropy loss of key tokens (e.g. the perplexity of the Answer token for Needles in Haystacks (NIH) is highly correlated with downstream task accuracy (that is, the NIH tasks). Here, we use the Cross Entropy loss of the output bit of our synthetic task (shown in Figure 5) as metric for our synthetic
task.
Here, we use synthetic dataset similar to that mentioned in the main paper, except that it has more than 500 context bits (though most tasks require only first 100 context bits). In this section, fix the size of the training data, train multiple iterations till overfitting, and take the best validation loss the validation loss of that (training dataset size, context length) pair. Results shown in Figure 11 From the result, we make such observation:
There exists optimal context length for most training dataset size used, and such optimal context length increases with the amount of available training data.
This proves the concept that, when training dataset is limited, optimal context length smaller than the task length could potentially exists for tasks resembling the two-needle-in in-haystacks tasks, and larger training dataset leads to larger optimal context length.
G EXPERIMENTS ON OTHER LANGUAGE MODELS
In our main paper (Figure 2. we present the relationship between Intrinsic Entropy and Cross Entropy Loss across multiple Language Models: Llama-3.1-8B, Qwen3-8B-Base and RecurrentGemma-99 on OpenWeeTexx.
For Qwen3-88 Baann3-8B-Base Base, the linear relationship between Intrinsic Entropy and Cross Entropy loss holds quite well. For RecurrenGGemma-99, we observe that its Cross Entropy loss significantly higher than Llama-3.1-8D and Qwen3-8B-Base for small context length (the 3 high points drawn on the figure), while other points show similar cross entropy loss. Therefore, we conclude that RecurrentGemma-99 is not good approximation for Bayes Model for these outlier points (i.e. can't model low-context quite well with Cross Entropy loss > 5, potentially because of its architecture training pipeline), and Qwen3-8B-Ban -Bse as Bayes Model for regression.
the points where it is closer Llama-3.1 3.1---3.11-BB and
Experiment in this section proves that, (1) our proposed Intrinsic Entropy and Cross Entropy loss relationship holds across different series of Language Models with different architectures when they well-trained and can represent Bayes Models; and (2) the discovered relationship only holds when the measured Language Model approximates Bayes Model well.
H BIGENVALUE-BBSEED INTRINSIC ENTROPY ESTIMATION
In the main paper, Gaussian-KDD estimate Intrinsic Entropy. Here present al- ternative eigenvalue- based estimation method, and show that both methods yield consistent linear relationships between Intrinsic Entropy and Cross Entropy loss.
28

---

Published as a conference paper at ICLR 2026
H.1 BAYES RISK vs. CONTEXT LENGTH ON OPENWEBTEXT
We use well-trained Large Language Models to approximate the Bayes Risk (P) on OpenWeb- Text. As shown in Figure [2 the relationship H (P..)) Co + C/17 (Equation approximates the experimented behavior on OpenWebText well.
VEAL oparivccii cemins F04d Por-e
Figure 12: Bayes Risk vs. Context Length: Bayes Risk is approximated by Cross Entropy loss measured with LLaMa-3.1 series on OpenWebTexx for different context length.
H.2 EIGENVALUEE-RASED ENTROPY MEASUREMENT
To establish a relationship between Cross Entropy and Intrinsic Space, we run LLL.Ma-3-hh on a subset of the Openwebtext dataset and obtain the feature of the last token the feature representa- tion, or Intrinsic Space of the approximated Bayes Model. For certain context length, we gather the feature representation of multiple (2 10000) samples, and conduct PCA analysis on these samples to obtain eigenvalues for the specific context length, results are presented in Figure [4] We see that the model with larger context length tends have larger relative eigenvalues in intrinsic space, thus containing more information.
According Statistical Physics, entropy of a system be defined as 5 5 log Ω where Ω is the possible number of states of the system (Landau & Lifshitz 1980). Similarly, we use the sum of logarithm of eigen values as proxy for measuring Information Entropy
log
log V ///ding (V) where V is the volume in intrinsic space
Σ log releeigaal/h
log logrel_iggval + Const
Here h is the 'plank constant', meaning that one state corresponds to unit hyper- of d in the Intrinsic Space. different value of h would only add a constant to and would not affect change in Entropy. Thus, use Σ, rel eiguall as Entropy in Intrinsic Space.
H.3 CE LOSS VS. EIGENVALUE- BASED INTRINSIC ENTROPY
Experiments show that, no matter what subspace we use, the Cross Entropy Loss usually follows linear relationship with the Entropy we measured in the subspace, supporting the claim that the next token prediction task likely lies in some subspace of the Intrinsic Space, or (statistically) its Entropy should be some weighted average of Entropy of several subspaces of similar dimen- sion.. This also suggests that Hatp is approximately linear with His, which validates our previous assumptions and claims.
Similar estimation can also he derived from the assumption of Gaussian differential entropy with homoge- ncous reference measure.
29

---

Published as a conference paper at ICLR 2026
Eigewoluess Mid
Relative
Context
327
301
10p
26:11
3071
1
4828020
30
1280087401
1575
Fode of Elppraaaa
Inder Rigenca
Figure 13: Left: Relative Eigen Value Measured for the last token, for LLaMa-3.1 3-BB on a subset of OpenWebText. Right: relative increment of relative eigenvalues (for different context lengths measured). We can see that the relative eigenvalues approximately increase at a same scale.
Entrpy fintopy
an eakee
Entmpy Messared Intrrdd Eittopy
<0,m
(pari
138
188
180
Figure 14: Upper-left, Upper-right, Bottom-left: Cross Entropy Loss measured Intrinsic En- tropy with N first Eigen Values: Eien log rel_eig-val; Bottom-right: correlation between minus CE loss and Eier log rel eig val. All experiments are for LLaMa-3.1-8B on a subset of Open- WebText. From the first three figure, we see CE loss is linear with the Entropy of certain subspaces. From the bottom-right figure, we see that Entropy measured in different subspaces are highly corre- lated (corr > 0.97), which are also highly correlated with the CE loss for Next Token Prediction.
We observe a fairly linear relationship between CE Loss and Entropy measured (supporting our theory), validating our theoretical assumptions:
RBaees RS = S(P((P) + Const,
which aligns well with Equation 2 thus validating our entropy-based deduction.
30

---

Published as a conference paper at ICLR 2026
Length, Web
29
300M tokend
350M tokers
500M tokeno
38
730M tokons
37
2000 4000
6009
5000 10000
Concext Longth
Figure 15: Openwebtext subset, Validation Loss vs. Context Length, for different dataset sizes. Different curves represent different amount of training data used. more readable figure can be found in Figure 1 where the minimum validation loss reachable for each training dataset size is subtracted.
I EXPERIMENT SETTINGS
1.1 NATURAL LANGUAGE DATA
1.1.1 OPTIMAT. CONTEXT LENGTH EXPERIMENTS
As described Section we use nanogpt (Karpthy 2022 and train a model with GPT-2 (Rad- ford et al. 2019) architecture AdamW (Loshchilov Hutter 2019) optimizer, learning rate of vee-4, 4, weight decay of 1e-1, 1000 warm-up iterations. For given token number, all models with different context length are trained with same number of iterations, where iteration number equals roughly to token token_ wwwberrreeeeeeeeee (0.1M).
subset of Open web'Text dataset. For training, we use the
We first select text corpora with context length beyond specific limits larger than the maximum training context length from WebTet, then split into Training set and Validation Set. The validation set has 134M tokens.
Experiments presented in Figure ! and Figure 15 took around 300 gpu hours on 8 AMD MI-250X GPUs (which are similar in performance to Nvidia A100 gpus).
1.1.2 INTRINSIC DIMENSION EXPERIMENTS
We select long enough text corpora from the Openwehtext dataset. Then, following previous prac- tice (Cheng et al. 2023), we conduct experiments with LLaMa-3.1-8 1-bb on 10000 samples of this subset. We extract the feature representation of the last token in the last layer, the Intrinsic Rep- resentation of samples.
Conducting all intrinsic dimension measurements cost up to around 100 gpu hours for MI-250X gpus.
RELATED WORK
J.1 ENLARGING CONTEXT LENGTH FOR LMS
Previous work has made attempts to enlarge the context length of Language Models. Work repre- sented by RoPE (uu et al. 2023) rotary positional embedding support generalizing L.Ms to
31

---

Published as a conference paper at ICLR 2026
longer context in inference compared the training process. These work use modified positional embeddings to model the relative position dependency in attention mechanism.
There is also work about enhancing long context understanding and exploring Scaling Laws for context length (Xiong et al.] 2024). These work utilize an adjusted pretraining and instruction- finetuning process with more long-context data to enhance the models' ability on long contexts.
Other work modifying architectures has also been proposed to enhance long context modeling or to simplify deployment of long context LLMs. For example, (Tang et al. (024) proposes training- free RazorAttention algorithm to largely compress the KV cache while maintaining performance unchanged.
Architectures and inference methods have been proposed to reduce inference time and memory cost for Language Models, represented by a series of linear transformer RNN-based meth- ods (Katharopoulos et al. [2000] Gu & Dao 2024 Sun et al. (0024). These methods, largely reducing the computational cost and memory usage for long input contexts, have displayed a margin ahead of traditional attention-based (Lan et al. 2025 Li et al. 2025 Cai et al. 2025) language models for long context inference.
Currently common practice to train very large Language Models supporting long context is to use pretrain the model with shorter contexts, then finetune them with longer contexts, as presented in tech reports of LLaMa-3 (Grattafiori et al.) 2024) and DeepSeek-v3 (DeepSeek-Al et al. 2024).
J.2 IRRELEVANT LONG CONTEXT HURTS PERFORMANCE OF LMs
Besides context length scaling with relevant contexts, previous research have studied how LLMs perform for long irrelevant contexts. As an example, (Levy al. 2024 studies the performance of current LLMs on an adjusted version of 'needle in a haystack task, where two pieces of key information are embedded into long text corpora and a question related to both asked, similar to that presented in Figure [ The conclusion of these work is that LLMs would perform worse when there is too much irrelevant information.
J.3 LONG CONTEXT IN ANOTHER FIELD: TIME SERIES FORECASTING
Context length, representing the length of input context, is not unique Nature Language. For time series forecasting, where machine learning play an important row, there is also work discussing the impact of context length, represented by (Shi et al. 2024). These investigations find that there exists an optimal look-back horizon, which increases with dataset size. However, time series datasets are relatively small compared to NLP datasets, and thus whether this conclusion holds on NL. remains an open problem for this work to study.
J.4 RELATED THEORIES FOR SCALING LAWS
Since the discovery of Scaling Laws for Large Language Models (Kaplan et al. 2020) or even carlier, there has been theoretical work trying explain why model performance could benefit from more data points and more model parameters. For example, (Sharma & Kaplan 2022) studies the dataset and model scaling from the data manifold perspective.
Specially for Language Models, there is also previous work proposing all kinds of theoretical mod- els. For example, (Michaud et al.) (224)) proposes feature quant based theory: (Aghajanyan al. 2020) views the effect of tine-tuning from the intrinsic dimension perspective; (Havrilla & Liao proposes to understand scaling with intrinsic dimensions.
2024
K INTRINSIC DIMENSION PERSPECTIVE: MEASUREMENTS IN INTRINSIC SPACE
K.1
BAYES RISK FROM AN INTRINSIC DIMENSION PERSPECTIVE: ASSUMPTIONS
Here we derive similar results as in Section 2.2 but from an Intrinsic Dimension perspective rather than Information Entropy perspective.
32

---

Published as a conference paper at ICLR 2026
We propose simple theory model to relate ((P.)) Pt) with the intrinsic dimension dim(U) of the intrinsic space space, of the text corpora length / (for the next-token prediction task).
We assume these assumptions hold for Intrinsic Space (please see formal definitions of Intrinsic Space in Appendix D).
Assumption 1. Intrinsic Dimension of the Bayes Model lim dim(!) dom is finite, which is the Intrinsic Dimension of next token prediction of language itself.
Assumption 2. 14, la such that I l2, dim(l1) < dim(l2). This is because longer context contains more information about the next possible token.
To simplify deduction, we further assume that,
Assumption 3. Uniform Information Gain (s-bits for next token prediction per Intrin- sie Dimension): Each intrinsic dimension would add s bits of information to the next- token prediction task, there are dim(!) bits of information that can be represented in space, for the next-token prediction. This means the KL-divergence for the Bayes Model of context length 1, Pt, with Bayes Model of infinite context length, P = Pas. is: (Rr Pi) * (dimm(x) dim(()). Note this does not mean these are the only information in the Intrinsic Space, hence be small,
smaller than 1.
With these assumptions, we can derive H(P, P.) with dim(
Bayes = H(P, Pt) - dim(() + Const
(34)
This linear relationship can be observed in experiments for LMs and synthetic data, providing an alternative explanation the entropy-based approach in the main paper.
K.2 EXPERIMENTALLY MEASURE INTRINSIC DIMENSION USING PCA
We further use PCA as metric to measure the Intrinsic Dimension of Dataset with respect to context length. We provide the relative degradation of the eigenvalue the feature space of LLaMa-3.1- 8B, for the last token. We see that larger input length would indeed provide feature with lower degradation in the intrinsic space. Notably, when < idr 1500 the curves is similar Zip-f distribution (log eig = Co C * log idx), and for 500 < idr < 4000 it resembles exponential idx).
degradation (log eig = Co
Instead, following previous practice, here we use some threshold to decide the transformation index of these two states as Intrinsic Dimension: max;az rela_eig(izz) Z threshold used as the mea- sured Intrinsic Dimension. Notably, the threshold here is hyperparameter which is set constants in previous work(e.g. work(e.g.1/20 1.2020 in (Aghajanyan et al. 2020 but we observe here that many thresholds would validate the linear correspondence of Cross Entropy Intrinsic Dimension, which further enhance the robustness of our result. We use thresholds from 0.002 to 0.25.
For certain threshold, conduct experiments several context lengths, and measure CE Loss on certain text corpora with these context lengths. We observe a fairly linear relationship between CE Loss and ID measured (supporting our theory), shown in Figure 16 We that, matter what threshold we use, the Cross Entropy Loss usually follows linear re relationship with the Intrinsic Dimension we measured, showing the robustness of the PCA evaluation method, and validating our theoretical assumptions:
Rayes ~ - dim(!) + Const,
which aligns well with Equation 34 thus validating our intrinsic dimension-ban based deduction.
K.3 MLP-BASED SYNTHETIC DATASET: INTRINSIC DIMENSION EXPERIMENTS
We train a large enough MLP on data generated on the synthetic tasks, and evaluate our model on the validation dataset. We train until overfitting the training dataset. We assume dimension in
33

---

Figure 16: Left figures: Relative Eigen Value for 1LaMa-3.1-8B on a subset of Web presented in different x-axis scales, with different context length visible to Language Model. Gray lines represent different thresholds we take to measure the intrinsic dimension of the current model. Right figures: Cross Entropy Loss vs. Measured Intrinsic Dimension. Each line represents certain threshold used to measure ID in the intrinsic space of the used L.M.. Different Measurements Published as a conference paper at ICLR 2026
Enrrpe
Hert
Tear 2)
Yotee Traed thod 1o0
den
3.75
12303
Neauard Dimsrali Thrasol Enropy Maaures Dimarake Trrshld
Thes dcta and Not
con fod out
Shaae cook (ni (ec)
66.3
$1110
1.2
200 360 400
30/
would give IID values that are linear w.r.t. each other, and they are all linear w.r.t. CE loss.
Intrinsic Space can store information about subtask, hence we take ID(I)= = ((1) as its theoretical value here.
Let f(x, C, Co. 7) Co - C/2" and g(r,k,)) = k
The fitted results are:
+.7 + 0.3 10", 7 = 1.18 + 0.06, ID & CL: ID & (ICL,C,Co,7).C. = 51.1 = 1.0, C
R² = 0.9997.
.
CE & CL: CE & R² 0.9997.
0..15 + 0.013, C = 23.8 + 13, 7 1.18 + 0.06,
CE ID: ID:CE (ID,k, k= 0.693 + 1* 10 - b 0.0139 + 10 R² = -
As shown, construct synthetic data such that ID( ID, - /T", and our measurements show CE C+C More importantly, for the synthetic data example, Cross Entropy loss is almost perfectly linear with the Intrinsic Dimension as we defined previously. This validates the linear relationship between Cross Entropy Loss and Intrinsic Dimension; and we have also provided a construction match the measured relationship CE x Co + C/D7
pred logit
100101101..011
MLP
control bits
0010..0
context bits
MLP
Context Feature for PCA analysis
Figure 17: Model trained on the proposed synthetic dataset; @ represents feature concatenation. Only the first bits used as input to context MLP when the context length is set to (. We conduct PCA on Context Feature to analyze the intrinsic dimension of input context bits for various context lengths.
34

---

Published as a conference paper at ICLR 2026
We train model with specialized architecture, allowing the feature representation of
a middle layer as feature vector for input context bits, as shown Figure 17 After training the model data with different context length, conduct PCA the obtain context feature representation to study the Intrinsic Space of this model.
Eigenvalues Nid-festures
Expervatio Mae Eigerrceeee
10
20
30
30
Inddeas
Figure 18: Relative eigen value vs. index, for models trained on different context length. Vertical lines: 21 = [I)) and x ID(L 1). For example, context length 27 has 16 subtasks visible, corresponding to 16 bits in Intrinsic Space. Assuming 1 dimension in Intrinsic Space represents bit, the leftmost purple rectangle drawn means a range of Ythreshold that would provide an accurate estimation of (27) 16 for context length 27. There exists y that would provide estimation of ID for all context lengths, as shown the figure.
We find that: (1) the neural network would indeed learn the key information in the context bits. For models with different input context lengths, although their inner dimensions are the same (80), the representation of inputs in this inner space mainly lies in the first ID dimensions, and the eigen values corresponding to other dimensions are very small; and (2) there exists such threshold y that would estimate I ID for all context lengths accurately. We can take some threshold y" to estimate the intrinsic dimension, by obtaining the maximum index of the relative eigen value such that the relative eigen value is larger than y", which would give accurate and consistent estimates.
K.4 BRIDGING THE GAP BETWEEN INTRINSIC DIMENSION EXPLANATION AND INTRINSIC ENTROPY EXPLANATION
Here, starting from previous assumptions and measurements w.r.t. Entropy in Intrinsic Space, we explain why CE is linear w.r.t. Intrinsic Dimension measured in Section 2.2 for idr 500. We in Figure 13 that for idr > 500, the relative eigenvalues mainly follow an exponential decay:
releigval, idæ releigoal,, exp{-ac idr). for certain context length
where is the context length, idæ is the index of some certain eigen value, and ay is the exponential decay coefficient for this certain context length 1.
We also see from the previous results (Figure 13) that for different context lengths, the relative eigenvalues increase almost in the same proportion, especially for idr > 1000. That is, og We define 7() = releigraly, releiggal and thus we have: releigvali ids religval 71) exp(-a idr).
For the subspace for the next token prediction task, we denote its dimension to be m. Hence, the entropy should be proportional log of volume in the subspace; that
Soubspace (1)
Σ
log s0 exp(-a idx)
id2E (dimension subspace)
(35)
m log (1) + Const
which is the result of the Intrinsic Entropy Explanation.
35

---

Published as a conference paper at ICLR 2026
For Intrinsic Dimension explanation, if we are using certain threshold thres to measure Intrinse Dimension, the measured dimension dim(l, thres) should satisfy:
releigvalae,0 7(1) = exp( * dim((, thrrs)] thres,
hence the measured dimension is dim(l, thres) 1/a (1) + of Plugging this into Equation (35 we have:
LOF - suhspace (1) + Const mma dim(l, Uhrre) + Cons((thres).
(36)
Thus, we derive our assumptions in Section where = ma. Equation (36) can also be validated in the lower-right part of Figure [6] where the Intrinsic Dimensions (for idæ 2 500) are measured in the exponential decay area, and these lines, though measured with different threshold (thres), share similar slopes w.r.t. CE Loss (that is not related with threshold, as shown in Equation 36
L DISCLOSURE OF LLM USAGE
LLMs are used in this work for polishing writing only.
36


Reviews:
[{"review_id": "maS0VxHiRu", "reviewer": "Reviewer_8bxz", "summary": "This paper investigates the impact of context length on Language Model (LM) performance through a novel theoretical lens termed \"Intrinsic Entropy.\" The authors posit that the cross-entropy loss can be decomposed into Bayes Risk (the irreducible error of an optimal model) and Approximation Loss (the error from the trained model failing to match the optimal model). The core contribution is the proposal that the Bayes Risk is linearly related to the Intrinsic Entropy of the data manifold, which itself is a function of context length. They further argue that while longer contexts reduce Bayes Risk by providing more information, they simultaneously increase the intrinsic dimension, making the Approximation Loss harder to minimize, especially with limited data. This trade-off leads to an \"optimal context length\" for a given training dataset size, beyond which validation loss increases. The paper supports its claims with experiments on both natural language (OpenWebText with LLaMa and GPT-2) and a carefully constructed synthetic dataset.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. While the decomposition of loss into Bayes Risk and Approximation Error is standard, and the use of an \"intrinsic space\" is inspired by prior work, the formulation of \"Intrinsic Entropy\" as a central concept to bridge context length and loss is novel. \n2. The idea that there is a fundamental trade-off leading to an optimal context length is a significant and non-obvious insight that challenges the simplistic view that \"more context is always better.\" \n3. The paper is generally well-structured and clear.", "weaknesses": "My main concern is that existing work [1] has pointed out that PPL cannot serve as a standard for evaluating long-text performance. Therefore, experimental results on more diverse long-text benchmarks are necessary, such as RULER, HELMET, and Longbench v2.\n\n[1]Fang L, Wang Y, Liu Z, et al. What is Wrong with Perplexity for Long-context Language Modeling?[J]. arXiv preprint arXiv:2410.23771, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiba7evsiG", "reviewer": "Reviewer_Mnty", "summary": "The paper proposes to investigates into LLM context length from model entropy perspective. The paper leverages cross entropy (and treated as Bayes Risk in the paper, for example in Figure 2 and explained in 2.2.2) and show that it empirically bears a linear relationship with intrisic entropy which is calculated as log of the eigenvalues. Furthermore, by approxmimating the model approximation error in section 2.3, the paper shows theoretically that there exists an optimal context length in section 3 and also demonstrate it empirically.", "questions": "Figure 1 and Figure 7 shows the correlation between cross entropy and N. How are samples obtained for a particular N please? More in detail, how is correlation calculated for a particular (N, cross entropy) pair?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The direction which contructs a decomposition and shows that optimally there exists an optimal context length as shown in section 3 is very interesting. Eq6/7 effectively shows that D would imply an optimal L. There are several empirical results that demonstrate this; besides, the linear relationship between cross entropy and intrinsic entropy seems novel and insightful.", "weaknesses": "I personally think that the mathematical construction lacks rigor. For example, there is no justification of equaling states with volume divided by dimensions. Although eq(3) is understandable, it is not an acceptable exposition since l is discrete. It is also problematic to directly treat crossentropy as Bayes Risk particularly there is no justification provided throughout the paper.\n\nThe paper is badly presented. The figures are not very readable most of the time with non informative titles (e.g. Figure 1, part1/part2/part3) and equations through 381-386. All these make papers more read as a draft instead of a formal conference submission. All the above significantly makes the reading process harder. For example, I am not able to understand the synthetic task presented in Figure 5 right part (with carefully reading its descriptions 372-377)/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HlXm4b76mK", "reviewer": "Reviewer_vDBt", "summary": "The authors propose a new scaling law that takes context length into account. It has the form of\n$$L = C_0 + C/l^\\gamma + A(l)/D^{\\alpha(l)}$$\nWhere l is the context length. This form is a natural generalization of the scaling law suggested by Kaplan et. al. to the case of a variable context length. It captures the trade-off between reduced training sample efficiency and increased generation confidence that LLM experiences as the context length grows.\n\nThe authors study the form of $\\alpha(l)$ and show that it can be represented as $\\alpha(l) = c\\dim(l)$ with $dim(l)$ being the dimensionality of the latent space necessary to represent text of length l. Since larger text naturally may contain more information, the latent dimensionality grows with l, which means that $\\alpha(l)$ decreases. Thus, the suggested scaling law consists of two terms, one of which increases with l while the other one decreases, which implies the existence of an optimal value of l.\n\nThe results summarized here are shown rigorously in the presented work. It is worth noticing that the relation between the coefficient $\\alpha$ of the scaling law and the intrinsic dimension of the data was known in the literature. The authors push it further by leveraging the relationship between the intrinsic dimension and the context length.", "questions": "NA", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper presents an interesting result, consisting of a non-trivial observation and a sound and consistent theory.\n\nThe authors provide a convincing set of evidence that supports their theoretical assumptions.", "weaknesses": "The paper claims that the theoretical framework presented can provide practical insights, such as establishing that the size of the training dataset dictates an optimal context length and bounds context-length scaling. It seems to miss that, in practice, the context length is determined not to minimize the loss function but rather for other considerations, such as the typical context length in the target use cases and the capabilities of the computational hardware. If the prescribed “optimal” context length is larger than the typical context length in applications, the context length will still be kept within the application's demands due to computational constraints. If the prescribed “optimal” context length is shorter than the typical context length in applications, the context length will still be kept up to the application's demand, or the system would not be deployable.\n\nAnother argument against the practicality of the delivered result is that the real objective behind LLM training is not the cross-entropy values, but the evaluation results on downstream metrics. To prove the practicality of the results, the authors would need to conduct experiments that hold compute/memory constant and study task metrics (e.g., RAG accuracy, long-form QA F1, code completion pass@k) against context length", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAWJO47mwf", "reviewer": "Reviewer_ULtS", "summary": "The paper introduces a theoretical and empirical framework for\nunderstanding how context length affects the performance and scaling\nbehavior of large language models. The authors propose the concept of\nIntrinsic Entropy, which quantifies the information a model can encode\nwithin its internal (“intrinsic”) representation space given a\ncertain context length. They decompose cross-entropy loss into two\nparts — the Bayes Risk, representing the irreducible uncertainty from\nlimited context, and the Approximation Loss, representing imperfect\nmodel learning. From this perspective, the Bayes Risk decreases\nmonotonically with longer context since more information is available,\nwhereas the Approximation Loss tends to increase because longer\ncontexts enlarge the model’s effective input dimension and complicate\noptimization. The trade-off between these opposing effects gives rise\nto an optimal context length that minimizes total loss for a given\ndataset size.\n\nEmpirically, experiments using LLaMA-3.1 (8B and 70B) on OpenWebText\nsubsets and a synthetic parity dataset confirm the theory. The\nmeasured relationship between Intrinsic Entropy and Cross-Entropy Loss\nis nearly linear (correlation > 0.97), and the validation loss across\ncontext lengths follows a scaling law of the form H=C0 + C / l^gamma.\n. Moreover, when varying training-data size, the optimal context\nlength shifts upward — indicating that larger datasets justify\nproportionally longer contexts. The findings imply that excessively\nextending context windows can degrade performance when data are\ninsufficient, emphasizing the need for joint optimization of context\nlength, data size, and model capacity. This framework bridges\nstatistical physics–style scaling laws with information-theoretic\nreasoning, offering a principled explanation for when and why long\ncontexts help or harm LLMs.", "questions": "1. The entire theory rests on the existence of a stable Intrinsic Space\nwhere internal representations of data reside and where entropy can be\nmeasured. However, as the authors themselves admit, this space may\ndepend not only on the data but also on the neural network\narchitecture and training dynamics.  Can Intrinsic Entropy be defined independently of specific\narchitectures or layer representations, making it a universal property\nof language modeling itself?\n\n\n2. While the framework explains cross-entropy scaling, it is unclear\nwhether higher or lower Intrinsic Entropy predicts better reasoning,\nretrieval, or long-document comprehension. The authors validate on\nsynthetic and language data but do not link the proposed measure to\nfunctional metrics. Does Intrinsic Entropy correlate with actual task-level performance,\nor does it capture only internal statistical regularities?\n\n3. Can the optimal context-length principle generalize across models,\ndata types, and modalities? The observed law that the optimal context length increases with\ndataset size is based on a limited set of models (LLaMA-3.1) and\ncorpora (OpenWebText, synthetic parity). Whether this scaling\nrelationship extends to multimodal LLMs, retrieval-augmented\narchitectures, or dialogue models remains untested.  Is the “optimal context length” phenomenon universal across model\nfamilies and data modalities, or an artifact of specific transformer\ntraining regimes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel concept — Intrinsic Entropy — that\nlinks information-theoretic measures to model-internal representations\n(“intrinsic space”). This provides a principled bridge between\nentropy, Bayes risk, and approximation loss, enabling a quantitative\nexplanation for the existence of an optimal context length. By\nderiving analytical relationships the work extends traditional\nscaling-law theory (e.g., Kaplan et al. 2020) to the underexplored\ndimension of context length. This is conceptually powerful: it\nreframes the intuition that “longer is better” into a mathematically\ngrounded trade-off between information gain and representational\ncomplexity.\n\n\n\n\n2. Experimentally, the paper validates its theory using LLaMA 3.1 models\nand synthetic data, showing a near-perfect linear correlation (R^2 >\n0.97) between Intrinsic Entropy and Cross-Entropy loss, across varying\ncontext lengths. The authors further identify a data-dependent optimal\ncontext length, which scales predictably with dataset size, and\nreproduce this result across different datasets. This dual\ntheoretical–empirical consistency gives the framework strong\ncredibility and practical relevance: it offers a diagnostic tool for\noptimizing context length in training regimes and provides insight\ninto the physics-like scaling behavior of LLMs.  In short, the paper’s strength lies in its integration of\nfirst-principles reasoning, strong empirical validation, and broad\napplicability — offering both a new theoretical lens and actionable\ninsight into how LLMs handle long contexts.", "weaknesses": "1. The paper’s theory hinges on several idealized assumptions that may\nnot hold in practice: The notion of a stable “Intrinsic Space” — a well-behaved latent\nmanifold shared across models — is assumed rather than\ndemonstrated. In reality, representations in transformer layers are\nnon-stationary and architecture-dependent, making it difficult to\nassert that a single intrinsic space can be defined across context\nlengths. The linearity assumptions — e.g., that Bayes Risk is linearly\nproportional to Intrinsic Entropy or Intrinsic Dimension — simplify\nthe analysis but are only loosely supported empirically. These\nrelationships might hold approximately in narrow regimes but could\nbreak under different architectures, data modalities, or tokenization\nschemes. While mathematically elegant, the framework risks being too abstract\nto describe the heterogeneous behaviors of real LLMs under diverse\nconditions.\n\n\n\n2. The experiments, though carefully designed, rely on a small number of\ndatasets and model families: Only LLaMA 3.1 (8B and 70B) and OpenWebText subsets are tested for\nnatural language, with one synthetic dataset for controlled\nvalidation. There is no downstream evaluation (e.g., reasoning or QA tasks) to\nverify whether the proposed “optimal context length” correlates with\nfunctional performance. The paper also lacks cross-model validation — results are not shown\nfor alternative architectures like Mistral or Gemini-style\nmixture-of-experts, which could challenge the universality of the\nscaling law. The findings, while compelling within the tested settings, remain\ntentative as a general principle across model types and data regimes.\n\n\n\n3. Although the “Intrinsic Entropy” construct is theoretically neat, it\nis not directly measurable without heavy computational overhead (e.g.,\neigenvalue decompositions of hidden representations across\ncontexts). Moreover, its connection to optimization dynamics and\narchitecture-level design remains abstract — the paper suggests\nguidelines for tuning context length but provides no concrete\nalgorithm or scaling heuristic to implement this in real-world\ntraining. The work’s insights are conceptually illuminating but operationally\ndifficult to apply for practitioners designing next-generation\nlong-context LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)