L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling
Authors: Zhuo Chen
Oriol Mayné i Comas
Zhuotao Jin
Di Luo
Marin Soljacic
Venue: NeurIPS 2025 poster ()
Keywords: language models
information theory
mutual information
predictive information
large language models
long context length modeling
long-range dependence
scaling laws
long-context language modeling
sequence modeling
autoregressive models
mutual information estimation
transformers
recurrent neural networks
state-space models
Primary area: deep_learning
TL;DR: Bipartite mutual information in natural text exhibits sub-volume growth; from this, we prove a lower bound on how the history state must scale, setting a necessary condition for architectures to be effective at long-context language modeling.

Abstract: We present a universal theoretical framework for understanding *long-context language modeling* based on a *bipartite* mutual information scaling law that we rigorously verify in natural language. We demonstrate that bipartite mutual information captures multi-token interactions distinct from and scaling independently of conventional two-point mutual information, and show that this provides a more complete characterization of the dependencies needed for accurately modeling long sequences. Leveraging this scaling law, we formulate the **L**ong-context **L**anguage **M**odeling (**L**$^2$**M**) condition, which lower bounds the necessary scaling of a model's history state—the latent variables responsible for storing past information—for effective long-context modeling. We validate the framework and its predictions on transformer and state-space models. Our work provides a principled foundation to understand long-context modeling and to design more efficient architectures with stronger long-context capabilities, with potential applications beyond natural language.

L2M: Mutual Information Scaling Law for Long-Context Language Modeling
Zhuo Chen
Oriol Mayné Comas
Zhuotao Jin
Di L
Marin Soljačié12
NSF Al Institute for Artificial Intelligence and Fundamental Interactions
2 Massachusetts Institute of Technology
3 Polytechnic University of Catalonia
Harvard University
University of California, Los Angeles
ccenzhuo
omayne, jinzhta diluo, soljacic@@mmt.eee
Abstract
We present a universal] theoretical framework for understanding long-contexl language modeling based on bipartile mutual information scaling law that we rigorously verify in natural language. We demonstrate that bipartite mutual infor- mation captures multi-token interactions distinct from and scaling independently of conventional two-point mutual information, and show that this provides more complete characterization the dependencies needed for accurately modeling long sequences. Leveraging this scaling law, we formulate the Long-context Language Modeling (L. ML condition, which lower bounds the necessary scaling of model's s history state the latent variables responsible for storing past information for effective lontecontext modeling. We validate the framework and its predictions on transformer and state-space models. Our work provides principled foundation to understand long-context modeling and to design more efficient architectures with stronger long- long-context capabilities, with potential applications beyond natural language.
Introduction
Large language models (LLMs) have revolutionized natural language processing, achieving remark- able capabilities across a wide range of tasks [1-4. Recent advances in large language models, including ChatGPT 33) Claude, Gemini (6, 7). Grok, LLaMA 48,, DeepSeek [0], and Qwen 12 have achieved breakthroughs across diverse tasks, including code generation, mathematical problem solving, summarization, and creative writing [13166] These models have become increasingly powerful and versatile, pushing the boundaries of what's possible in natural language processing and marking significant steps toward artificial general intelligence (17
In pushing these advances further, the ability to handle long contexts has become increasingly crucial. This ability is the key to document-level understanding, multi-turn dialogue, and complex reasoning. Models like GPT-o1/o3, Claude Opus, Gemini 2.5 pro, and DeepSeek-RR often generate extensive chains of thought, spanning tens of thousands of tokens to solve complex problems [20 21 However, processing long contexts remains a significant challenge. Despite their success and expressiveness, transformer architectures suffer from an intrinsic quadratic computational cost in sequence length,
*Corresponding author
'Our framework applies to autoregressive language models, which encompass all widely-used LLMSs such GPT, Claude, Gemini, and LLaMA.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
(b)
(Ge
Biportite Mutual Information:
1x:: Y) - 1
deuup - 14:00/ X011
History state
X
U8
The quick brown fox
jumps over the lazy
W805 322
3/1
322
Figure (a) The bipartite mutual information between two text segments scales as a power law (sub-volume law) with sequence length L. (b) In autoregressive models, conditional distributions parameterized through the history state x, the latent variables that store past information. Examples of the history state include the recurrent states in state-space models or recurrent neural networks, and the key-value pairs in transformers. (c) The maximum bipartite mutual information a model can express scales with the dimensionality of its history state, dim(z) z). To model long contexts effectively, dim(? 2) must grow at least as fast as the power-law scaling of the true bipartite mutual information.
creating challenges for long sequence generation. Recent advances like DeepSeek have improved per-token efficiency [9, yet the fundamental quadratic cost persists.
Although various architectures have been proposed to address the quadratic scaling [22 these approaches still struggle with truly long sequences in practice. A fundamental gap persists in our theoretical understanding of what is necessary for capturing multi-token long-range dependencies in natural language. Despite efforts characterize these dependencies through various statistical measures 32 35 a theory that can guide practical architecture design remains lacking.
In this work, address the challenges of understanding long- congecontext language modeling through the following contributions (Fig.
1. We present a universal theoretical framework for autoregressive long-context language modeling based on bipartite mutual information.
2. We demonstrate a bipartite mutual information scaling law in natural language and provide reliable empirical validations of power-law scaling across diverse natural language datasets using state-of-theearr LLMs.
3. We derive the L condition from this scaling law, lower bounding the necessary scaling of a model's history state dimension for effective long-context modeling.
4. We validate framework and its predictions across transformer and state-space model (SSM) architectures on both synthetic and natural language datasets of varying lengths.
Our theoretical framework offers crucial insights into understanding an LLM's capability to model long sequences based on its architectural design. By identifying the minimum required growth rate of the history state, our work provides concrete guidance for designing efficient architectures that can effectively handle long contexts, avoiding the quadratic cost of transformers or the capacity limitations of fixed-state models, paving the way for future AI systems.
2 Related Works
Mutual Information Estimation and Application in Machine Learning
Mutual information estimation and optimization have been extensively studied in machine learning, with approaches including variational bounds (36), neural estimators [37), nearest-neighbor methods 38 33], and various upper bounds [40] It has found wide application in areas such as feature
selection 4T. representation learning 42 disentanglement 43] and generative modeling 44
Statistical Properties of Natural Language
Natural language exhibits characteristic statistical scaling behaviors across different levels of analysis. Zipf's law (45) describes how word frequencies decay with their rank, following a plwer-law distribution. Heaps' law [45] characterizes vocabulary growth, showing that the number of unique words scales sublinearly with text length. Hilberg's 's conjecture and relaxed version posit specific scaling laws for entropy and bipartite mutual information in natural language, respectively [46).

---

Neural Scaling Laws
Power-law relationships between model performance, architecture, and computational requirements have been first empirically observed in neural networks [97 with theoretical understanding still
being developed [50] [11, including recent information-theoreti approach [52]. These observations
have guided the development of larger models at fixed context lengths, whereas our work examines
distinct but complementary question: what determines whether model architectures can maintain
performance as context length increases?
Universal Prediction and Markov Modeling
show that they can, in principle. model arbitrary variable-order Markov processes, establishing their theoretical universality in prediction. Our analysis focuses instead on the cnermation-theoredn scaling that governs how much past information growth observed in natural language.
Recent studies on transformers as universal predictors [53
must be stored to reproduce the
Architectures for Efficient Long-Context Modeling
Various approaches have been proposed for processing long sequences. Architectural innovations
targeting quadratic complexity include sparse attention [55
26
25 56), recurrent mechanisms [57]-
59], and alternative formulations [22 60 23 37
61 63 31 Efficient attention implementations
like Flash Attention (64) Lightning Attention [67], and Paged Attention [68] have improved
per-token computational efficiency despite maintaining the underlying complexity scaling.
Long-Form Reasoning and Context Utilization
Chain-of-the -ftthhugtthhhht prompting (20) and seratchpad methods (69 demonstrate the importance of ex- tended context for complex reasoning tasks, emphasizing the urgent need for effective long-range dependency modeling.
Information Theory and Physics-Inspinee Approaches
Recent work has demonstrated how information -ieeerrttio-tcooetiic principles and physics inssiss- approaches can guide machine learning 770 72, leading to novel architectures (73 80, training methods 83
and broad applications 84
88
3 Preliminaries
Mutual Information
Mutual information T(X;Y) quantifies the statistical dependence between random variables and Y, defined as T(X;Y) = (PxY lpx py), where (+I(-) is the Kullback-Leibler (KI.) divergence, and pxy is the joint distribution of X and Y. For discrete random variables, mutual information permits equivalent formulations
I(X;Y) = H(X)+ + H(Y) - H(XY) = H(X) - H(X))) H(Y - H(
where (-)) is the (Shannon) entropy and (-I--) is the conditional entropy. This definition natu- rally extends collections of random variables: [(X1mm Y1.n), with Xay denoting the sequence XX). For notational convenience, we will use boldface notation X := Xi, when the indices are clear from context. Similarly, we will drop the index of single variable X == X, when convenient.
Autoregressive Neural Networks
Modern LLMs predominantly employ autoregressive neural architectures. An autoregres- sive neural network models sequence of conditional probability distributions over tokens ([(w//Wis-p WBBO)) where WBos is the beginning-of-sequeee
token. Throughout this pa- per, use denote model-generated probability distributions (and sometimes the model itself) and p to denote the true underlying distributions. Upper case letters denote random variables, and lower case letters denote specific values realizations of these random variables. These conditional distributions jointly model the probability for a sequence of tokens given a prefix as
q(we. W1:e-1 WBOs)
II i=
(w W1:1-14 WBos).
(2)

---

When l = 1, this reduces to the distribution of unconditional generation g(W1.1 (BBss. During inference, tokens are sampled sequentially from these conditional distributions to generate text or respond to prompts.
For complete list of notation and conventions used throughout this paper, we refer the reader to Appx.
(a)
(b)
(c)
Biportite Mutual Information:
Bipartre Mutual Information
10"
Direct 2:27 (,77
Tao Information
10¹
1.10 --0-----2
VCLUB: 3.26 1771
X
10"
10"
Two--point Mutual Information:
10::1) d-
10"
O
10'
10"
10'
10"
10"
10"
10"
10"
10"
10"
Sequence Length
(0)
Figure 2: (a) Illustration of bipartite and two-point mutual information. The bipartite mutual information measures statistical dependence between two adjacent segments within a text block of length L, whereas the two-point mutual information measures the dependence between two tokens separated by a distance d. (b) Estimates of bipartite mutual information using LLaMA 3.1 405B model [89] on PG19 dataset [90] of pre- 1919 books. (c) Estimates of two-point mutual information on PG 9 dataset. See Appx. B.I B.1l and B.VI for additional results.
Mutual Information Scaling Laws
4.1 Bipartite Mutual Information as Predictive Information
While classical scaling laws in natural language, such as Zipf's and Heaps' laws, primarily ad- dress token-level statistics, a deeper understanding of language modeling necessitates analyzing dependencies between entire text segments. A central challenge in modeling language effectively to characterize how information is carried over from an existing block of text, X, to inform the generation of subsequent block, Y. The bipartite mutual information between such adjacent blocks directly quantifies this inter-segment information transfer, emerging as a particularly revealing measure.
Definition 4.1 (Bipartite Mutual Information [Fig. 2a))). For a consecutive sequence of tokens (random variables) WI... of length L, consider a bipartition of the tokens: X1 := Wi: and Y = WIT1-L- The bipartite mutual information is the mutual information between the two parts 7BP ELL = 18 V..L-e).
The role of bipartite mutual information in quantifying this predictive relationship is formally illuminated by decomposing the entropy of the subsequent block Y:
H(Y) (Y|)) T(X;)) H(Y BB.
This decomposition shows that the total information in Y (its entropy H(Y)) consists of two distinct components: new information unique to Y given X (the conditional entropy H(Y)x)), and information that Y shares with X (the bipartite mutual information /BP). Consequently, TBP precisely measures the amount of information from the preceding block X that is predictive of the next block Y. and therefore, bipartite mutual information is also referred to as predictive information [91].
Despite its crucial role in quantifying predictive information, this form of mutual information in language has remained relatively underexplored. This research gap is primarily due to two factors: the absence of comprehensive theory of natural language that would permit direct calculation, and the substantial challenges in empirically measuring entropy and mutual information for high digensinnnsi distributions from samples.
Existing literature offers differing perspectives on its scaling properties. On one hand, analogies drawing from critical physical systems 992 often based on two-point mutual information

---

scaling (discussed later) suggest that bipartite mutual information should scale logarithmically with sequence length. On the other hand, research in computational linguistics has proposed that it follows power-law growth [98]. behavior often referred to as the sub-volume law (these terms are used interchangeably been constrained by methodological biases and the curse of dimensionality 46 98 99]. Although existing evidence tends to favor sub-volume law growth, these limitations have prevented a definitive characterization. In Sec. 4.3 we address these challenges by leveraging state-of-the-ar LLMs as density estimators, establishing clear power-law scaling for bipartite mutual information across diverse datasets.
this paper). Previous empirical efforts to measure such scaling have
4.2 Two-point Mutual Information
Before presenting our main results concerning bipartite mutual information scaling, it is instructive to discuss two-point mutual information. This measure has conventionally been used to assess long-range dependencies in natural language, and its scaling properties are relatively well understood.
Definition 4.2 (Two-point Mutual Information [Fig. 2a))). The two-point mutual information measures the mutual information between two tokens (random variables) X and Y separated by distance d: I Y).
Specifically, two-point mutual information has been observed follow power-law decay, I a 192 This characteristic decay has prompted arguments that natural language shares structural properties with critical physical systems, which exhibit similar two-point correlation behavior 996 97 However, we contend that such analogies, while offering certain insights, can be misleading when assessing the full complexity of multi-token dependencies crucial for language modeling. The limitations of two-point mutual information in this regard, and why it provides an incomplete characterization for this task, will be detailed in Sec. 4.4 and Appx. [B.III| Our present discussion of two-point mutual information serves primarily to contrast it with the bipartite measure that is central to our work.
4.3 Empirical Verification of Mutual Information Scaling Laws
Bipartite Mutual Information. Measuring bipartite mutual information presents significant chal- lenges without access the underlying probability distribution p. Traditional estimation methods face severe limitations in our setting: K-nearest neighbor estimators [39] and neural estimators like MINE (37 and InfoNCE (100) struggle with the high dimensionality of long text sequences, with errors that increase rapidly as sequence length grows. Additionally, neural estimators require substantial training on large amounts of data to learn representations of natural language distributions, especially for long sequences. Fortunately, recent advances in LLMs allow us to circumvent training our own density estimators by offering high-qualij, approximations q to these distributions (see Appx. V additional discussions). As autoregressive models, LLMs enable efficient computation of conditional probabilities (Sec. 3) and their associated cross-entropies (negative log- lielihoko):)
H (pY| X x == Exxx log g( (())))) (X),
where the expectation is taken over samples from the true underlying distribution pxy. The cross- entropy provides upper bound to the true entropy:
H (PY)x x = (prix x) H (X),)),
(5)
where the conditional cross-entropy and KL divergence implicitly average over px, and HP (or H9) denotes the entropy computed with respect to distribution p (or q).
Using these properties, we can construct a direct estimator for bipartite mutual information:
ddrrett
Epxx [log gY [ - log q( (Y) = 1" (X Y) + spp.g)
(6)
where [2 Y) denotes mutual information with respect to and s(p, g) = Dx (lx) DEL (PY x (1 x)). While this estimator no longer provides bound, it preserves the key property that e (p,,g) 9) + q +
We note that this estimation method faces a specific challenge with modern L.LMs: they model q(w, W11--1, wBos) rather than q(w//wi- (1;-1),, where WBos denotes the BOS token. When sampling
5

---

from the dataset, we can ensure X starts at sentence beginnings, making q( (Y| X, (BBO) ) = q
However, Y may start mid-sentence, creating a mismatch where q * q (BBS))) This introduces errors in estimating H (pr, ,.)). We address this using n-gram corrections for the first two tokens, which are the primary source of this bias (see Appx. [BIV).
To circumvent issues with estimating q ())) we also employ the vCLUB estimator
44
BB.CCLUB = Epxy log ( - Epx Opy log q( ( (Y|X),
ELL.
where the second term be calculated by shuffling the second halves of samples in the dataset. Analysis in [40] shows that vCLUB provides an upper bound on the true bipartite mutual information when q closely approximates p. Even when q deviates moderately from p. though the upper bound property may not hold, vCLUB continues to provide reliable estimates of the true bipartite mutual information.
Our empirical analysis in Fig. 2(b) focuses on equal-length partitions of X and Y L/2), where the bipartite mutual information tends to maximize for fixed L's Nevertheless, the same analysis can be carried out using other partitions where similar results can be obtained (with the results in Ap.). Using both the bias-corrected direct estimator (Fq..6)] and vCLUB estimator [Eq. (7), we measure scaling on the PG19 datase (90) (a collection of books before 1919), employing the LLaMA 3.1 405B model 89 as density estimator q. All measurements robustly demonstrate a clear power-law scaling that extends across thousands of tokens. Additional measurements on WIKIPEDIA [OT and using additional LLMs, along with varying L/L ratios, can be found in Appx. B.1 and B. We note that both estimators likely underestimate the true exponent / (see Appx. B.IlI for discussions).
Two-point Mutual Information. For completeness, we also measure two-point mutual information scaling the same datasets, confirming the expected power-law decay [Fig. (c)). Detailed methodologies for these measurements are provided in Appx. BBVI a B.VII
4.4 Failures of Two-point Mutual Information
As previously noted, while two-point mutual information is easier to measure and more frequently studied existing literature, it often fails to adequately capture the long-range multi-token dependen- cies crucial for natural language modeling. When modeling language, our primary concern is the accurate prediction of future tokens given a preceding context, i.e., modeling of this conditional distribution necessitates clear understanding of the multi-token de- pendencies between the context w1:4-1 and the subsequent tokens We,L- It is important to recognize that this multi-token dependency cannot always be accurately represented by simple aggregation of pairwise (two-point interactions; such an approach can be insufficient or even misleading in certain contexts. The following examples illustrate these potential limitations, and we provide more formal derivations in Appx. VIII
( Effective
Consider simple distribution where all tokens must be identical: p(=1,,2,.... .r.) = [z) 122 xr..) //M, where I1()) is the indicator function that evaluates to when the condition is satisfied and 0 otherwise, and M is the vocabulary size. This distribution permits a Markov chain construction, as p(x1) = 1/M and (=)/=1111) = L(x; = I1-1). thus possessing a simple -oen-ttttokeen dependency structure. Despite this inherent simplicity, the two-point mutual information suggests misleadingly strong "long-range" dependency: maintains a large, constant value of 1 = log M regardless of the distance d, significantly larger than the decaying two-poin mutual information typically observed in natural languages. In contrast, bipartite mutual information correctly reflects this simple dependency structure, with 122 log M remaining constant for any choice of & and L. This indicates that any two segments share exactly the same amount of information (log M), which is no more than the information shared between just two adjacent tokens, accurately capturing the limited nature of the dependency.
For a more realistic setting, we refer to Appx. C for discussion of two families of multivariate Gaussian distributions of varying lengths (details of their construction are in Appx C.U.p. Notably, both families exhibit identical law decay in their two-point mutual information when measured between variables at maximum separation. However, their bipartite mutual information scaling differs dramatically: one scales as L akin to natural language, while the other scales as log L, similar to
*We avoid the BOOKS3 dataset due to copyright infringement concerns.

---

that observed in critical physical systems. This disparity further underscores that two-point mutual information alone may be insufficient distinguish between systems with fundamentally different long-range correlational structures.
Long-Context Language Modeling (L.2M) Condition
Having established bipartite mutual information as a crucial tool for measuring long-range dependen- cies, we analyze how model's capacity to handle long contexts fundamentally depends on its ability to store past information, using bipartite mutual information scaling as our theoretical framework. Intuitively, model natural language effectively. model must be able capture all dependencies between past and future tokens. Since these dependencies (measured by bipartite mutual information) grow with sequence length, the model's state capacity for storing past information (the history state) must necessarily grow as well. We formalize this intuition through the 1.2M condition and explore its implications in detail throughout this section.
5.1 Theoretical Derivations
To analyze how models handle long-ran longrrange dependencies, we first formalize the notion of history state.
Definition 5.1. Consider sequence of tokens W1... Denote Z1 == Wie and VI.L-t == We+ I:L.
Autoregressive neural networks parameterize conditional probabilities by first encoding the input tokens T1:--1 into a set of latent intermediate variables
(x1.2-1 before outputting the conditional probabilities as ((Vi:I.e(<x1.)) = ( ) We define the history state as the smallest set of such latent intermediate variables that fully characterizes the model's output conditional probability. [Fig. [[)].
As illuminating examples, the history state corresponds to the recurrent state RNNs and SSMs after processing token wy-1, and to the key-value pairs up to token th-1 for transformers (see Appx. D Generally, the history state zς is the smallest hidden state responsible for caching all historical information.
The following theorem shows that this history state upper bounds a model's capacity capture bipartite mutual information:
Theorem 5.2. The bipartite mutual information that model can capture is bounded by the size of its history state:
IBPA
- (dii((zL/)) log(M)
(8)
L/22:1
where C some constant and M denotes the vocabulary size.
Proof. This theorem admits multiple independent proofs under different mild and practical assump- tions. See Appx. for details.
We now use this bound to analyze when architectures can maintain performance as sequence length increases. Consider a series of natural language datasets (21 of different lengths, which can be thought of as truncations of an idcal infinite-length dataset.
Definition 5.3. A model is MI-capable the maximum bipartite mutual information it can express satisfies max Br, 2 L/2.1 for any sequence length L, where the maximum is taken over all 0 L/22L
model parameters
Since a model's ability to capture mutual information is bounded by its history state dimension, we immediately obtain]]
*We separate 2z from 2; to accurately reflect distinct role as current input token autoregressive models, though including in zạ would not affect the main results this paper. See Appx.[A A for conventions asymptotic notations.
7

---

Theorem 5.4 (L²M Condition for Single Models), For a model to be MI-capable across all sequence lengths, its history states 2 2L,/2 must satisfy dim(z1),22 2 L/2:1 LP.
Proof. We prove by contrapositive. By Thm. 5.2
if (2 x RP
1/2,11 then max 1./2:1. 8
IPP
1./2:1
implying there exists some L where max I
L/2:L L/2,L= violating MI-capability.
BP
7BP
For some architectures, a single fixed-size model may not satisfy this condition across all sequence lengths. In such cases, we can extend our framework families of models where model size grows with sequence length. Consider a series of models {qr. L=1 of the same architecture, where model size may increase with L.
Definition 5.5. A series of models {9L) is MI-capable if the maximum bipartite mutual informa-
tion each model can express satisfics max 1r..
0r, L/2;L 2 1 IBP L/2;L for its corresponding sequence length L, where the maximum taken over all parameters 0, of model qr-
Theorem 5.6 (L'M Condition for Model Series). For series of models {qu] be MI- capable, the history states zL/2 of each model must satisfy: dim(22//2) X T./2,1. L8.
Note that an MI-capable single model trivially induces an MI-capable series when applied to all sequence lengths, though the converse not true.
5.2 Implications to Common LLM Architectures
We can now apply our framework analyze whether different architectures satisfy the L condition and thus can capture long-range dependencies as sequence length grows.
In transformer-baat models (excluding sparse attention and linear attention variants), the history state consists of stored key-value pairs for all previous tokens. Even with fixed model size, these key-value pairs grow linearly with sequence length: dim(21,/2 L L", This means a single
transformer model naturally satisfies the L'M (single model) condition across all sequence lengths, notwithstanding the quadratic computational cost.
In contrast, SSMs, RNNs, and linear attention models, despite being celebrated for their "infinite" context length and linear complexity, cannot satisfy the L'M condition with a single fixed-size model. Their history state dimension remains constant regardless of sequence length, and our theory demonstrates that this constant-size state cannot capture the growing mutual information. However, these architectures can achieve MI-capability (model-series) through series of models {fr where model size, and thus history state dimension, increases with sequence length. This requirement effectively offsets their computational efficiency advantage when modeling long sequences [
For other architectures, such as sparse attention models and log-linear models, we can similarly analyze their history state scaling to determine whether they satisfy the L condition as single models or require series of growing models. Crucially, any architecture must exhibit pawer-law growth its history state dimension with sequence length in order truly satisfy the single-model L'M condition.
We note that the L. condition addresses model's capacity capture long-range dependencies, its overall language modeling capability. It is a necessary but not sufficient condition: architectures that fail satisfy it will have inherent limitations at longer sequences. while satisfying it does not guarantee effective language modeling, As discussed in Sec. 2 the L condition is also distinct from neural scaling laws, which typically study how model performance scales with model size, dataset size, and compute budget at a fixed sequence length.
"And new model must be trained for each sequence length, which can be prohibitively expensive.

---

(a)
(b)
(c)
10"
gp12 (125m)
ginl-madium (355m))
10
mambe-130m
mambe-780m
nambe-370m
namba--0m
mamba-500 50m
mamba-155 150m
KL KL-Ovarreenss
10
3p12 large (774m)
Kt. (L-Divrgence
10"
mambe2-370m
mambe2-130m
KL-Dwergence
10"
mamba-33 mamba-S70m 370m
150m
10" -
mamba2-700 780m
mamba 1.45
10
10
10"
10
10"
10"
255
1024
4096
1024
4095
10"
10"
10"
Sequence Length
Sequence Length
(7)) (Hidden_din dim (em_rr)
Figure 3:
Evaluation KL-divergence across model architectures trained on synthetic data that satisifes the bipartite mutual information scaling, (a, b) Average KL-divergence per token for models trained on different sequence lengths. (c) Average KL-divergence per token as a function of the ratio between bipartite mutual information and Mamba recurrent state sizes.
(a)
(b)
(c)
3.5x10"
3.5 10"
3.5x10
344100
gol2 (125m)
16"
gotü-medum (355m)
3,4x10"
go12-targe (774m)
mambe- 130m
3.3 10"
mamba 370m
3.3x10
mamoa: 790m
a,3x10"
32 16"
3,2x10
10
3.1x 10"
Cunditonal
3,1x10
3x'10
10
10"
2.0x10'
3x10
28 10"
2,8x10"
28x16 10
27x100
2,3x10
10
10¹
10"
10"
10
10"
10
10"
10'
10"
10"
10"
Token Position
Toxan Powiion
Token Positon
Figure 4: Position-wii wise conditional negative log likelihood (NLL) evaluation for models trained on 8192-token sequences on the PG 19 dataset [90]
Empirical Verification
Sub-volume-law Gaussian Test. We first validate our theory using a synthetic dataset comprising a family of multivariate Gaussian distributions (see Appx.C]ror details). This distribution family closely mimics the scaling of both bipartite and two-point mutual information observed in natural language, while crucially allowing for the efficient calculation of conditional probabilities and KL divergences- calculations that would be intractable with wal-world natural language datasets. Furthermore, the synthetic data enables an isolated assessment of model's ability to handle long sequences, without interference from its capacity to understand the semantic meanings of natural language.
In Fig. we present the average per-token KL divergence (defined in Appx. F.IV) for GPT2, Mamba, and Mamba2 models, serving as representative transformer and SSM architectures. Panels (a) and (b) show that GPT2 maintains consistent KI. divergence across different sequence lengths. In contrast, smaller Mamba and Mamba2 models exhibit increasing difficulty with longer contexts, necessitating substantially larger model sizes to achieve comparable performance
sequence length of 4096. Panel (c) offers direct confirmation of our theoretical framework: it plots KL divergence against the ratio of bipartite mutual information to the recurrent state size for Mamba models of varying configurations. For this, we varied sequence lengths from 64 to 16,384 and model sizes from 50M to 1.4B parameters. The KL divergence values from these diverse configurations remarkably collapse onto single curve, demonstrating that model performance depends only on the ratio BB ddin(z) ()). This finding precisely confirms our theory that for effective long-context modeling, a model's history state size must scale at least as fast as the bipartite mutual information present in the data.
These findings have important implications for modeling very long sequences. Extrapolating from the measured scaling in Fig. 2 (which likely underestimates the true exponent), the bipartite mutual information for a sequence one million tokens could exceed 60,000 nats. Our results in Fig. ( suggest that maintaining low KL divergence at such bipartite mutual information levels would require recurrent state dimensions approaching one million.
PG19 Test. We then extend our analysis to the PG19 dataset 90 a high-quality collection of pre-191 1919 books exhibiting long contextual dependencies.
In Fig. we show the position- wise conditional negative log likelihood (NLL) models trained on the PG19 dataset [90] with 8 8922-token sequences, where calculating KL-divergence is not feasible.

---

Note that, unlike conditional KL divergence, conditional NLL naturally decreases with token position (see Appx. for details). Two key patterns emerge from this experiment: First, Mamba models typically outperform GPT2 models of comparable size at early token positions, but this advantage diminishes and eventually reverses later positions. Most notably, Mamba's 's NLL tends to plateau beyond certain positions unless the model size is increased, while GPT2's s NLL continues to improve. Second, the performance gap between Mamba and GPT2 narrows with increasing model size. Both observations align with our theoretical predictions: since Mamba's ' history state size remains fixed regardless of sequence position, its performance inevitably degrades beyond a certain token position unless model size increases. As model size grows, the history state size also increases, eventually becoming sufficient to capture the mutual information present in 8192-token sequences.
We note that Mamba's 's linear computational complexity can make larger Mamba models practically more efficient than smaller transformers. Our results should not interpreted as suggesting Mamba's architectural inferiority. Rather, they demonstrate how different architectures handle long sequences differently, and that a model's capacity for capturing long-range dependencies aligns with our theoretical L framework, regardless of the architecture.
Additional experimental results can be found in Appx.
7 Discussion
The L condition establishes fundamental relationship between the information structure of data and architectural requirements. This relationship manifests differently across architectures: transform- ers with linearly growing key-value caches naturally satisfy the condition as single models (given our measured sublinear mutual information scaling with B < 1), though at quadratic computational cost, while SSMs and similar fixed-state architectures require model size to scale with sequence length to achieve comparable mutual information capability.
Interestingly, transformers appear to over-provision their history state relative to the measured mutual information scaling: their linear growth exceeds the sublinear (L with 3 < 1) scaling we observe. This observation provides a clear goal for future architecture design. Although it remains unclear whether the ver-provisioning necessary for other aspects of language modeling beyond pure information storage, the gap between the linear growth of transformers and the L requirement suggests a concrete target: architectures that precisely match the required sublinear scaling could potentially achieve substantially improved efficiency while maintaining the capacity to capture long-range dependencies.
Our framework applies to autoregressive language models, which encompass the vast majority of widely-used LLMs. While diffusion-based language models represent an alternative generative paradigm, they typically still operate autoregressively at a higher level of granularity, making our framework applicable in practice. Extending our framework to hybrid architectures that combine different mechanisms represents an important research direction that could unify our understanding of how diverse architectural choices affect long-context capabilities. Applying our framework to other sequential domains, such as biological sequences like proteins or DNA, or computer code, also presents particularly promising direction, as different mutual information scaling behaviors in these domains could provide principled explanation for the observed differences in model requirements across domains.
Additional discussions on limitations and broader impacts can be found in Appx. Hand]]
8 Conclusion
We establish a bipartite mutual information scaling law that characterizes long-range dependencies in natural language and introduce the L'MM condition, which lower bounds the necessary scaling of a model's history state for effective long-context modeling. By identifying the minimum required growth rate of the history state, our work provides a principled foundation for understanding how different architectures handle long contexts. This framework establishes concrete, information- theoretical, and data-driven targets that could guide the design of architectures balancing compu- tational efficiency with the capacity to capture long-range dependencies in natural language and potentially beyond.
10

---

Acknowledgements
The authors acknowledge support from the National Science Foundation under Cooperative Agree- ment PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions). Z.C. acknowledges support from the Mathworks Fellowship. Z.C. thanks Rumen Dangovski for help- ful discussions.
Z.C. and O.M. thank Amazon Web Services account team, including Brian McCarthy and Jared Novotny, for technical support. The research was sponsored by the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750- 100000-9--2 1-00- The computations this paper were partly run the FASRC cluster supported by the FAS Division of Science Research Computing Group at Harvard University. This research used the DeltaAl advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois, through allocation CIS240904 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #21 138259,, #21 #2138286, #2138307, #2137603, and #2138296, and through the National Artificial Intelligence Research Resource (NAIRR) Pilot NAIRR250043.
References
[1] Brown, T. er al. Language Models are Few-Shot Learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. Lin, H. (eds.) Advances in Neu- ral Information Processing Systems, vol. 33, 1877-1901 (Curran Associates, Inc.,
2020). URL https: wttp:///pcoeedngneeeiii.iee wwtpp/cccppppe..e.eeeee
1457 (olebrtbbb407711111faaaaaac44441
[2] Chowdhery, A. et al. PaLM: Scaling Language Modeling with Pathways (2022). URL
https://arxiv. hwgp://rxxvv...g./ooo/o http:/..rg/abbbaabaeeboeb 2204. 02311 2204.02311
[3] Touvron, H. et al. LLaMA: Open and Efficient Foundation Language Models (2023). URL
https://arxiv.ort org/ abs ab//a 13971
2302. 13971
[4] Touvron, H. al. Llama 2: Open Foundation and Fine-Tunel Chat Models (2023). URL https://arxiv.o org/ abs/ 220. 09288 09288
[5] OpenAl et al. GPT-4 Technical Report (2024). URL https://azziv.or//abb////
08774 2303.08774
[6] Team, G. et al. Gemini: A family of highly capable multimodal models (2025). URL ht
11805 2312. 11805
[7] Comanici, G. el al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context, and generation agentic capabilities (2025). htt..htts::/arrxw.rrr
abs 228/2207.06661 2507 .66261
[8] Grattafiori, A. el al. The Llama 3 Herd of Models (2024). URL URRLAttpps///arrvoorr//// 2407. 21783 2407 .21783
[9] DeepSeck-Al et al. DeepSeek-V3 Technical Report (2024). URL abs 2bs22412.11 19437 2412. 19437
[10] Guo, D. et al. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning, Nature 645, 633 638 (2025). http:////../aaa////.///////////0
[11] Qwen al. Qwen2.5 technical report (2025). URL /ttp:///////////////////////////
2412. 15115
[12] Yang, A. et al. Qwen3 technical report (2025). URL https://arxivvorg/alt 09388 093888
[13] Stiennon, N. et al. Learning to summarize from human feedback (2020).
[14] Yuan, A., Coenen, A., Reif, E. & Ippolito, D. Wordcraft: Story Writing With Large Language Models (2022).

---

[15] Gou, Z. er al. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (2023).
[16] Wang, Y. el al. CodeTT+: Open Code Large Language Models for Code Understanding and Generation (2023).
[17] Bubeck, S. et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4 (2023).
[18] Ge, el al. OpenAGI: When LLM Meets Domain Experts (2023).
[19] Kosinski, M. Theory of Mind May Have Spontaneously Emerged in Large Language Models (2023).
[20] Wei, J. et al. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Oh, H., Agarwal, A., Belgrave, D. Cho, K. (eds.) Advances in Neural Information Processing Systems (2022).
[21] Wang. J., Meng, F., Liang. Y. & Zhou, DRT-ol: Optimized Deep Reasoning Translation via Long Chain-of-Thought (2024). URL
2412
17498
[22] Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. Transformers are RNNs: Fast Autore- gressive Transformers with Linear Attention (2020). URL BtRppttts://axxi org wwwgg/abs/2000 16236 2006.. 16236
[23] Gu, A., Goel, K. & Re, C. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference Learning Representations (2022). URL https: //openreview
[24] Zhu, C. al. Looo-Short Transformer: Efficient Transformers for Language and Vision (2021). htLhttpps//arrii.orr////as/
02192 02092
[25] Beltagy, I., Peters, M. E, & Cohan, A. Longformer: The Long- Transformer (2020).
URL https: /tttps://axiivi www.ots/////./oooooo//..oo....at h2tp://bs/000..... 2200bb05100
2004. 05140
[26] Ding, J. et al. LongNet: Scaling Transformers to 1,000,000,000 Tokens (2023). URL
https://arxiv.ct wwwp://v.wrrabbb////eeee/ 02486 2307. 02486
[27] Gu, A. & Dao, T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2024). hww.lttppp//pppeeeeeeeeeeee//err
[28] Dao, T. & Gu, A. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (2024). URL attps://arxiv> www.xiiv.rgg/bb//////// 21060 2405.21060
[29] Peng, et RWKV: Reinventing RNNs for the Transformer Era. The 2023 Conference Empirical Methods in Natural Language Processing (2023). URL http::htt:://// htttps:://penrrevie
net/forum?id htt./forummidd77axcaaaa332
[30] Sun, Y. et al. Learning (Learn at Test Time): RNNs with Expressive Hidden States (2024).
ptl.hhttss///rxiit
/a00s//24700000 04620) 2467.04620
[31] Guo, H. et al. Log-linear attention (2025). URL httpl//pp..aaat...oo///////../.0 2476.04761
[32] Ebeling, W. & Püschel, T. Entropy and Long-Range Correlations in Literary English (1994).
[33] Debowski, L. Excess entropy in natural language: present state and perspectives (2011).
[34] Bentz, C., Alikaniotis, D., Cysouw, M. & Cancho, F. The Entropy of Words Learnability and Expressivity across More than 1000 Languages (2017).
12

---

[35] Futrell, R., Qian, P., Gibson, E... Fedorenko, E. & Blank, I. Syntactic dependencies correspond word pairs with high mutual information. In Gerdes, K. & Kahane, S. (eds.) Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019). 3-13 (Association for Computational Linguisties, Paris, France, 2019). URL httpp://
aclanthology Twantthoooy..oog/BB.... orgg/19-7703/
[36] Poole, B., Ozair, S., van den Oord, A., Alemi, A. Tucker, G. On Variational Bounds of Mutual Information (2019). URL wwtpt:::t//ttttte wwtg/a/////////eoo 06922 $905.0697)
[37] Belghazi, M. I. et al. MINE: Mutual Information Neural Estimation (2021). URL https:
//arxiv. org httg/abs/18011. 04062
1801. 04062
[38] Kraskov, A., Stögbauer, H. & Grassberger, P. Estimatiing muttual information. Phys. Rev. E 69, 066138 (2004). http:ppppp/.....a.p.pooo///////p
69. 066138
[39] Gao, S., Steeg, G. V. & Galstyan, A. Efficient Estimation of Mutual Information for Strongly Dependent Variables (2015).
1411 2003
[40] Cheng, P. et al. CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information (2020).
URL httpt://a/axxi org/ hwwgg/abs/20000 2206 12013 2006.12013
[41] Brown, G., Pocock, A., Zhao, M.-J. & Luján, M. Conditional likelihood maximisation: A unifying framework for information theoretic feature selection. Journal of Machine Learning Research 13, 27-66 (2012). URL http: //mprr.r////..////////// wwwgg/aaaars//////rorroe// 2a httml
[42] Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S. & Lucic, M. On Mutual Informa- tion Maximization for Representation Learning. In International Conference on Learning Representations (2020). URL httptttst////ereeeeeeee
net. wet.foorummiite htte:/oorrrmmtdddoooo22220 id=rkxoh24FPIi
[43] Chen, X. et al. InfoGAN: Interpretable Representation Learning by Information Maximiz- ing Generative Adversarial Nets. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I. & Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29 (Curran As- sociates, Inc., 2016). URL http:: httpsss:/ppooceeenggg.. htw.eeeeeddinnneeeenespeeeeeeeee wwwrips-ep.... cc/paper wiles/paper.rr pdf
[44] Hjelm, R. D. et al. Learning deep represeenttttonnnnnnn by mutual information estimation and maximization (2019). URL URLLttps://arrvvv.org/abbb/
06670 1 1808.0665
[45] Gelbukh, A. & Sidorov, G. Zipf and Heaps Laws' Coefficients Depend Language. In Conference on Intelligent Text Processing and Computational Linguistics (2001). URL https://api. semanticscholar wrtp:ooroo/ooos////// 20344161
[46] Hilberg, W. Der bekannte Grenzwert der redundanzfreien Information in Texten eine Fehlinterpretation der Shannonschen Experimente? Frequenz 44, 243 248 (1990). URL pttps://apil semant icschooar oot.aatiieecctooaa.ooooooooorroo 124878549
[47] Hoffmann, J. al. Training compute-optimal large language models (2022). URL https
/awxxiiorg/abs//////. /2003 15556 2203. 15556
[48] Kaplan, J. al. Scaling Laws for Neural Language Models (2020). URL htlthtttss/arrtt
htg/abbs/201.00835
08361
[49] Biderman, S. et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023). htt.:httpps://rrr..ooo//aaa htsp:/bb/2200000 013373 2304.01373
[50] Bahri, Y., Dyer, E., Kaplan, J., Lee, J. Sharma, U. Explaining neural scaling laws, Pro- ceedings of the National Academy of Sciences 121, 02311878111 (2024). URL wwtps:ttp//.. /ww pnas. orgooogg/oooo hts//di//abbae www1177//ass.... 10.1073/ pnasppps..... 231 1878121
1073 /pta//aass..o 231 1878121
www.:pnas.oog/ooeeom oor /pdf
[51] Bordelon, B., Atanasoy, Pehlevan, C. Dynamical Model of Neural Scaling Laws (2024). https://://aax org/ abs////s..22...///////////// 01092 2402.01090
13

---

[52] Nayak, A. K. & Varshney, L. R. An information theory of compute-optimal size scaling, emergence, and plateaus in language models (2024). URL
01243 2410.01243
[53] Basu, S., Choraria, M. & Varshney, L. R. Transformers universal predictors (2023). URL 07843 2307. 07843
https://arxiv. org/ wttp:/iwwwwgg////////aaaao
[54] Zhou, R., Tian, C. & Diggavi, S. Transformers learn variable-order markov chains in-context (2024). URL http:pp:://rrii.oorrgaaaaaaaaeee www.:iii.wrgg/aas...eee haarr//bb//21100 05493 2410. 05493
[55] Child, R., Gray, S., Radford, A. & Sutskever, Generating Long Sequences with Sparse Transformers (2019).
10509 1904. 10509
[56) Zaheer, M. al. Big Bird: Transformers for Longer Sequences. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds.) Advances in Neu- ral Information Processing Systems, vol. 33, 17283-17297 (Curran Associates, Inc., 2020). URL c8512d1 2 241111168888885556666763333555a0 pdf
[57] Dai, Z. et al. Transformer-X1: Attentve Language Models Beyond a Fixed-Length Context (2019). URL.https://rxxi..rrr//
00266 1901. 02860
[58] Rac, J. W., Potapenko, A., Jayakumar, S. M. & Lillicrap. T. P. Compressive Transformers
for Lang-Range Sequence Modelling (2019). URL htRtpt:::: www.aaaww....aa/aaaaaraeeeeeeeee
[59] Sukhbaatar, S., Grave, E., Bojanowski, P. Joulin, A. Adaptive Attention Span Transform- ers URL
07799 1905.07799
[60] He, Z.. Qin, Z., Prakriya, N., Sun, Y. & Cong, J. HMT: Hierarchical Memory Transformer for Long Context Language Processing (2024). URL U 06067 2405.06067.
[61] Gu, A., Goel, K. & Ré, C. Efficiently Modeling Long Sequences with Structured State Spaces (2022). URL
2131.00396
[62] Beck, M. et al. XLSTM: Extended Long Short-Term Memory (2024). URL httpss://arxa org/abs/2405.n 04517 2405.04511
[63] De, S. et al. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (2024).
19427 2402. 19427
[64] Dao, T., Fu, D. Y., Ermon, S.. Rudra, A. & Ré, C. FlashAttention: Fast and Memory- Exact Attention with IO-Awareness (2022). URL https://arxiv..rrg /20.. 14135
2205. 14135]
[65] Dao, FlashAttention-2:: Faster Attention with Better Parallelism and Work Partitioning (2023). htt.htps://axii.orr///eee/
08691 2307. 08691
[66] Shah, el al. FlashAttention-::: Fast and Accurate Attention with Asynchrony and Low- precision (2024). lttp::ttp:// http://rxxii..orr/a/////////// 08608 2407.08608
[67] Qin, Z. et al. Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths Large Language Models (2024). URL 7401.04658
04658
[68] Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023). URLLAttge//axxxvvoo////////2...0
2309 .61180
[69] Nye, M. et al. Show Your Work: Scratchpads Intermecci Computation with Language Models (2021). JKLL:ttpe///rrrr..oog/////2222.0
2112.003
[70] Tishby, N. Zaslavsky, Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 1-5 (2015).
14

---

[71] Goldfeld, Z. & Polyanskiy. Y. The Information Bottleneck Problem and its Applications in Machine Learning, IEEE Journal on Selected Areas in Information Theory 19-38 (2020).
[72] Chen, Z. & Luo. D. Entangling Intelligence: Al-Quantum Crossovers and Perspectives. In 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), 516-519 (IEEE, 2024). URL httts://ieeexpl.ee iieee #tg/doccuennt/108355
[73] Luo, D. & Clark, B. K. Backflow Transformations via Neural Networks for Quantum Many- Body Wave Functions, Phys. Rev. Lett. 122, 226401 (2019). URL https:///ini aps org/
doi/10. 1w11/100011110 Pw..0///yseeeeett...mo 122. 226401
[74] Luo, D. et al. Gauge-invariant and anyonic-symmetric autoregressive neural network for quantum lattice models. Physical Review Research 013216 (2023). URL https://
journals www.ppp.oggg. wttppppp...ppppeeeerraaa//aaaaaa
01..21
[75] Wang, J. et al. Spacetime neural network for high dimensional quantum dynamics. arXiv preprint arXiv:2108..222220 02200 (2021). URL https::/aazivv..rrrg.... wwwprgg///////////ee///ooeoeeaae http://wwwwaaaaa...///// /2108.0220000
[76] Chen, Z., Luo, D., Hu, K. & Clark, B. K. Simulating 2+ 1d lattice quantum electrodynamics at finite density with neural flow wavefunctions. arXiv preprint urXiv:2212.0683: (2022). URL
https://arx:: v. or h///s/////// 221222222.22 06835
[77] Lami, G., Carleo, G. & Collura, M. Matrix product states with backflow correlations. Phys.
htt.a/aa......./oo/o/.ooo... Rev. B 106, 1081111 (2022). URL https: htttpa///inkk aps wwwp//////..ooo/ 1103/ 10t3//hyo..100///heeeee
106. L081111
[78] Wu, D., Rossi, R., Vicentini. F. Carleo, G. From tensor-network quantum states tensorial recurrent neural networks. Physical Review Research 5 (2023). URL http://dx. doi org/
10.
[79] Chen, Z. Newhouse, L., Chen, E., Luo, D. & Soljacic, M. ANTN: Bridg- ing autoregressive neural networks and tensor networks for quantum many-body simulation. Advances in neural information processing systems 36. 450 476 (2023). URL https::/ppooceeding nww. hww.r//dnnnnnoirri.cc.cppppppppe httpppppppreeeeppppppeeeeeeepeee http://ppppee/22///eee 22023/222h/ htt7aa/wwaaaac0ccceeeeeeaaeeaeee
html
[80] Dugan, O. et al. OccamLLM: Fast and Exact Language Model Arithmetic in Single Step. Advances in Neural Information Processing Systems 37, 35665-35699 (2024). URL hrRlitttps//pproceddiggg.o neur fappr_fileesssseeeepepppppppeeee paper (haper/202/hhane /2024/hash/ Secrr7/fffooofffae7777ss0002244-
html
[81] Stokes, J.. Izaac, J.. Killoran, N. & Carleo, G. Quantum Natural Gradient. Quantum 4, 269 (2020). URL htttt::t////ii..oo org/ 20w.0/223312222
050 25- 25-269
[82] Chen, Z., Mccarran, J.. Vizcaino. E., Soljacic, M. & Luo, D. TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision. In International Conference on Machine Learning, 7 7143-7162 (PMLR, 2024). URL https::/pproceedingg
mlr mlr.press/ 22335/chen24e html
[83] Chen, Z. et al. QuanTA: Efficient high-rank fine-tuning Ilms with quantum-informed tensor adaptation. Advances in Neural Information Processing Systems 37, 92210-92245 (2024). URL https: http:///wwocccoggoo.oooooooo.oop
www.innn......unneeee nwwrips...... cttp:sccccppppeeeaeeeeeeee www.fflllapppeeee paper /222//ash///////
html
[84] Carleo, G. et al. Machine learning and the physical sciences. Rev Mod. Phys. 91, 045002 URL
[85] Lee, C.-H. et al. Deep Learning Enabled Strain Mapping of Single-Atom Defects in Two- Dimensional Transition Metal Dichalcogenides with Sub- Picometer Precision. Nano Letters 20, 3369-3377 (2020). httphtt::///oo:
1021/ 2000////// accs. nanolett. 10001ett.000000 0c00269
15

---

[86] Luo, D., Chen, Z., Carrasquilla, J. & Clark, B. K. Autoregressive neural network for simulating open quantum systems via a probabilistic formulation. Physical review let- ters 128, 090501 (2022). URL ott.hh/ppr///oraa........o./oo/p
PhysRevLett. 128. 090501
propectyypredictionnnttoo [87] Moro, V. et al. Multimodal foundation models for material property prwdictionn and discovery. Newton 1 (2025). URL htt.://///..a.oooooo/ooooo//////
00008-8
[88] Choi, S. el al. Photonic probabilistic machine learning using quantum noise. Nature Communications 15, 7760 (2024). URL https: /twp///wwww/// nature. com/ articlss/ c41467-024-61509-1
[89] Meta. URL Bethhtts::/eee meta. http:////....atccc..oooo/o//aaaa 3-11/
[90] Rae, J. W., Potapenko. A.. Jayakumar, S. M., Hillier, C. & Lillicrap, T. P. Compressive Transformers for Long-Range Sequence Modelling. International Conference on Learning Representations (2020). URL https:: het/foorumiad
[91] Bialek, W. & Tishby, N. Predictive Information (1999). URL http::t/aa.ii...rg/////.
cond- mand-mmt///e /9902333191
cond- fand-mat/9900 cond-mat/9902341
[92] Ebeling, W. & Pöschel, T. Entropy and Long-Range Correlations in Literary English. Euro-
physics Letters 26, 241 (1994). http 10. 10w.12209/9990 5075/26/4/
001
[93] Ebeling, W. & Neiman, A. Long-range correlations between letters and sentences in texts.
Physica A: Statistical Mechanics and its Applications 215. 233-241 (1995). URL httpa:
7/ //www. h/www.scient sciencediieectt....ccccmccc. com htt/s/ciincceeceeeee /ttp://iieeeee/attiill////////// httpe///./////77777717713/ /ttp:////aa.tcee////////////////
1955002223
[94] MONTEMURRO, M. A. & PURY, P. A. LONG-RANGE FRACTAL
IN
LITERARY CORPORA. Fractals 10. 451-46116111 161 (2002). URL /ttp::://doi.oo///111
30218348X0200125
https://ddtt hwleddii.ogggg////oe 10. 1111111////222114000000000000000
[95] Shen, Mutual Information Scaling and Expressive Power of Sequence Models (2019). URL
https://arxx: www.ivi.org.oooo org http:/bbs/190..022/ 04271 1905. 04271
[96] Lin, H. W. & Tegmark, M. Critical Behavior in Physics and Probabilistic Formal Languages. Entropy 19 (2017). Itt.ttts::/ h/wwttss:/wwww.... http://wwwwmmpp..comm/ 1099 4300//9//////9999
[97] Stanley, H. E. Power laws and universality. Nature 378, 554-555 (1995).
[98] Lukasz Debowski. The Relaxed Hilberg Conjecture: A Review and New Experimental Support. Journal of Quantitative Linguistics 22, 311-331 (2015). URL /tttts:://oi... org/10. 1080/ 09296174. 2015. 1106268
14tp///0000000000000777714100010 2015. 1106268
[99] Lu, S., Kanász-Nagy, M., Kukuljan. & Cirac, J. I. Tensor networks and efficient descriptions classical data (2024). URL http:/:/wwwwvvx...aaa//b 2213..08722 06872 2103. 06872
[100] van den Oord, A., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding (2019). URL / http://///aaaaoorr/a/////0000/0
1807.03745
[101] Foundation, W. Wikimedia downloads. URL httpt::/t//::pp.. htt.://pppssiiiiiiiii... wikimedia.a.....m
[102] Grassberger, P. Entropy Estimates from Insufficient Samplings (2008). URL Utt.httss://arxin
org/abs ht//abb//pphsiccs/0000/ 138 physiccs/0007/.3
[103] Inc., W. R. Mathematica, Version 14.2. URL https:// hwtp://ww...a....ommaaaaeaeeeeee Champaign, IL. 2024.
[104] Donsker, M. D. & Varadhan, S. S. Asymptotic evaluation of certain markov process ex- pectations for large time, Communications on pure and applied mathematics 28, 1-47 (1975).
16

---

[105] Elhage, N. et al. Toy Models of Superposition (2022). URL https///axxiv.org///ss
2209. 10652 2209.10652
[106] Park, K., Choe, Y. J. & Veitch, V. The Linear Representation Hypothesis and the Geometry of Large Language Models (2023).
[107] Jiang, Y., Rajendran, G., Ravikumar, P., Aragam, B. & Veitch, V. On the Origins of Linear Representations in Large Language Models (2024).
[108] Kabatiansky, G. A. & Levenshtein, I. On bounds for packings on a sphere and in space.
Problemy Peredachi Informatsii 14, 3-25 (1978). URL htthttp: /mi. mathnet... mwt.hettrrrpppisttt........... ru/ppi1518 English translation: Problems Inform. Transmission, vol. 14, no. pp. 1-17, 1978.
[109] Cohn, H. & Zhao, Y. Sphere packing bounds via spherical codes. Duke Mathematical Journal 163 (2014). URL, URL.htpp: http://dx. doi org/10. 121.00011110/000000077777020000
[110] Kawabata,
Dembo, A. The rate-distortion dimension sets and measures. IEEE Transactions on Information Theory 40, 1564-1572 (1994).
[111] Black, S. et al. GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022). 2204. 06745
[112] Wolf, T. er al. HuggingFFaces Transoooonssssss Statc-of-the-ar Natural Language Processing (2020). UKL.Ittgee//arrrr..o////////0...
1910.0377)
[113] Paszke, A. et al. PyTorch: An Imperative Style, Hegh-Hiigghhheeee Performance Deep Learning Library (2019). It..:ttp..//arrtt.or////////////
1912.01703
[114] J. et al. FlexAttention Efficin Resolution Vision-Language Models (2024). URL
v. htapa://aarai..oggbb/////// 20228
20228 2407. 20228
17

---

A Notations and Conventions
Basic notations
Random variables: Uppercase letters (e.g., X, Y, W) denote random variables.
Lowercase letters (e.g., V. w) denote specific values or realizations of random variables.
Realizations:
Sequences: X denotes the sequence (X,, Ki Xj).
Bold notation: X == Xay when indices are clear from context.
Single variable shorthand: X := X; when the index is clear from context.
Logarithms: While the choice of base does not affect the scaling laws (only the multiplicative
constants), all logarithms natural logarithms (base e) unless otherwise specified.
Information-theorriic quantities
H()): Shannon entropy.
H(-l-): Conditional (Shannon) entropy.
[(5-): (Shannon) mutual information.
(xl(-l)): Kullback-Leibler divergence.
H-)():: Entropy computed with respect to distribution p.
(.):): Entropy computed with respect to distribution q.
(p, 4): Cross-entropy between distributions p and
Asymptotic notations
f(n) g(n): and g have the same asymptotic growth rate, i.e., f(n) O(g(n).
f(n) g(n): grows strictly faster than g asymptotically, i.e., f(n) w(g(n)).
f(n) k g(n): f grows at least as fast as g. asymptotically, i.c., f(n) = 0(g(n)).
f(n) g(n): grows strictly slower than g asymptotically, i... f(n) o(g(n)).
[(m) z g()): f grows at most as fast as asymptotically, i.e., f(n) = (g(n)).
Distributions and expectations
p: True underlying probability distribution (of natural language).
q: Model-generated probability distribution (sometimes refers to the model itself).
Epp-J: Expectation with respect to distribution p.
Px & Py: Product distribution marginals px and py-
Model-specific notations
WBos: Beginning-of token.
M: Vocabulary size.
L: Sequence length.
f: Position of sequence split for bipartite mutual information.
dim Dimensionality of the history state
8: Model parameters.
Special notations
/BP: Bipartite mutual information.
IP, Two-point mutual information.
1(-): Indicator function (equals 1 when condition true, 0 otherwise).
The notations are used consistently throughout the main text and appendices unless otherwise specified in local contexts.
18

---

B Additional Details on Mutual Information Scalings
B.I Bipartite Mutual Information Scaling with Additional LLMs on Additional Datasets
In the main text, we use the I.LMM 3.1 40SB model as the density estimator and measured the hipar- tite mutual information scaling on PG19 dataset. In this section, we provide additional estimations of the bipartite mutual information scaling using the DeepSeek V3 Base model and on WIKIPEDIA dataset. We note that because we merely measuring the conditional probabilities the input tokens without interactions with the agent, we believe the non non-instruction. -iinttunedded model better suits our tasks.
Biparitte Mutual Information
10
Blpartio Information
Direct 3.79-1
vCLUB: 4.13 4.13-25.0
10"
10"
Direct 4,50 4,50-L440 L.00
vCLUB: 7.01 8.00
10'
10'
10"
10"
10'
10
Sequence Langth (4.
Sequence Langth (6)
Figure B.1: Bipartite mutual information estimation using (left) LLaMA 3.1 405B the WIKIPEDIA dataset and (right) Deepseek V3 Base model on the PG19 dataset. All direct measurements include the bias correction described in Appx. B.IV
In Fig. B. report the results WIKIPEDIA dataset using LLaMA 3.1 405B model as the density estimator and on PG19 dataset using Deepseek V3 Base model as the density estimator. We find that in both cases, clear sub-vole volume growth behavior is observed. We note that the measured exponent should be taken with a grain of salt and likely underestimates the true mutual information scaling due reasons explained in Appx. B.llI
B.II Bipartite Mutual Information Scaling Under Various Ratios of E/L
In the main text, we focused on the bipartite mutual information with equal splits. However, the bipartite mutual information scaling is not limited to equal biparitition. In this section, we provide additional results for various ratios L/L.
In Fig. B,2 we provide estimation of the bipartite mutual information scaling for L/L = 3 and f/L= 4. All results show clear power-law relations, and are consistent with Fig. in the main text. These results can be used to support the L'M condition with similar arguments as in the main text.
B.III Why The Estimated Exponent 3 Is Likely An Underestimation?
In the main text, we mentioned that measured exponent 3 using LLMs likely underestimates the true 3. Here, we discuss the reasons.
For the direct estimator,
BP direct
= H(y - Hv
(B.1)
ELL
both terms (without the minus sign) overestimates the true (conditional) entropy, but for different extent and different scales.
At small L, the first term suffers from the bias from the BOS token as discussed in Appx. B.IV The second term, despite also overestimation, does not suffer from the BOS token issue. Therelore, at small L, the direct estimator tends to overestimate the true entropy.
At large L, the bias from the BOS token is less severe. However, modeling p requires the model to correctly capture all the dependencies between X and Y, making significantly harder than modeling pYY) alone. Therefore, X is likely worse estimation of the true distribution than ((Y), resulting more overestimation the second term, and an underestimation of the bipartite mutual information.
19

---

PG19 (Books) \/L
PG19 (Books)
Elparite Mutual Informmaiin
10"
Direct: 2.01 2.01-1am
Biparite Motuel Informmtton
10"
Direct: 1.65 -2a
vCLUB: 2.93-La2
2.47 -
10"
10"
10
10
10
10"
10
10"
10"
Sequence Length (L)
Sequence Length (L)
Wikipedia I/L 3
Wikipedia E/L
Biparite Musual Information
10
Biparite Musual Information
10"
10²
10"
Direct: L.57
4.37 (9:5
VGLUB: 7.11 6.
VOLUB: 6,54 1-44-%%0
10
10"
10"
10"
10"
10'
Sequence Length (L)
Sequence Length (L)
Figure B.2: Bipartite mutual information estimation using different ratios of E/L. All results suggest the existence of power-law scaling, with various fitted exponents.
This means that the direct estimator tends overestimate the true bipartite mutual information small L and underestimate at large L, resulting in an underestimation of the fitted exponent.
The vCLUB estimator, pointed out in 40), is an upper bound to the true mutual information if q is close to p, but fails to maintain the property when the KL--dieereence between them increases. Therefore, it is likely that this estimator also overestimates the true bipartite mutual information at small L and undercstimates it at large L, resulting in a similar underestimation direct estimator. As our fitted exponent for the vCLUB estimator is smaller than that of the direct estimator, we conclude that the vCLUB estimator has larger bias in this case, and is reasonable believe that the true exponent is even larger.
the fitted exponent
B.IV Direct Estimation of Bipartite Mutual Information Using LLMs
In the main text, our direct estimator for the bipartite mutual information is
BP, direct = Epxy [log q( Y () log q( Y)) = HpY (((py.y)))) (p))))(((xxxxxxxx)) = (X; + 9).
EL
(B.2)
where as usual, of tokens. However, as discussed in the main text, the H .Y term suffers from an additional bias-we cannot guarantee that Y starts at the beginning of sentence, but LLMs model distributions conditioned on BOS token. To mitigate this issue, we use n-gram calculations to correct the entropy of the first two tokens explained below.
X1.t = W1:e and : -1:L- := We+1.L with WILL being sequence
We first rewrite the (marginal) cross entropy as
L-t
H(py, = log q Y)) = Σ
Ep log (Y.-Y)))))
(B.3)
where as usual, the expectation over the conditional variable omitted but implied in the cross entropy caleulation.
In modern LLMs we can only compute (11311/1111,1 WBos) # q(uil1rs-1) resulting in an additional error in the bipartite mutual information estimation. In practice, this difference becomes less pronounced for larger i, because it matters less the sequence starts the beginning of a sequence not if there many V1s-1 prior tokens to conditional on. Therefore, focusing reducing the
20

---

bias for small i. In addition, if i is small, we can iterate over the dataset and construct a histogram for the i-gram distribution p()):)).
We denote the count for each i-tuple of tokens with nn and the total number of samples with N. Then, the entropy of the distribution can be estimated naively
A mareo (Yas)
Tyyii log
log
Σ
log ylus
(B.4)
N
Misi
where the summation runs over all possible combination of tokens V1:1 == (V1 U2.... yi).
However, this estimation is severely biased and underestimates the entropy, due the concavity of logarithm function. In [102], bias-corrected estimator is proposed by replacing the logarithm function with new function
(Y1=)) = log N
Σ
yyn (ryyra)
(B.5)
N
where
G(n)= (()) + (-1)"
10
(B.6)
2
with (b()) the digamma function. We note that Ref. 102
was not able to obtain the closed form
expression for G(-), which we derived with the help of Wolfram Mathematica [OX]
PG19 (Books)
Wikipedia
10
Blpartte Mutuel Information
Biparite Information
10
Direct (without correction)
Direct (without correction)
Direct (with correction)
Direct (with correction)
Sequence Length (L)
Sequence Length (1)
Figure B.3: Effect of bias correction method in the direct estimator. The bias only affects the estimation at small sequence lengths, and all methods converge at large sequence lengths.
This bias-corrected estimator still underestimates the true entropy, but much less compared to the original naive estimator. In the main text, we estimate the (marginal) cross entropy with 2-gram correction in the following way. Breaking up the cross entropy
H
Σ Ep (YY//) H H (Pp),,;))))))
(B.7)
For the first term, ond term, we mitigate the bias from L.M estimation by combining it with Eq. (B.5) as H (pv Yr w/oss// )/5 + AY In Fig, B.3] we also present the result without this correction and show that this bías correction mostly affects the estimation small lengths L, and does not alter the general scaling behavior. In addition, since the result from this bias-corrected direct estimator agrees with the vCLUB 40) estimator, we believe this correction is reasonable.
LLM generated (334111:-11 wBs) approximation. For the sec-
B.V Additional Discussion Mutual Information Estimation Methods
In the main text, we briefly discussed the limitations of traditional mutual information estimation methods for sigessiieniinnl,, long-seoe sequenee setting. Here provide additional technical details on why these methods are challenging to apply to our settings.
Neural Estimators: MINE and InfoNCE. Neural estimators like MINE 37 and InfoNCE (100 train deep neural networks as critics estimate mutual information. Both methods can fundamentally be viewed training unnormalized density estimators density ratio estimators.
21

---

MINE uses the Konsker-Varadhan representation of KL divergence [04 and trains a critic (2 maximize:
v)
(ote,,))To(zz v)] - log (p())p()) (T
(B.8)
The optimal critic approximates the log density ratio log p(a)p(y) However, this objective suffers from high variance and numerical instability when mutual information is large, which is especially challenging given the high-dimensional nature long sequences we analyze.
p(a.))
InfoNCE uses noise-contrastive estimation with multiple negative samples:
T(X;Y) E log
(B.9)
プラ=ı (
where the expectation is over K' independent samples from the joint distribution. The bound is upper bounded by log K, which means for our setting where mutual information can be on the order of thousands, this would require prohibitively large batch sizes to obtain accurate estimates.
Both methods require training critics from scratch to learn representations of natural language distributions, which could require datasets and computational resources comparable to training L.LMs themselves.
Other variational bounds [36] face similar challenges.
K-Nearest Neighbor Estimators. K-nearest neighbor (K-NN) estimators [39] estimate mutual
information based on distances between samples in joint and marginal spaces. While asymptotically unbiased and training-free they also face challenges for text.
Text consists of discrete tokens that must be embedded into continuous spaces for K-NN estimation. Modern token embeddings have dimensions in the thousands, and for sequences of thousands of tokens, the combined dimensionality make K-NN estimation impractical as the number of samples required for reliable K-NN estimates grows exponentially with dimension.
Connection to Our LLM-Based Approach. Our approach leverages pre-trained LLMs den- sity estimators, providing g æ)) directly through conditional probabilities and approximating ((v) efficiently. This avoids training critics from scratch and the curse of dimensionality from distance- based estimation. We believe this well-suited for our use case of analyzing long natural language sequences.
B.VI Estimation of Two-Point Mutual Information
In the main text, we included the results for two-point mutual information for completeness. this section, we explain how the results are obtained.
Two-point mutual information estimation is more straightforward compared to bipartite mutual information, requiring only 1- and 2-gram statistics without LLM approximations. We estimate this quantity using entropy calculations for individual tokens and token pairs separated by distance d. Following [102), we employ their bias-reduced entropy estimator:
= =
Σ n'm (nem) m=11
(B.
N
where N is the total number of tokens, M is the vocabulary size, nm is the number of tokens whose token ID equals m, and G(.) is defined as
G(n) = ((n) (-1)"
71 + 1
(B.
2
5
with (b)-) the digamma function.
The entropy of pairs of tokens is estimated analogously, with the summation running all ordered pairs of tokens (m,, my). ). resulting in the total number terms quadratic in the vocabulary size. The mutual information is then estimated as
IT (x;))) =
- - (XY).
(B.12)
22

---

We note that this mutual information estimator exhibits systematic bias. The entropy estimator has a negative bias that increases (in absolute value) with dimension of the sample space [[2]. Since |021 = |Sty| = M where as |Ωxy| = 2 the bias in (XY) exceeds that in (X) = (Y). leading to a positive bias in la. This bias becomes particularly problematic at large distances d. where (XY) (X) + HY) and La approaches zero. To mitigate this issue, we perform additional bias correction by fitting the systematic bias from the data (see Appx. B.VII for details).
PG19 (Books)
Wikipedia
Two point Mutual Information
10
1.10- gf-.0
Two point Intormation
1.87 d-0.34
100
10"
10"
10"
10"
10"
10
10
10"
10"
10"
10'
10'
10"
10"
Distance (d)
Distance (d)
Figure B.4: Two-point mutual information scaling on PG19 and WIKIPEDIA datasets.
We apply this methodology measure two-point mutual information both the PG19 dataset and WIKIPEDIA, confirming power-law decay in both cases [Fig. B.4
B.VII Bias Correction for Two-point Mutual Information
As discussed previously, the estimation of two-point mutual information can be calculated directly using the n n-gram approximation [Eq. [B.3)]. and compute the two-point mutual information as
IT (X;(x) (x
(B.13)
As discussed in Appx. IVIV this entropy estimator has negative bias, whose magnitude depends on the ratio I//,, with (2 the size of the corresponding sample space. Since the sample space for the joint distribution is larger, it has a larger negative bias, resulting in positive bias in f. When d is small, this bias is relatively small compared to the mutual information itself. However, as d becomes larger, X and Y become less correlated, and (XY)) - H(X)= + HYY). In this case, the estimator can be dominated by this bias, and fitting for the power-law exponent becomes impossible.
To mitigate this issue, we propose bias-corrected estimator.
TTP corrected (x:)) =
- H (XY) C,
(B.14)
where C' is an unknown positive constant that does not depend on the distance d, which accounts for the bias of the original estimator.
To obtain this bias correction term and fit the power-law exponent, we minimize the following loss function
Σ (log (iTP C) - (log A - a d))
(B.15)
which is just iTP = Ad-a + C' fitted in log-log space. Then, we take the fitted C as the systematic bias and a as the fitted power-law exponent.
In Fig. B.5 we compare the corrected and uncorrected two-point mutual information as a function of d (only the corrected version is shown in the main text). Without the bias correction, the data appear to have larger long-range dependencies, but after the bias correction, all points lie straight line in a log-log plot. The bias correction constant is much smaller than the entropies involved in the calculation, even the smallest two-token entropy measured is 12.5, at least two orders of magnitude larger than the fitted bias correction. In addition, the fact that a single variable added to the fitting function can fit the data so well suggests the bias correction is reasonable and highly effective.
We note that on WIKIPEDIA, we were only able to measure the two-point mutual information up to d 256, due to limited long-context length data in WIKIPEDIA.
23

---

PG19 (Books)
Wikipedia
prinpoi Mutual Mutual Information
Information
10
10"
10"
10
1.10 d-o + 0.02
1.87 -877-d 177-d-000 0.12
d-0s1
--
10
uncoraaced
10
uncorrected
correctedd
corrected
10"
10
10"
Distenoe (d)
Dissance
Figure B.5: Effect of bias correction for two-point mutual information. The bias causes plateau at large distances.
V.VIII Failures of Two-point Mutual Information
In this section, we further demonstrate the relation between two-point and bipartite mutual information, and why two-point mutual information cannot properly capture the full multi-token dependencies needed for modeling natural language.
In existing literature, the scaling of two-point mutual information has been used to demonstrate the existence of long-range dependencies in natural language. In particular, it is believed that short- range dependencies are characterized by an exponential decay in two-point mutual information, as seen typically in finite-state Markov chains, and the existence of power-law decay in (wo-point mutual information in natural language indicates non-trivial long-range dependencies. Although this perspective is correct if one defines the existence of long-range dependence non-expe non-exxonennial-dee decay two-point mutual information, this definition does not properly account for the mutual information between token pairs when other tokens present. This can be made more clear by considering the following decomposition bipartite mutual information.
the existence of
For sequence of tokens Wi:L with X1.E W1.e and YELL- = We+1:L, the bipartite mutual information reads
EL I(X1;; VILLL)-
(B.16)
Standard information theory allows mutual information to be decomposed as
T(XZ;Y) = ((X;))(X)))))))) +(X;Y)+ ((ZY)())
(B.17)
where [(; X) is the conditional mutual information between & and given X. Using this relation repeatedly, the bipartite mutual information can be decomposed as
11 1(X (1
=((i; (K2; L-E
ΣΣ
(X
(B.18)
L-1
Σ I(X;))
ΣΣ'!! THe
In fact, it is in general not even clear whether the conditional mutual information Y, -1,, is greater or less than the marginal mutual information [Xx;; Yj). Therefore, as demonstrated here, when considering dependencies between blocks of text, simple aggregation of the two-point mutual information gives very incomplete picture. Due this reason, weakly cor- related systems, such as the example mentioned in S.c. could exhibit seeming strong long-range two-point dependencies, and systems with very different bipartite mutual information, could share very similar two-point information scaling, as we will show later in Appx.

---

C Multivariate Gaussian Distributions
In the main text, we considered two families of multivariate Gaussian distributions of different sequence lengths to demonstrate the distinction between bipartite and two-point mutual information scalings. particular, one designed mimic natural language, both in terms of the sub-volume law growth of the bipartite mutual information and the power-law decay of two-point mutual information. This family of distributions is also used empirically verify our theory L condition for different LLM architectures. The other is designed to have the same two-point mutual information scaling, but very different bipartite mutual information scaling, showcasing that one can have distributions with the same two-point mutual information scaling, but drastically different bipartite mutual information scalings.
C.1 Mutual Information Scalings
(a)
(b)
Sporttte Mutial Information
10
Sub-voluue Gaussian
Infomaaion
10"
sub-valume Gaussian
loganihhmic Gaussian
logarithmic Gaussiar
10
10
10
10
10
10"
10"
10
10
10"
10'
10"
10"
10"
Sequence Length (L)
Distance (d)
Figure C.6: Bipartite and two-point mutual information of the two families of Gaussian distributions.
Before showing the construction details, first present both the bipartite and two-point mutual information scalings of the two families of Gaussian distributions. As shown in Fig. C.6 they have drastically different bipartite mutual information scaling other logarithmic- their antipodal two-point mutual information scaling is identical. This further demonstrates that two-point mutual information alone gives incomplete information of the multi- token long-range dependencies present in sequence data, and the simple aggregation two-point mutual information does not tell the full picture of the bipartite mutual information. In fact, construct distributions with the same power-law decay in two-point mutual information, but has constant bipartite mutual information scaling well.
has power-law relation and the
C.11 Construction
Let's start by considering the family of distributions with sub-volume law growth. The distributions are constructed in a hierarchical manner.
We start at the first layer, with four independent standard Gaussian random variables (X1, X2, X3, X4). Then, define the change-of-coording matrix
7
M =
7
7
(C.19)
- 7
p
-
where we choose 7 v5/4 and p 1/4. The output of the first layer is defined
Y M
(C.20)
where the random variables are now correlated. It is easy to verify that this operation only changes the off-diagonal elements in the covariance matrix, and leaves the diagonal elements unaffected.
25

---

For the second layer and up, we first stack three independently sampled copies from the previous layer and attach additional independent standard Gaussian random variables the fourth elements as
Y1.1
Y1.2
V1.3
Wi
2211
Y2.
12,3
W2
X =
Y3,1
3,2
W3
(C.21)
where Yi, refers to the ith output from previous layer of the jth copy, and W; refers the ith newly sampled standard Gaussian random variable.
Note that this point, all coordinate matrix M at each row to correlate them. The matrix is then flattened to obtain Z
independent from each other; therefore we apply the change of
Z = 1,,,, 2,, Z1,,, Z1, Z2,1, 22,2, 22, 22, Z Za,,, Zz,3, Za,4.-..)
(C.22)
where the subscripts denote the variables' original position in the matrix. Before outputting from this layer, we perform addition operation to each pair of random variables (Zi,, Zi41,1), by applying a coordinate transformation that modifies their correlations as
corr Zi,4, Zi+1,)) corrr Z1,3, Zi,4) + corr corr(Z+11,1) 24112) 1
2
(C.23)
This operation may arbitrary, but it is crucial to introduce correlations that give linear ordering of the random variables. Without this operation, the distribution simply forms a tree structure.
Now, we can truncate the construction at different layers / and form family of distributions with different sequence lengths L = 4 In Fig. of the main text, we consider up to 8 layers, and in Fig. 3 and 4 we consider 1 = 4, 5 and, 6.
The second family of distributions is constructed analogously. The only difference is that we replace Eq. (C.21) with a single copy of Y and three independent copies of W as
Y1
W1.1
W1,
W1,3
Y2
W2.1
W2,2
W2,3
X=
=
Y3
W3,1
W3,2
3.3
(C.24)
:
C.III Properties
These two series of distributions have many nice properties, in addition to their bipartite and two- point mutual information scalings. In addition, these constructions directly defines the multi-variate probability distribution due to their Gaussian nature. This allows for exact calculations of conditional probability distributions for training LLMs, as well as direct computation of the bipartite and two- point mutual information without LLM approximations.
D Model State for Storing Past Information
In Definition 5.1 in the main text, give concrete definition of "model state for storing past information" as history state, and claim that is the past key-value pairs for transformers and recurrent state for SSMs and RNNs. Here, we explain them more detail.
D.I Transformers
In transformers, only the attention block mixes information among different tokens, therefore only need to analyze the behavior the attention block. We will be assuming the existence of the causal mask, as our theory mainly applied to autoregressive LLMs. Denoting the input and output of the attention layer as x and (notice they are no longer two parts sequence), the self-attention mechanism is defined
Baattmax((W,xx(((() Wa.,
(D.25)
26

---

where Wa W& and We are the weight matrices. For simplicity, we drop the usual vhaim normaliza- tion and the output weight matrix, as they are irrelevant to our discussion.
Separating the calculation for each token, the mechanism can be rewritten as
V/ = e (W
(W.)) + Ej=1 e(Woze)kuu
(D.26)
- e (W.:))Wa..)
e(W,z.)(Wez) + Σ3 WWo.)
where k; = Wkyt and Uj W are keys and values which sum over past tokens. Clearly, the attention output only depends on the current token z; and the past key-value pairs k1s-1 and 11:--1 This arguments extends all ye with k 2 i, where all yı 's dependency on F1s-1 is via k1--11 and -1-i-1 1- Therefore, vey-value pairs form the history state, and their size grows linearly with input sequence length. We note that Eq. (D266) also describes how key-value caching works.
D.II State Space Models and RNNs
State space models (SSMs) and RNNs, on the other hand, are casier to analyze. These models in general all have some recurrent state with fixed size, and some mechanism to update the state when a new token is observed. The output depends only on the previous recurrent state, and the current token. They can in general be written in the following way.
h; = 1,,,,
(D.27)
y/ = 0,;
for some update function f and output function g. It is obvious that the history state is exactly this recurrent state (or the collection of recurrent states of different layers), which does not grow with the input sequence.
We note that this discussion also applies linear attention models, whose key-value pairs can be merged into recurrent state with fixed size, due to the replacement of softmax function. Test time training (TTT) models can also be included in this discussion. They can be viewed RNNs with inner model parameters as recurrent state, and test time training process as update function.
D.III
Other Architectures
For other models, such sparse transformers or some compression-baset models, the analysis has to be performed separately. Nevertheless, the I.2M framework general: after identifying the history state, one can always compare its scaling with the bipartite mutual information scaling to see whether the model is capable of capturing the long-range dependencies in the data.
E Proofs of Theorem 5.2
We provide three proofs of Theorem 5.2 under different assumptions, demonstrating the universality and robustness of the result. Importantly, all three sets of assumptions are extremely mild and directly reflect realistic conditions in modern neural networks, whether through the discrete nature of floating-poin point arithmetic, empirically observed geometric properties of neural representations, or basic continuity requirements.
E.I Proof Under Discreteness Assumption
The discreteness assumption is already quite reasonable in practice. Modern neural networks use floating-point representations, which are inherently discrete with finite precision. Moreover, neural networks have been shown to retain strong performance demonstrating that discrete representations with limited precision are sufficient to capture the essential information. This discreteness assumption thus provides natural and practical starting point for our proof.
under aggressive quantization,
Theorem E.1 (Theorem 5.2 Discrete Version). Assume the history state takes discrete values.
Then a model's 's capacity to capture bipartite mutual information bounded by the size of its history state as
BP& < C - dim(zz) + log(M)
(E.28)
where C. is some constant and M denotes the vocabulary size.
27

---

Proof. By the data processing inequality: 19(X1:) YI-L-e) S ( This is upper bounded by the entropy: H (z))). which further upper bounded by (z HX;)), - C' dim((;) + log(M), where the last inequality follows from the bound on entropies of discrete variables.
E.II Proof Under Almost Orthogonal Directions (AOD) Assumption
The discreteness assumption can be relaxed we instead assume the following observed fact about neural networks: neural networks store distinct information in almost orthogonal directions (AODs) of the hidden state [17].
Theorem E.2 (Theorem 5.2, AOD Version). Assume neural networks store distinct information in almost orthogonal directions (AODs) of the hidden state. Then model's capacity capture bipartite mutual information is bounded as
BP&a E.L.
S C. -dim(zz)) + log(M)
(E.29)
where C' is some constant and M denotes the vocabulary size.
Proof. An autoregressive neural network's 's dependency on past tokens is through the intermediate variable 2z ( such that ((y (r) 4 dez ze). This can be viewed as the process X (z Xx) + Y. According to the data processing inequality,
(X ViL-e) S 19( Xe; Y1 - 719 Zt, Xx) S (Zt) (X)),,
(E.30)
where H to denote generalized notion of entropy which measures the amount of information that can be stored in Z₁ We only care about the scaling of H. so its exact definition is irrelevant to discussion.
Under the AOD assumption, neural networks store distinct information in almost orthogonal directions of the hidden state. Therefore, H should scale at most logarithmically with respect to the number of AODs as the state size increases. According to the Kabatjankki-Levenster
bound [T08] [09],
given error tolerance E, the number of AODs is upper bounded by exp(f(=) dim()z) for some function f that depends purely on the error threshold. Therefore, the generalized entropy scales as (Z( Zt) 3 log (E) dze) dim(zz). Since (X))) I log(M) where M is the vocabulary size, we conclude
BP < C dimi + log(M).
(E.31)
E.III
Proof Under Lipschitz Continuity Assumption
The theorem can also be proved assuming only certain Lipschitz continuity conditions on the neural network.
Theorem E.3 (Theorem 5.2
Lipschitz Version). Assume the history state mapping f :14-11 + zz f (1111)) < K # x'(e-1) and the neural network is entropy- satisfies (
Lipschitz, satisfying HY - H" 21)1 S KHllze 21ll2- Then a model's capacity to cap-
ture bipartite mutual information is bounded as
TEL < C dim(zz) + log(M)
(E.22)
where C. some constant and M denotes the vocabulary size.
Proof. We start with the data processing inequality and rewrite the bound as
(X VI.L-e & (" Xe; VI.L-e)
= =I9(Z,; Yu..-.) 1) + 19 (Xe; YI.z-e/Ze)
(E.33)
< (Z Y1.L-)) + log(M),
where the last inequality uses (X;;; VI..-o S H*(X)) S log(M) with M being the vocabu- lary size.
28

---

The history state is function of the imput tokens ž = f with 21--1 E (1,2.... -
Under our assumption on f. ze lives d-dimensional ball of radius Ky, where dim (ze)
Consider a quantization Q (zz) that maps each 2ς to the nearest point in an e-covering this ball. Then
[ 1 (Q(Z)); Y) + H ( ((Z)) - Zz))
(E.34)
By the entropy- Lipschitt assumption, (QQ))( (Y Ze) Σ KE Since Q( is discrete and takes at most (2Kx/e)² /e) values (by a covering number argument), we have ( Y) v (z()) d (olog(2) //))
Therefore, ( Z7; Y) I dlog(2K appropriately). This concludes
KHE S C d for some constant C (by choosing
BP e,T.
< C' d log(M) C dim(zz) + +lgg(M.
(E.35)
E.IV Discussion
We believe this theorem is more universal and can be proved in additional ways, such as by connecting it channel capacity and potentially showing 15 dlog(1 + SNR) log(MM). We also believe the theorem be established with more relaxed assumptions, similar how information dimension is proved to be the upper bound of lossless compression of continuous random variables [110]. However, additional proofs are heyond the scope of this work, and the three proofs provided should already be applicable in any practical settings.
F
Additional Details on Experimental Setup
In this section, we provide detailed information about our experimental setup, including dataset construction, model configurations, training procedures, and evaluation metrics.
EI
Synthetic Gaussian Distribution Dataset
For experiments on the multivariate Gaussian distribution, we use the sub-volume Gaussian distribu- tions described in Appx. C which exhibits power-law bipartite mutual information scaling with an exponent 0.79. To fully stress the LLMs, we stack 64 copies of the distribution and group the 64 Gaussian variables at each position to form single token. More specifically, example sample looks like
W (W1, W2, ..., WL)
(F.36)
((Z1,1 Z ..... Z1,61) (Z,,,, Z2,,. Z2,61) .... (ZL, ZL... ZL,,1),,
where the two subscripts (i,j) refer to the ith random variable from the jth copy. In this way, the bipartite mutual information matches better with natural language, not only scaling, but also in magnitude (multiplicative constant). We additionally prepend an all-zero token Wo to each sample to mimic the effect of the BOS token.
In order to process continuous random variables, we replace the embedding layers of GPT2 and Mamba(2) models with two-layer MLPs. For output, since all the conditional distributions also Gaussian, we use different two-layer MLP to output the 64 conditional means M Fos 1.5 standard deviations Ozz,,y 1o.--1.3
and
As discussed in Appx. C.IlI due to the analytical construction, the Gaussian distribution permits efficient calculation of conditional probabilities. Therefore, instead of simply training the neural networks with negative log likelihood samples alone, use the average conditional KL-divergence estimated as DEL (pllga)
64
T g 101-11 (Maz,, Bon-1ss 19z,, , 1Zon- )2
Epz
log
T
64
Op2,, Boon-1
20
i=1
92,, 1Bou-i
(F.37)
29

---

to reduce sampling variance.
F.I Natural Language Dataset (PG19)
For the PG19 dataset, train standard average negative log likelihood. We first split the dataset into samples with a length of approximately 1.2 times the target length, ensuring each sample starts the beginning of sentence. We then train the models for epochs (approximately 450k iterations) with a batch size of 16,384 tokens. To maintain consistency across different models, we always use the same tokenizer from GPT-Neo-X
E.III
Training Configuration
For the Gaussian distribution training, during each iteration, we use a batch size of (4 times sequence length number of tokens) with freshly generated samples, meaning therefore have effectively a single epoch, thanks to the "infinite" dataset size. We train all neural networks using the A optimizer and cosine decay scheduler with warmup. We use peak learning rate of x 10-5, a weight decay of 0.01, 2000 warmup steps, and 500,000 training steps in total. The results reported
reuse any sample. We
the end of training.
For the PG19 dataset experiments, we use similar hyperparameters: AdamW optimizer with a cosine decay scheduler with warmup, peak learning rate of 5 x 10 weight decay of 0.01, 2000 warmup steps, and 500,000 steps in total. The results reported are at the end training using a separate evaluation dataset containing 10,000 samples.
FIV Evaluation Metrics
In this paper, we report results on the position taste conditional KL-divergence
DKL,; Dx (Pw.)Wie- [gw,jwa.-)) = Ep [log
log ((W, Wh--i), (F.38)
average KL-divergence
DAVK T. Σ DKL.
(F.39)
and position-wise conditional NLL
NLL, Ep [log a (W(WW(n-))
(F.40)
One can also define average NLL
L
NL. avg
NLL;
(F.41)
L
which we use in Appx. G
FIV.1
Understanding the Behavior of KL--dvergence and NLL
It is important to understand how KL-divergence and NLL behave differently as token position increases. Using the relationship between cross-entropy and KL-divergence from Eq. (5). we can decompose the conditional NLL as
NLL, = DKL, + (",W,
(F.42)
where HIP(W, EE [1) is the conditional entropy of the true distribution at position
This decomposition reveals why KL-divergence and NLL exhibit opposite trends with token position. As token position increases, the conditional entropy Hº HW:W.WW..1) typically decreases because more context is available, making the next token more predictable. In natural language, this reflects that with more preceding text, there less uncertainty about what comes next. Meanwhile, conditional KL-divergence D often increases with position, because learning all long-range dependencies becomes more challenging, resulting worse model estimations compared the true conditional distribution.
30

---

For models with sufficient capacity relative to sequence length, the decrease in conditional entropy dominates, causing NLL decrease with position despite increasing KL-divergence. However, for models with insufficient capacity (such as fixed-state models at long sequence lengths), the Kl.-divergence can increase rapidly enough that NL.. plateaus even increases later positions. This behavior is precisely what we observe in our experiments with Mamba models in Fig. f where Mamba's NLL plateaus at later positions while transformers' NLL continues to improve.
The same reasoning applies to average quantities: NLL typically decreases with sequence length as the average conditional entropy decreases, while D may increase the model's ' capacity becomes insufficient relative to the growing sequence length.
When reporting conditional NLL, smooth the curves using small window around nearby tokens to reduce noise in the results.
FV Model Configurations
In Tables F.I and F.2, include the model configurations and sequence lengths for all experiments performed in this paper.
Table F.1: Models and configurations for synthetic dataset experiments.
Model
num hidden layers
hidden size
seq len
GPT2
12
768
256, 1024,4096
GPT2-medium
24
1024
256, 1024,4096
GPT2-large
36
1280
256, (024,4096
Mamba-50m
12
512
64,256, 1099,
Mamba-70m
24
512
64,256, 1024,4096,1638
Mamba- 130m
24
768
64,256, 1024,4096,10 16384
48
1024
64,256, 1024,4096
Mamba-790m
48
1536
64,256, 1024,4096
Mamba-1.4 1..4b
48
2048
64,256, 1024,4098
Mamba2-1 --30m
24
768
256, 1024,4096
Mamba2-3745 2m0m
48
1024
102,1024, 1024,4096
Mamba2-790m
48
1536
256, 1024,4096
Table F.2: Models and configurations for PG experiments.
Model
num num_hidden Jayers
hidden size
seq_len
GPT2
12
768
4096,8192
GPT2-medium
24
1024
4096,8192
GPT2-large
36
1280
4096,8192
Mamba- 130m
24
768
4096,8192
Mamba-3 370mm
48
1024
4096,8192
Mamba-790m
48
1536
4096,8192
E.VI
Computational Resources and Implementation Details
Our experiments are performed primarily on H100 GPUs, with varying VRAM sizes between 80GB and 96GB. Some experiments use A100 GPUs with 80GB VRAM instead. We use the vLLM library 668 when running inference to estimate the mutual information scaling. For both LLaMA 3.1 405B and DeepSeek V3, run the FP8 version using 8 H100 GPUs (with 96GB VRAM each). The model weights and configurations are downloaded from HuggingFace 112
When training GPT and Mamba(2) models on the Gaussian distribution, we use our custom library developed PyTorch [113). When training GPT and Mamba models on the PG19 dataset, we use the trainer from the HuggingFace transformert library. All models are initialized from scratch, with model configurations taken from HuggingFace. All training experiments are performed on individual H100 and A100 GPUs with FP32 precision to avoid possible training failures. Although
31

---

training with FPI6 would make the experiments run faster, it should not affect the actual results. We note that for Mamba2, we use the official implementation instead the HuggingFace version. For the GPT2 experiments on PG19, we re-implement the attention mechanism with Flex Attention [114] to save memory, as the official FlashAttention 664 66
does not support FP32 precision.
VVIII Code Availability
The code for reproducing our mutual information estimation and the PG19 results is available at https:/ gi thub. woaaboooo////uued///.. aattaarrdddmuuuuaaaaaaaa inl_iino_s_aca Seroo_aallnnn__eeeeeeee law
G Additional Experimental Results
(a)
(b)
XL (LDeeggence
10"
10"
2212 (125m)
got2-modium
get2-large (774m)
10"
16"
mamrss--11 1:30m
mamba- 100m
mamti 790m
mamnas- 7700m
mamba-370mm
10
10
Suqueroe Lungth
Sequence Length
(c)
(d)
Cardiine (t-Diampezz
1o"
9922 (125m)
Condioonal KL Ohargenze
mambe-188 150m
10
ger2-madlum (165m))
gpr2-large
10
mamba-750m
mamba-370m
00"
"02
70
00
10
10
10
10
16
(e)
Trien Postiio
(f)
Token Postion
Coneioont Divergnne
go12 (125m))
Comfiooo Ho. Divegenns
mambe 130m
10"1
9212 modium (365m)
gat2-arge (774m)
10
mambar 370m
mamba 750mm
10
10
10""
10
10
10
10'
10"
10"
(g)
Token Posttion
(h)
Token Fosition
Carditona Ei-eeeggrrro
To²
gu22 (125m)
Condioonal Shorqunce
mamba-1300 130m
70"
spo2-large (774m)
10²
mamba 780m
mamba-470m
10
10
10"
10"
10
10' Tokan Posilion
10"
10"
10'
10'
10'
Toleon Position
Figure G.7: Evaluation of KL-divergence across model architectures trained on sub-volume Gaussian distributions. (a. b) Average KL-divergence per token for models trained on different sequence lengths [same as Fig. (a, b)]. (c, d) Position-wise conditional KL-divergence for models trained on sequence length 256. (e, f) Position-wise conditional KL.-divergence for models trained on sequence length 1024. (g. h) Position-wis conditional KL-divergence for models trained sequence length 4096 [same as Fig. 3 (c. d)). Lower values indicate better performance.
In this section, we show additional experimental results. In Fig. G./ we include positional-wise conditional KL-divergences of models trained on sub-volume Gaussian distributions with sequence length 256 (c. d), 1024 (e. f), and 4096 (g, h). As clearly demonstrated in the figure, for short sequence lengths, Mamba maintains similar performances to GPT2; Mamba models of different sizes also appear to have smaller performance gap. However, as go to longer sequence lengths,
32

---

smaller Mamba models starts to fail, while GPT2 always maintain relatively stable performances, consistent with our theory.
72
sp12 (125m)
apt2-medium (165m))
Megetiv Log Looe
72
mambe-133m mamba-33/mm
71
gpiz-arga (774m)
71
manba2:11 150m
nomba-790m
Megativa Log
70
70
256
1024
4086
256
1024
4096
Sequence Length
Sequence Length
spi2 (12bm)
4pt2-meddimm (385m)
mamba 70m
gpiz-arge (776m)
mamba-1 130m
-80
Condsone
20
10" Tokan Postiion
10
10"
Token Position
Figure G.8: Negative log likelihood (NLL) across model architectures trained on sub-volume Gaussian distributions (a, b) Average NLL per token for models trained on different sequence lengths. (c, d) Position-wise conditional NLLL for models trained sequence length 4096. Lower values indicate better performance.
In Fig. G.8 we show the negative log likelihood (NLL) of models trained on sub-volume Gaussian distributions. We note that, because NLL combines the KL-divergence with the intrinsic entropy of the underlying distribution (the average and position-wise conditional of which decays as sequence lengths), the differences between model performances are less visible. It's worth noting that, since Gaussian random variables are continuous, NLL values can differ by an arbitrary additive constant by rescaling the distribution. Therefore, the exact values of conditional NLL do not carry intrinsic meaning, though relative comparisons (which is exactly the same as the KL--divergence) between models remain valid.
In Fig. G.9 we show the position-wise conditional negative log likelihood (NLL) of models trained on the PG19 dataset [90] with 4096-token sequences. The results here is consistent with the 8192- token-sequence results in the main text.
H Limitations
Our theoretical framework specifically examines models' capacity to capture long-range dependencies through the lens of bipartite mutual information and does not address other aspects of language modeling, such as reasoning capabilities or world knowledge. The L'M condition establishes necessary but not sufficient conditions for effective long-context modeling. Understanding how this theoretical capacity translates to actual downstream task performance remains an important open question. The relationship likely depends on additional factors including optimization dynamics, architectural inductive biases, and task-specific requirements. Systematic evaluation across diverse long-context benchmarks represents crucial next step to clarify these relationships and identify any gaps between theoretical capability and practical performance.
While our empirical validation on synthetic Gaussian distributions with controlled mutual information scaling provides clean verification of the theoretical predictions, it may not capture all complexities present in natural language. Our theory focuses on autoregressive language models, which remains broadly applicable as even diffusion-based approaches typically employ autoregressive generation for extended sequences in practice. Nonetheless, exploring whether similar information-theored principles govern fundamentally different generative paradigms or multimodal models represents an interesting direction for future work.
33

---

3.4
gpt2 (125m)
mamba 130m
3.4
mamba 370m
gptz-medium (355m)
3.4
mamba-790m
gpt2-lagge (774m)
Conditionl NLL
Canditoral NLL
Canditonal NLL
3.2
3.2
3.0
3.0
3.0
2.8
2.8
2.6
10"
10"
10"
10'
10"
10"
Position
Token
Figure G.9: Position-wise conditional negative log likelihood (NLL) evaluation for models trained on 4096-token sequences the PG19 dataset [90
The methodology we employ, using LLMs as density estimators for mutual information measurement, represents practical approach given the severe challenges of nigh-dimensional estimation in long sequences. Alternative methods like K-NN and neural estimators face fundamental difficulties with dimensionality and sequence length. While our approach yields consistent power-law behavior across different models and estimators, both methods likely underestimate the true exponent, and developing more accurate estimation techniques remains important challenge.
Our evaluations rely primarily on open-source models; further verification using state-of-the-art closed-soune models would provide additional validation.
I Broader Impact
This work advances our theoretical understanding of how language models process long-range dependencies, with implications for the design and deployment of more efficient L.M architec- tures. By establishing the I.2M condition, we provide principled framework for evaluating an architecture's fundamental capacity for long-context modeling. This could lead more efficient models that maintain effectiveness while reducing computational resources, potentially decreasing the environmental footprint of training and inference. Our findings may influence the development of specialized architectures for tasks requiring long-context understanding, such as legal document analysis, scientific research, and complex reasoning.
However, improved the propagation of bias over longer contexts and enhanced capabilities for generating persuasive misinformation. Research applying the I.2M framework should consider these ethical dimensions, particularly how improvements in long-range dependency modeling might affect model safety, fairness, and the verifiability of model outputs.
modeling could also amplify existing challenges in LLMs, including
34

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes].
Justification:
All claims are supported by either theoretical or experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
The limitations are discussed in Appx. H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
35

---

All the proofs are included in either the main text or in Appendix, together with the necessary assumptions.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]..
Justification: All the experimental details are described in the appendix, with main code provided at
ctwm://Squareeehh/mttaaaaaaaaaaa si___nnfo_s_ooo sacos_sallii oocaling__e law
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification:
The main code is at info_scalinne ing__aw
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ public/guides CodeSSumiasiiioPPooo.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips cttpccppppcccccccc http:pulllccceeee wwtcc//guiss//see
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: the details are described the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No].
Justification: For the measurement of mutual information scaling, the error bar is small to see on the figure, so we omit it when drawing the figures. For the experimental results, it is too costly train multiple neural networks for each setup. Nevertheless, our experimental results are very clean and consistent with our theory predictions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]. All the compute resources used are described in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes].
Justification: Everything conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
The broader impact discussed in Appx.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We cite all the datasets and models used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, wwwprswi thcode. com/ con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

The synthetic dataset generation process is For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes].
Justification: The main code used in the paper is posted at httts:///i com/ gaapuared//mutuse info wetooossaalnnnnngngeeeeee law fully described in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
40

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanstandard components.
Please refer to our LLM policy (https: for what should or should not be described.
/ /ttp:/awiii.ccooooofffrrrrr/////
41


Reviews:
[{"review_id": "HUTTHMYacP", "reviewer": "Reviewer_jrcZ", "summary": "This paper is concerned with scaling laws for long-context language modeling and specifically develops a seemingly novel information measure that ends up having an empirical power-law characterization.  This is what they call the bipartite mutual information, and is the standard Shannon mutual information but over long disjoint blocks.  An implementable (suboptimal/biased) estimator for this quantity is developed through the use of LLMs.  A condition on the state space requirements for LLMs is developed using the scaling law, and used to analyze Transformers and state-space models.", "questions": "If one looked at some other corpora, e.g. proteins, DNA sequences, or computer code, would the approach in this paper help clarify the distinctions between the domains, e.g. why smaller models seem to be okay for biological sequences than natural language?  \n\n\nAuthors say that theoretical understanding of neural scaling laws remains limited, but they might be interested in work such as Nayak and Varshney \"An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models\" (NeurIPS 2024 workshop) that actually uses information-theoretic principles.  See also references thereto.  I would be curious to know if there are connections between scaling laws for language as in the present work and the information-theoretic explanation of neural scaling laws.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "STRENGTHS\nThe empirical finding (assuming the bipartite mutual information is well-estimated) of the scaling law is quite intriguing, in some sense moreso for computational linguistics than for machine learning.\n\nThe implication of the predictability characterization of natural language on the state space requirements is insightful.\n\nWEAKNESSES\nI'm not convinced that the estimator for bipartite mutual information developed and used in this work is the best it could be. Appendix A.IV is an improvement on [91] but doesn't draw on the huge literature on mutual information estimation that includes unbiasedness and optimality results.  \n\nThere does not seem to be much connection to the LLM-as-universal predictor literature, which might help clarify some things on the state space requirement derivation.  Indeed, the universal prediction literature in information theory is often concerned with state space size.  \"Transformers Learn to Compress Variable-order Markov Chains in-Context\" and \"Transformers are Universal Predictors\" are two papers I could find.\n\nThe fact there are really three versions of Theorem 5.2 is not mentioned in the main paper, this could be clarified.", "was_revised": true, "final_justification": "This is a good paper, and hope it is accepted.  The authors have addressed many of the comments of all the reviewers.", "rebuttal": ">**Strengths And Weaknesses:**\n\n>STRENGTHS The empirical finding (assuming the bipartite mutual information is well-estimated) of the scaling law is quite intriguing, in some sense moreso for computational linguistics than for machine learning.\n\n>The implication of the predictability characterization of natural language on the state space requirements is insightful.\n\nWe thank the reviewer for recognizing that our bipartite mutual information scaling law is both intriguing and impactful for computational linguistics, and that the L²M condition offers valuable insight into the latent state requirements of language models.\n\n>WEAKNESSES I'm not convinced that the estimator for bipartite mutual information developed and used in this work is the best it could be. Appendix A.IV is an improvement on [91] but doesn't draw on the huge literature on mutual information estimation that includes unbiasedness and optimality results.\n\nWe agree with the reviewer that our estimator is not perfect, including its tendency to slightly underestimate the exponent. We tried many methods for estimating mutual information scaling, but found most suffer from more severe issues. \n\nIn particular, estimators like K-nearest neighbor (KNN), while asymptotically unbiased, struggle severely with the high dimensionality of our task. In addition, since KNN only works for continuous random variables, tokens must be embedded into latent vectors first, which creates a significant tradeoff between choosing higher embedding dimensions for better representation versus lower dimensions where KNN works better. Neural estimators like MINE and InfoNCE face great difficulty in training neural networks on the vast distribution of long-sequence natural language. We suspect training such neural estimators is no less difficult than training LLMs themselves, since they must learn good representations of natural language correlations to provide meaningful mutual information estimates. Additionally, all other methods we tested exibits rapidly increasing error as sequence length grows (due to increasing dimensionality), either from inherent issues with high dimensionality (KNN) or increased difficulty in training neural networks (MINE and InfoNCE).\n\nWe ultimately found vCLUB suffers the least from these problems, particularly due to its ability to leverage existing LLMs as the variational distribution. The estimated mutual information also doesn't suffer from increased dimensionality as sequence length increases. To validate our results, we also proposed the direct estimator, and cross checked scalings using different LLMs, all of which achieved consistent results, demonstrating the effectiveness of our approach. While the methods aren't perfect, our reported measurements represent the best currently achievable, and are reasonably good for our purpose as a first meaningful attempt to measure the mutual information scaling using LLMs. We would be happy to learn any other suggestions for more advanced estimation methods to refine these measurements in future work.\n\nWe will update the paper to include a more detailed discussion of alternative estimation methods we considered and their limitations, to further contextualize our methodological approach within the broader literature on mutual information estimation.\n\n>There does not seem to be much connection to the LLM-as-universal predictor literature, which might help clarify some things on the state space requirement derivation. Indeed, the universal prediction literature in information theory is often concerned with state space size. \"Transformers Learn to Compress Variable-order Markov Chains in-Context\" and \"Transformers are Universal Predictors\" are two papers I could find.\n\nThere is indeed a real but complementary connection. Works like \"Transformers Learn to Compress Variable-order Markov Chains in-Context\" and \"Transformers are Universal Predictors\" demonstrate that transformers can, in principle, approximate arbitrary variable-order Markov processes, establishing their universality in prediction. Our contribution is orthogonal but complementary: we identify the minimal latent state growth rate required to capture the empirically observed bipartite mutual information scaling in natural language. While universal predictor results show what is possible in principle, our L²M condition specifies what is necessary in practice to match the scaling structure of natural language. \n\nWe will include in the updated paper discussions on the connection to universal prediction literature and how our work complements these theoretical capabilities with empirical requirements.\n\n>The fact there are really three versions of Theorem 5.2 is not mentioned in the main paper, this could be clarified.\n\nWe thank the reviewer for this suggestion. Theorem 5.2 indeed has three versions under different assumptions. We will update the main text to explicitly note this and will separate the variants with their assumptions into different theorems in the appendix for clarity.\n\n>**Questions:**\n\n>If one looked at some other corpora, e.g. proteins, DNA sequences, or computer code, would the approach in this paper help clarify the distinctions between the domains, e.g. why smaller models seem to be okay for biological sequences than natural language?\n\nOur method is not restricted to natural language and can, in principle, be applied to any sequential domain. If biological sequences like proteins or DNA exhibit smaller scaling exponents $\\beta$, or exhibit logarithmic or even slower mutual information scaling, the L²M condition predicts correspondingly smaller latent state requirements, which may explain why smaller models suffice in practice. Conversely, domains like code, where $\\beta$ is closer to natural language, may require larger latent states. These are interesting and meaningful future directions worth further exploration.\n\nWe will add a brief discussion in the conclusion about the potential application of our framework to other sequential domains.\n\n>Authors say that theoretical understanding of neural scaling laws remains limited, but they might be interested in work such as Nayak and Varshney \"An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models\" (NeurIPS 2024 workshop) that actually uses information-theoretic principles. See also references thereto. I would be curious to know if there are connections between scaling laws for language as in the present work and the information-theoretic explanation of neural scaling laws.\n\nWe thank the reviewer for bringing up this paper. Their work explains compute-optimal scaling using information-theoretic principles and provides valuable insights into neural scaling laws. Our framework complements their approach: while they focus on how compute and model size must scale for optimal training, our L²M condition establishes how latent states must scale to satisfy the predictive information structure of the data. \n\nWe will include discussion of this work and its connections to our framework in Section 2 and discuss how both approaches use information theory but address different aspects of scaling.\n\n>**Paper Formatting Concerns:**\n\n>References don't seem to be capitalized correctly.\n\nWe thank the reviewer for noticing this and will fix it in the updated version.\n\n>**Rating:**  4: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.\n\nWe thank you again for your constructive feedback. We hope our responses and planned revisions have fully addressed your concerns. We would be happy to answer any further questions and kindly ask that you consider these clarifications in your final evaluation."}, {"review_id": "CwCF7DCfh8", "reviewer": "Reviewer_cZLb", "summary": "This paper proposes and empirically substantiates a scaling law for bipartite mutual information in natural language, arguing that such a measure more comprehensively captures the long-range dependencies critical for long-context language modeling than traditional two-point (token-to-token) mutual information. Using empirical analyses with LLMs and diverse datasets, the authors demonstrate that bipartite mutual information scales as a power law. From this, they formalize the \"Long-context Language Modeling (L²M) condition,\" which prescribes that the latent state size of a model must scale at least as rapidly as the bipartite mutual information with respect to context length to effectively store predictive information. Theoretical results and extensive empirical studies are provided to validate these claims and elucidate architecture-specific implications for transformers and state-space models.", "questions": "* Can the framework be generalized or adapted for hybrid models (hybrid transformer and state-space model) or even other generative paradigms (e.g., diffusion models), or will fundamentally new information scaling laws be needed in those domains?\n* Are there empirical indications showing that architectures meeting the L²M condition perform better in practical long-context tasks (e.g., document QA) compared to those that violate it? What might be the gaps between theoretical MI scaling capability and actual task performance?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Clear Theoretical Innovation: The manuscript introduces a rigorous, information-theoretically grounded scaling law for bipartite mutual information in natural language sequences. By establishing the distinction and practical importance of bipartite over two-point mutual information, the authors provide a novel lens through which to view long-range dependencies.\n\n* Empirical Verification: The claims are validated both on synthetic datasets and on real-world data. The L²M condition can give actionable guidance for model designers, linking information scaling properties with minimal latent state requirements.\n\n\n**Weaknesses**\n\n* Bipartite MI Estimation: While two estimators (direct, vCLUB) are used, the paper acknowledges that both likely underestimate the true scaling exponent β, yet the quantitative effect remains unclear.  The reliability of exponent fits completely resolved.\n* It appears that the appendix is missing.", "was_revised": true, "final_justification": "Overall, I believe this paper can provide a valuable theoretical supplement to the field of long context modeling. However, its integration with the downstream performance of modern PLMs with large context window in real-world scenarios remains insufficient, and thus it cannot offer direct insights for now. Therefore, I will maintain my score (borderline accept).", "rebuttal": ">**Strengths And Weaknesses:**\n\n>**Strengths**\n\n>-   Clear Theoretical Innovation: The manuscript introduces a rigorous, information-theoretically grounded scaling law for bipartite mutual information in natural language sequences. By establishing the distinction and practical importance of bipartite over two-point mutual information, the authors provide a novel lens through which to view long-range dependencies.\n   \n>-   Empirical Verification: The claims are validated both on synthetic datasets and on real-world data. The L²M condition can give actionable guidance for model designers, linking information scaling properties with minimal latent state requirements.\n\nWe deeply appreciate the reviewer's recognition of our theoretical innovation and empirical validation. \n\n>**Weaknesses**\n\n>-   Bipartite MI Estimation: While two estimators (direct, vCLUB) are used, the paper acknowledges that both likely underestimate the true scaling exponent β, yet the quantitative effect remains unclear. The reliability of exponent fits completely resolved.\n\nWe acknowledge that both estimators likely underestimate the true scaling exponent $\\beta$. While we tried many alternative methods, each had significant limitations in our high-dimensional, long-sequence setting. \n\nIn particular, K-nearest neighbor (KNN) estimators, while asymptotically unbiased, struggle severely with the high dimensionality of our task. In addition, since KNN only works for continuous random variables, tokens must be embedded into latent vectors first, which creates a significant tradeoff between choosing higher embedding dimensions for better representation versus lower dimensions where KNN works better. Neural estimators like MINE and InfoNCE face great difficulty in training neural networks on the vast distribution of long-sequence natural language. We suspect training such neural estimators is no less difficult than training LLMs themselves, since they must learn good representations of natural language correlations to provide meaningful mutual information estimates. Additionally, all other methods we tested exibits rapidly increasing error as sequence length grows (due to increasing dimensionality), either from inherent issues with high dimensionality (KNN) or increased difficulty in training neural networks (MINE and InfoNCE). \n\nWe ultimately found vCLUB suffers the least from these problems, particularly due to its ability to leverage existing LLMs as the variational distribution. The estimated mutual information also doesn't suffer from increased dimensionality as sequence length increases. To validate our results, we also proposed the direct estimator, and cross checked scalings using different LLMs, all of which achieved consistent results, demonstrating the effectiveness of our approach. While the methods aren't perfect, our reported measurements represent the best currently achievable, and are reasonably good for our purpose as a first meaningful attempt to measure the mutual information scaling using LLMs. We would be happy to learn any other suggestions for more advanced estimation methods to refine these measurements in future work.\n\nWe will update the paper to include a more detailed discussion of the challenges in estimating the mutual information scaling, and further clarify the limitations of our and other methods to further contextualize our methodological approach within the broader literature on mutual information estimation.\n\n>-   It appears that the appendix is missing.\n\nWe apologize for the confusion. The appendix was included in the supplementary materials. We will ensure it is properly integrated into the main submission in the updated version.\n\n>**Questions:**\n\n>-   Can the framework be generalized or adapted for hybrid models (hybrid transformer and state-space model) or even other generative paradigms (e.g., diffusion models), or will fundamentally new information scaling laws be needed in those domains?\n\nWe thank the reviewer for the question. The L²M condition is architecture-agnostic. Hybrid transformer-state space models can satisfy the condition as long as their effective history state scales faster than $L^\\beta$. For diffusion-based language models, while the notion of \"history state\" becomes less clear for the diffusion process itself, these models often still generate text autoregressively when producing extended long sequences. Our framework remains applicable at the scale where autoregressive generation is necessary.\n\nWe will add a discussion about the applicability of our framework to hybrid architectures and other generative paradigms.\n\n>-   Are there empirical indications showing that architectures meeting the L²M condition perform better in practical long-context tasks (e.g., document QA) compared to those that violate it? What might be the gaps between theoretical MI scaling capability and actual task performance?\n\nOur current results suggest that satisfying the L²M condition correlates with stronger performance in long-context tasks. However, there remains an interesting gap between theoretical MI capability and actual downstream performance. This gap likely arises from optimization dynamics, inductive biases, and task-specific effects. Closing this gap through systematic evaluation on diverse long-context benchmarks would be valuable future work. In addition, while different architectures can meet the same L²M condition, the scaling exponent for different architectures can be different, which may also affect the downstream task performance, and is worth further exploration in future works.\n\nWe will update the paper to add a paragraph discussing potential future works along this direction.\n\n>**Rating:**  4: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.\n\nWe thank you again for your constructive feedback. We hope our responses and planned revisions have fully addressed your concerns. We would be happy to answer any further questions and kindly ask that you consider these clarifications in your final evaluation."}, {"review_id": "cPjDrURfXC", "reviewer": "Reviewer_rSLc", "summary": "The authors study the long-range dependencies of natural language for large language models. More precisely, they establish bipartite mutual information scaling laws (as power laws) that explain multi-token interactions beyond the prior understanding given by two-point mutual information. They explain how to estimate these scaling laws with large language models and demonstrate their validity with GPT2 and Mamba(2) models on the PG-19 dataset. The authors use these scaling laws to provide a condition, the $\\text{L}^2\\text{M}$ condition, that defines the minimum size of the latent space of a model to be able to deal with long-context dependencies. They validate this condition experimentally with GPT-2 and Mamba(2) models on synthetic data, showing that transformer-based models naturally satisfy the $\\text{L}^2\\text{M}$ condition while SSM need to scale their latent space size with the sequence length to be able to satisfy it.", "questions": "1) In the proof of Thm. 5.5, line 281, shouldn't it be \"$I^{BP,q}$\" instead of \"$I^{BP}$\"? Otherwise, it leads to comparing the same quantity in front and after the inequality sign.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n- The i sdeaeems novel, the paper is well written, and the problem tackled is well motivated\n- I find the theoretical arguments compelling\n- The experiments are comprehensive\n- The connection between theory and experiments is done properly\n\n*Weaknesses*\n\nI list below what I believe are weaknesses, but I would be happy to be corrected if I misunderstood some important parts of the current submission.\n1) The takeaway of the paper is not clear to me: indeed, transformer-based models always validate the LM condition because the key-value pairs scale linearly with the sequence length. Given that current SOTA models are transformer-based ones, how does the current paper improve our understanding of such models?\n2) Related to the point above, to me, the author's proposal rather highlights the current limitations of SSM-based LLMs, whose sizes need to scale with the sequence length to be able to model long-context dependencies. As such, it is not clear how useful the proposed analysis and the LM condition are beyond \"SSMs might suffer more from long-context\". Could the author please elaborate on that?\n\n*Reason for my current score*: Overall, I find the paper interesting and novel, with a well-conducted theoretical study with empirical verifications. However, the main message and the impact of the paper for the design, study, or use of LLMs are not clear to me. Notably, long context is indeed an issue, but we have seen increasing context windows in the recent open-source LLMs like the Llama, Gemma, Qwen, or Deepseek families. How does the proposed approach improve long-context abilities or our understanding of them beyond state-space models? These are the reasons why I lean towards a borderline acceptance, but I am willing to change my score provided the authors correctly address my concerns.", "was_revised": true, "final_justification": "The current work is interesting, and the authors addressed my concerns during rebuttal, except one related to the practical impact of the paper (this is not an experimental work but more in terms of how it could help design better LLMs using the current submission). I am not convinced how the current work could enable scaling up to millions or billions of tokens (which was claimed by the authors to explain the potential impact of their work), and I think that this claim is not supported in the current submission (even for future work building upon your work). The practical benefits of the current work are also unclear to other reviewers. As such, I will maintain my score (which leans towards acceptance).", "rebuttal": ">**Strengths And Weaknesses:**\n\n>_Strengths_\n\n>-   The idea seems novel, the paper is well written, and the problem tackled is well motivated\n>-   I find the theoretical arguments compelling\n>-   The experiments are comprehensive\n>-   The connection between theory and experiments is done properly\n\nWe sincerely thank the reviewer for finding our work novel, well-written, and properly motivated. We appreciate the recognition of our compelling theoretical arguments and comprehensive experiments.\n\n>_Weaknesses_\n\n>I list below what I believe are weaknesses, but I would be happy to be corrected if I misunderstood some important parts of the current submission.\n\n>1.  The takeaway of the paper is not clear to me: indeed, transformer-based models always validate the L²M condition because the key-value pairs scale linearly with the sequence length. Given that current SOTA models are transformer-based ones, how does the current paper improve our understanding of such models?\n\n>2.  Related to the point above, to me, the author's proposal rather highlights the current limitations of SSM-based LLMs, whose sizes need to scale with the sequence length to be able to model long-context dependencies. As such, it is not clear how useful the proposed analysis and the L²M condition are beyond \"SSMs might suffer more from long-context\". Could the author please elaborate on that?\n\nWe thank the reviewer for these important questions that allow us to clarify our contributions. Our main contribution is not simply to observe that transformers satisfy the L²M condition, but to establish L²M as a **necessary and universal lower bound** on latent state growth for *any* architecture to model the long-range dependencies observed in natural language.\n\nThis perspective provides two key insights. First, it helps to reframe the challenges faced by architectures like SSMs. Our work suggests their struggle with long contexts is less about specific implementation details and more about a necessary architectural property we identify. The L²M condition requires the latent state to scale as $L^\\beta$, posing a clear **design challenge** for any architecture, such as standard SSMs, that aims to use a fixed-size state. Second, and more importantly, it reveals that transformers, with their linearly growing key-value caches, actually **over-provision** latent state, since our empirical results show that only a sublinear growth ($L^\\beta$ with $\\beta < 1$) is required for natural language. \n\nThis gap between the *necessary* sublinear scaling and the transformer's *actual* linear scaling opens an exciting avenue for future research: designing novel architectures (which could include sparse transformers or hierarchical transformers) that are more efficient than vanilla transformers (i.e., with sub-quadratic computational cost) while still precisely meeting the L²M requirement. Our work provides a concrete, data-driven target for this architectural innovation.\n\nWe will revise the introduction and conclusion to better emphasize these implications for future architecture design beyond just explaining current limitations.\n\n>**Questions:**\n\n>1.  In the proof of Thm. 5.5, line 281, shouldn't it be \"\" instead of \"\"? Otherwise, it leads to comparing the same quantity in front and after the inequality sign.\n\nThe reviewer is correct. We thank the reviewer for catching this error and will correct it in the updated version.\n\n>**Rating:**  4: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.\n\nWe thank you again for your constructive feedback. We hope our responses and planned revisions have fully addressed your concerns. We would be happy to answer any further questions and kindly ask that you consider these clarifications in your final evaluation."}, {"review_id": "S02XaqtCiG", "reviewer": "Reviewer_gdYy", "summary": "In this paper, the authors propose to depict the dependencies crucial for modeling long sequences by bipartite mutual information. Though comprehensive experiments, they establish a power-law scaling of bipartite mutual information across diverse natural language datasets using LLMs. With this observation, they introduce the concept of long-context language modeling condition and prove that the model's state size should outpace the scaling of bipartite mutual information to achieve effective long-context modeling. This finding is verified on both transformer and state-space models.", "questions": "1. In line 258, the authors claim that bipartite mutual information reaches its maximum for a given $L$ when $\\ell=L/2$. It is encouraged to provide more details to explain this result.\n2. Estimating mutual information is challenging, particularly for high dimension random variables. What is the error of the mutual information estimation method used in this paper? Does it increase with dimensionality?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work in a separate section after conclusion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The problem studied in this paper is significant and central, that is, the neural scaling law of LLMs. By conducting comprehensive experiments on recent state-of-the art LLMs, the authors establish a power-law scaling of bipartite mutual information for the first time, which is crucial to depict the long-range dependencies in natural language.\n- Based on the above observation, the authors further theoretically demonstrate that the history state dimension must grow at least as fast as the power-law scaling of bipartite mutual information in the data, which brings new insights for community to design or improve the network architecture of LLMs.\n\n**Weakness**\n- Overall, this is an interesting and solid work from my perspective. The possible weakness could be that the authors only analyze the model's history state size but do not establish a relationship between bipartite mutual information and the model's expressive power or generalization capability. This could be a potential direction for future research.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns in the rebuttal. From my perspective, this paper is an interesting and solid work. I fully agree to accept this paper.", "rebuttal": ">**Strengths And Weaknesses:**\n\n>**Strengths**\n\n>-   The problem studied in this paper is significant and central, that is, the neural scaling law of LLMs. By conducting comprehensive experiments on recent state-of-the art LLMs, the authors establish a power-law scaling of bipartite mutual information for the first time, which is crucial to depict the long-range dependencies in natural language.\n>-   Based on the above observation, the authors further theoretically demonstrate that the history state dimension must grow at least as fast as the power-law scaling of bipartite mutual information in the data, which brings new insights for community to design or improve the network architecture of LLMs.\n\nWe are grateful for the reviewer's assessment that our work addresses a significant and central problem in understanding neural scaling laws. Thank you for recognizing the importance of establishing power-law scaling of bipartite mutual information and its implications for architecture design.\n\n>**Weakness**\n\n>-   Overall, this is an interesting and solid work from my perspective. The possible weakness could be that the authors only analyze the model's history state size but do not establish a relationship between bipartite mutual information and the model's expressive power or generalization capability. This could be a potential direction for future research.\n\nWe thank the reviewer for noting this potential future direction. Our work on data-driven information scaling and existing work on model-centric scaling laws (e.g., relating model size to generalization) are complementary. The former establishes a necessary condition on model state based on the data's intrinsic structure, while the latter studies how model capacity affects downstream performance. We agree that bridging these two perspectives to create a unified theory is a valuable and important direction for future research.\n\nWe will add this to our discussion of limitations and future work, acknowledging that connecting information-theoretic scaling to model capabilities remains an open challenge.\n\n>**Questions:**\n\n>1.  In line 258, the authors claim that bipartite mutual information reaches its maximum for a given L when ℓ = L/2. It is encouraged to provide more details to explain this result.\n\nWe thank the reviewer for this question. Bipartite MI often tends to be maximized near balanced partitions because it generally depends more strongly on the shorter side of the split ($I(X;Y) \\le \\min(H(X), H(Y))$). However, the precise location of the maximum can vary. In our appendix, we show results for multiple $\\ell/L$ ratios demonstrating that the scaling law itself is robust to this choice.\n\nWe will revise the text to say \"tends to maximize\" rather than \"reaches its maximum\" and add a brief explanation that bipartite MI generally depends on the shorter partition length.\n\n>2.  Estimating mutual information is challenging, particularly for high dimension random variables. What is the error of the mutual information estimation method used in this paper? Does it increase with dimensionality?\n\nWe thank the reviewer for this important question. We acknowledge that estimating mutual information in high-dimensional settings is inherently challenging. Alternative methods like k-NN or neural estimators (MINE, InfoNCE) suffer from severe issues with the high and growing dimensionality of long text sequences. Our chosen methods, while not perfect, were the most robust and practical for this task, particularly as they do not exhibit rapidly increasing error with sequence length.\n\nWhile we provide preliminary error analysis in Appendix A, a complete quantification of the error is difficult, as it depends on how well the LLMs used in the estimation process capture the true data distribution. However, the clear and consistent power-law behavior we observe across different estimators and models gives us confidence in our main qualitative finding. We agree that refining the precision of the estimated exponent is a valuable direction for future work.\n\n>**Rating:**  5: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.\n\nWe thank you again for your constructive feedback. We hope our responses and planned revisions have fully addressed your concerns. We would be happy to answer any further questions and kindly ask that you consider these clarifications in your final evaluation."}]

Author rebuttal:

Meta-review: All four reviews of the paper leaned towards acceptance with three recommendations for borderline acceptance [jrcZ,cZLb,rSLc] and one recommendation for acceptance [gdYy].

Reviewers appreciated several aspects of the work:

+ The idea was considered novel [rSLc] and the problem was considered well motivated [rSLc] and significant [gdYy].
- The work was seen to have clear theoretical innovation [cZLb] and the theoretical arguments were considered compelling [rSLc]
+ The empirical finding of the scaling law was considered intriguing [jrcZ] and bringing new insights [gdYy]
+ The predictability characterisation of natural language on state space requirements was considered insightful [jrcZ]
+ The empirical verification was appreciated [cZLb] and experiments were considered comprehensive [rSLc], and connection between theory and experiments was appreciated [rSLc]
+ The paper was considered well written [rSLc]


Hovewer, a number of downsides were noted by the reviewers, and authors responded to them in the rebuttal stage:

- The estimator used for bipartite mutual information was criticised as not making use of existing literature on mutual information estimation [jrcZ]; authors argued several estimators had issues and stated they would add discussion.
- The takeaway was considered unclear in terms of how the paper improves understanding of transformer-based models [rSLc], and it was unclear how useful the analysis and condition are beyond SSMs suffering with long context [rSLc]. Authors argued their main contribution is to have the L^2M condition as a necessary universal lower bound, and discussed the implications on design challenges and overprovisioning in transformers. Some further discussion arose on potentially going beyond linear scaling and very long context windows.
- Lack of a connection between bipartite mutual information and expressive power/generalization capability was considered a weakness [gdYy]; authors considered it a future direction but would add brief discussion.
- Lack of connection to LLM-as-universal-predictor literature was criticised [jrcZ]; authors briefly discussed a connection.
- The quantitative effect of underestimating the scaling exponent and reliability of the exponent fits was considered unclear [cZLb]; authors provided some discussion on the estimation challenges.
- A clarity issue with versions of Theorem 5.2 was raised [jrcZ]; authors stated they would clarify.
- A question arose on generalisation to hybrid models or other generative paradigms [cZLb]; authors briefly commented hybrid models could satisfy the condition in some cases and stated to add a discussion on applicability of the framework.
- Another question was raised on whether architectures meeting the condition perform better in long-context tasks [cZLb]; authors noted their results suggest yes but discussed a gap to actual downstream performance, stating they would discuss this as future work.
- A question arose about use of the approach to clarify differences between domains [jrcZ]; authors provided brief discussion.
- A question on relation to other related work arose [jrcZ]; authors briefly discussed their framework is complementary.
- Additional detail on the maximum of the bipartite mutual information was requested [gdYy]; authors provided discussion and would add clarification.
- Discussion on error of the mutual information estimation method was requested [gdYy]; authors acknowledged the estimation is challenging and that complete error quantification is difficult, but left better estimation of the exponent as future work.
- An appendix was said to be missing [cZLb] but this was simply in the supplementary material.


After the rebuttal one reviewer [jrcZ] further hoped for an experiment on a biological sequence model/dataset but authors wished to leave it for follow-up work. Reviewer [cZLb] noted integration with downstream performance remained insufficient. Reviewer [rSLc] felt all but one concern was addressed, but was not convinced the work could enable scaling to large numbers of tokens, and felt practical benefits were unclear. Reviewer [gdYy] felt their concerns were addressed.

Overall, it seems reviewers appreciated the work, and it seems to make a useful contribution to understanding scaling laws for language modelling. While some concerns were left for future work, it seems the work can be at a sufficiently good state to be presented at NeurIPS.
Decision: Accept (poster)