Title: L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: language models

information theory

mutual information

predictive information

large language models

long context length modeling

long-range dependence

scaling laws

long-context language modeling

sequence modeling

autoregressive models

mutual information estimation

transformers

recurrent neural networks

state-space models
Abstract: We present a universal theoretical framework for understanding *long-context language modeling* based on a *bipartite* mutual information scaling law that we rigorously verify in natural language. We demonstrate that bipartite mutual information captures multi-token interactions distinct from and scaling independently of conventional two-point mutual information, and show that this provides a more complete characterization of the dependencies needed for accurately modeling long sequences. Leveraging this scaling law, we formulate the **L**ong-context **L**anguage **M**odeling (**L**$^2$**M**) condition, which lower bounds the necessary scaling of a model's history state—the latent variables responsible for storing past information—for effective long-context modeling. We validate the framework and its predictions on transformer and state-space models. Our work provides a principled foundation to understand long-context modeling and to design more efficient architectures with stronger long-context capabilities, with potential applications beyond natural language.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to depict the dependencies crucial for modeling long sequences by bipartite mutual information. Though comprehensive experiments, they establish a power-law scaling of bipartite mutual information across diverse natural language datasets using LLMs. With this observation, they introduce the concept of long-context language modeling condition and prove that the model's state size should outpace the scaling of bipartite mutual information to achieve effective long-context modeling. This finding is verified on both transformer and state-space models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The problem studied in this paper is significant and central, that is, the neural scaling law of LLMs. By conducting comprehensive experiments on recent state-of-the art LLMs, the authors establish a power-law scaling of bipartite mutual information for the first time, which is crucial to depict the long-range dependencies in natural language.\n- Based on the above observation, the authors further theoretically demonstrate that the history state dimension must grow at least as fast as the power-law scaling of bipartite mutual information in the data, which brings new insights for community to design or improve the network architecture of LLMs.\n\n**Weakness**\n- Overall, this is an interesting and solid work from my perspective. The possible weakness could be that the authors only analyze the model's history state size but do not establish a relationship between bipartite mutual information and the model's expressive power or generalization capability. This could be a potential direction for future research.\n\nQUESTIONS:\n1. In line 258, the authors claim that bipartite mutual information reaches its maximum for a given $L$ when $\\ell=L/2$. It is encouraged to provide more details to explain this result.\n2. Estimating mutual information is challenging, particularly for high dimension random variables. What is the error of the mutual information estimation method used in this paper? Does it increase with dimensionality?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work in a separate section after conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the long-range dependencies of natural language for large language models. More precisely, they establish bipartite mutual information scaling laws (as power laws) that explain multi-token interactions beyond the prior understanding given by two-point mutual information. They explain how to estimate these scaling laws with large language models and demonstrate their validity with GPT2 and Mamba(2) models on the PG-19 dataset. The authors use these scaling laws to provide a condition, the $\\text{L}^2\\text{M}$ condition, that defines the minimum size of the latent space of a model to be able to deal with long-context dependencies. They validate this condition experimentally with GPT-2 and Mamba(2) models on synthetic data, showing that transformer-based models naturally satisfy the $\\text{L}^2\\text{M}$ condition while SSM need to scale their latent space size with the sequence length to be able to satisfy it.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- The i sdeaeems novel, the paper is well written, and the problem tackled is well motivated\n- I find the theoretical arguments compelling\n- The experiments are comprehensive\n- The connection between theory and experiments is done properly\n\n*Weaknesses*\n\nI list below what I believe are weaknesses, but I would be happy to be corrected if I misunderstood some important parts of the current submission.\n1) The takeaway of the paper is not clear to me: indeed, transformer-based models always validate the LM condition because the key-value pairs scale linearly with the sequence length. Given that current SOTA models are transformer-based ones, how does the current paper improve our understanding of such models?\n2) Related to the point above, to me, the author's proposal rather highlights the current limitations of SSM-based LLMs, whose sizes need to scale with the sequence length to be able to model long-context dependencies. As such, it is not clear how useful the proposed analysis and the LM condition are beyond \"SSMs might suffer more from long-context\". Could the author please elaborate on that?\n\n*Reason for my current score*: Overall, I find the paper interesting and novel, with a well-conducted theoretical study with empirical verifications. However, the main message and the impact of the paper for the design, study, or use of LLMs are not clear to me. Notably, long context is indeed an issue, but we have seen increasing context windows in the recent open-source LLMs like the Llama, Gemma, Qwen, or Deepseek families. How does the proposed approach improve long-context abilities or our understanding of them beyond state-space models? These are the reasons why I lean towards a borderline acceptance, but I am willing to change my score provided the authors correctly address my concerns.\n\nQUESTIONS:\n1) In the proof of Thm. 5.5, line 281, shouldn't it be \"$I^{BP,q}$\" instead of \"$I^{BP}$\"? Otherwise, it leads to comparing the same quantity in front and after the inequality sign.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and empirically substantiates a scaling law for bipartite mutual information in natural language, arguing that such a measure more comprehensively captures the long-range dependencies critical for long-context language modeling than traditional two-point (token-to-token) mutual information. Using empirical analyses with LLMs and diverse datasets, the authors demonstrate that bipartite mutual information scales as a power law. From this, they formalize the \"Long-context Language Modeling (L²M) condition,\" which prescribes that the latent state size of a model must scale at least as rapidly as the bipartite mutual information with respect to context length to effectively store predictive information. Theoretical results and extensive empirical studies are provided to validate these claims and elucidate architecture-specific implications for transformers and state-space models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Clear Theoretical Innovation: The manuscript introduces a rigorous, information-theoretically grounded scaling law for bipartite mutual information in natural language sequences. By establishing the distinction and practical importance of bipartite over two-point mutual information, the authors provide a novel lens through which to view long-range dependencies.\n\n* Empirical Verification: The claims are validated both on synthetic datasets and on real-world data. The L²M condition can give actionable guidance for model designers, linking information scaling properties with minimal latent state requirements.\n\n\n**Weaknesses**\n\n* Bipartite MI Estimation: While two estimators (direct, vCLUB) are used, the paper acknowledges that both likely underestimate the true scaling exponent β, yet the quantitative effect remains unclear.  The reliability of exponent fits completely resolved.\n* It appears that the appendix is missing.\n\nQUESTIONS:\n* Can the framework be generalized or adapted for hybrid models (hybrid transformer and state-space model) or even other generative paradigms (e.g., diffusion models), or will fundamentally new information scaling laws be needed in those domains?\n* Are there empirical indications showing that architectures meeting the L²M condition perform better in practical long-context tasks (e.g., document QA) compared to those that violate it? What might be the gaps between theoretical MI scaling capability and actual task performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with scaling laws for long-context language modeling and specifically develops a seemingly novel information measure that ends up having an empirical power-law characterization.  This is what they call the bipartite mutual information, and is the standard Shannon mutual information but over long disjoint blocks.  An implementable (suboptimal/biased) estimator for this quantity is developed through the use of LLMs.  A condition on the state space requirements for LLMs is developed using the scaling law, and used to analyze Transformers and state-space models.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\nThe empirical finding (assuming the bipartite mutual information is well-estimated) of the scaling law is quite intriguing, in some sense moreso for computational linguistics than for machine learning.\n\nThe implication of the predictability characterization of natural language on the state space requirements is insightful.\n\nWEAKNESSES\nI'm not convinced that the estimator for bipartite mutual information developed and used in this work is the best it could be. Appendix A.IV is an improvement on [91] but doesn't draw on the huge literature on mutual information estimation that includes unbiasedness and optimality results.  \n\nThere does not seem to be much connection to the LLM-as-universal predictor literature, which might help clarify some things on the state space requirement derivation.  Indeed, the universal prediction literature in information theory is often concerned with state space size.  \"Transformers Learn to Compress Variable-order Markov Chains in-Context\" and \"Transformers are Universal Predictors\" are two papers I could find.\n\nThe fact there are really three versions of Theorem 5.2 is not mentioned in the main paper, this could be clarified.\n\nQUESTIONS:\nIf one looked at some other corpora, e.g. proteins, DNA sequences, or computer code, would the approach in this paper help clarify the distinctions between the domains, e.g. why smaller models seem to be okay for biological sequences than natural language?  \n\n\nAuthors say that theoretical understanding of neural scaling laws remains limited, but they might be interested in work such as Nayak and Varshney \"An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models\" (NeurIPS 2024 workshop) that actually uses information-theoretic principles.  See also references thereto.  I would be curious to know if there are connections between scaling laws for language as in the present work and the information-theoretic explanation of neural scaling laws.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}