Learning is Forgetting; LLM Training As Lossy Compression
Authors: Henry Conklin
Tom Hosking
Tan Yi-Chern
Jonathan D. Cohen
Sarah-Jane Leslie
Thomas L. Griffiths
Max Bartolo
Seraphina Goldfarb-Tarrant
Venue: ICLR 2026 Poster (2026)
Keywords: Compression
Information Theory
Learning
Generalisation
LLMs
Interpretability
Primary area: foundation or frontier models, including LLMs
TL;DR: LLMs learn an optimal compression of the internet.

Abstract: Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.

Published as a conference paper at ICLR 2026
LEARNING IS FORGETTING:
LLM TRAINING AS LOSSY COMPRESSION
Henry C. Conklin Tom Hosking", Tan Yi-Chern', Sarah-Jane Jonathan D. Cohen',
Thomas L. Griffiths", Max Bartolo", Seraphina Goldfarb-Tarrane
PPrinceton University
*Cohere
henry.comm conk1 prennkii@@prie inteoonnnonn.. edu,
ABSTRACT
Despite the increasing prevalence of large language models (LLMS), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre- pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of model's 's compression, and the information present in it, can predict downstream perfor- mance on across wide array of benchmarks, letting us directly link representa- tional structure to actionable insights about model performance. In the general case the work presented here offers a unified coeimatiioo-hhoreetee framing for how these models learn that is deployable at scale.
INTRODUCTION
We still have a limited understanding of how Large Language Models (LLMs) achieve impressive results across wide array of tasks (Devlin et al., 2019; Grattafiori et al., 2024). While a growing body of work interprets LLMs using behavioural experiments, probing, or causal interventions, the scale of these models makes understanding how their representation spaces are structured a contin- ued challenge. Here we look at an LLM as instance of lossy compression, offering account of how models represent information during training and what information matters for performance.
Lossy compression represents data efficiently by preserving only the information from source relevant to a goal. While audio recordings intended for human listeners can be gigabytes in size, MP3 files save space by discarding frequencies typically outside the range of human hearing (Jayant et al., 1993); similarly, a JPEG file omits subtle colour variations that are difficult for the human eye to perceive. We draw a parallel with LLMs, which are expected to generate responses humans prefer, after being trained on trillions of tokens more language data than human hears in 200 lifetimes. More generally, compression is thought to underpin learning in both humans and models (see Feldman, 2016), giving formal account of LLM pre- pre-training in terms of compression allows us work towards unified theory of representation learning. We present results showing that over the course of pre-training LLMs optimally compress the information present in their training data for next sequence prediction.
Compression is inherently opinionated - some information from the source is preserved, some is forgotten to save space. Information Theory (Shannon, 1948) provides a formal language to de- scribe this process, letting us both quantify the information present in representation and compute a bound where it is optimally compressed with respect to the data it represents. Our results build on the Information Bottleneck (IB) theory of deep learning (Tishby & Zaslavsky, 2015), showing pre- pre-training follows a two phase trajectory: first increasing mutual information with the training objective, before compressing input information. Across wide arry of L.LMs we find each model

---

Published as a conference paper at ICLR 2026
bound
Over pre-training models approach the bound on optimal compression.
OLMo 2 7B Pre-Trame Training
3.5
impossible
7
0.20
Log Takans (in billionss
Expresivitty IlY; 2)
0.15
Cross Entrpy Loss
6n
2.0
0.10
1.5
1.0
3
0.00
0.45
0.50
0.55
0.60
0.65
Complexity IDC: Z)
Compression Optimality
Expressivity ((Y; Z)/Complexity ([X; Z)
Figure 1: LLMs Learn Optimal Compression of the Internet (Left) The information plane for pre-training of the OLMo2 7B model. The horizontal axis shows mutual information between representations and the input (complexity), the vertical axis shows mutual information with the predicted output (expressivity). The dotted line indicates the bound where models are optimally compressed, hue indicates timepoint in training in terms of tokens in billions. Estimates based on 10,000 samples from the C4 dataset which is a broad crawl of the internet. (Right) The vertical axis shows the OLMo2 7B model's loss eext-token-predictinn of C4. The Horizontal axis shows the model's proximity to the bound. Representations begin to approach the bound as the loss saturates.
compresses differently, with the optimality of model's 's compression and the information it pre- serves predicting performance on downstream benchmarks.
hallmark large-scale distributed systems, like neural networks, that they are difficult to un- derstand as function of their parts alone (Anderson, 1972; Mitchell, 2009). Our approach to inter- pretability allows us to consider learning and generalisation the scale an entire model, rather than studying individual circuits or neurons within it. Additionally it allows us to frame how models do so well at so much in terms of existing theories of learning and compression, while providing actionable insights LLM scale.
In what follows we focus on offering concrete answers to three questions: Do LLMs optimally com- press their representations? What information survives that compression? What representational structures drive performance? In summary, the core findings are:
Pre-training dynamics for LLMs closely follow theoretical predictions from the Informa- tion Bottleneck, with models first expanding representations before slowly approaching optimal compression.
Scale conditions these dynamics, with smaller models (below billion parameters) strug- gling to achieve meaningful compression later in training.
How optimally compressed model is correlates significantly with performance across six benchmarks for six families of open-weights large language models, letting us directly relate representation structure behaviour.
By quantifying the amount of preference information in model we get a quantification of how aligned representations are with preference distinctions which significantly predicts downstream performance across 47 LLMs (r = 0.76, p < 0.001).
Finally, we compare wide array of open-weight models across model families, showing they all converge near optimal compression.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND & RELATED WORK
2.1 LEARNING, INFERENCE, AND COMPRESSION
Compression has been argued to underpin learning and inference in humans (Chater, 1997; Chater & Vitányi, 2003; Feldman, 2000; Pothos & Chater, 2001) and models (MacKay, 2003; Poggio al., 2004). Increasingly, probabilistic inference and complexity minimisation are seen deeply intertwined (Feldman, 2016) - a point perhaps made clearest by Bayesian inference, which implic- itly prefers the simplest hypotheses consistent with observed data (Edwards, 1972; Jeffreys, 1939; Vitányi & Li, 2000). Bayesian approaches to human cognition offer accounts of how a broad array of human behaviour can be productively thought of as this kind of inference (see Griffiths et al., 2024, for a review). In machine learning Occam's 's Razor has long been used as a model selection criterion - where the best model is the simplest one consistent with the data (Burnham & Anderson, 2002; Rissanen, 1978; Wallace & Boulton, 1968). The bias variance trade-off (Geman et al., 1992) makes this explicit in the context of neural networks, showing more complex models may achieve better fit to the training data, but they also generalise worse than their simpler counterparts. While some work has studied whether or not LLMs can match lossless compression algorithms in-context (e.g. Delétang et al., 2023), this is distinct from giving an account of LLM training itself pro- cess of lossy compression the object of study here. is worth noting that there is not universal agreement about how to assess compression (see MacKay, 2003, for discussion), but here we follow in the information-theorrtic tradition (Shannon, 1948).
2.2 RATE DISTORTION THEORY
Consider a function 0 that encodes an input X in a representation Z, Z = (X)). This representation is then decoded by function & to produce predictions Ý for an output with true label Y. Ý = o(2). Assuming that X and Y are independent, 0 were losslessly preserve all the information from the input, we would expect & to be able to precisely recover the corresponding output. with y = Y. Rate Distortion Theory (RDT) (Shannon, 1948) instead considers the lossy case Y * Y. where some amount of error in the prediction distortion is acceptable. It then becomes question of how much information about the input - termed the rate- - the encoder needs to preserve to achieve a given level of distortion.
The Information Bottleneck (IB) Tishby et al. (2000) looks at a particular case, where the rate is given as the mutual information between inputs and their representation T(X; Z), and distortion as the mutual information between a representation and the corresponding target prediction I(Y: Z) the 2D space this creates is called the information plane (shown in Figure 1). Since [(X; 2) reflects how much information about the input space is preserved it can be referred to as complexity (e.g. Zaslavsky al., 2018). Likewise [(Y; Z) is referred to accuracy given it quantifies how much information a representation has about the target output it will be used to predict. To distinguish this quantity from behavioural accuracy (e.g., exact match on task) we refer it as expressivity how uniquely a representation can refer to its target (in line with Kirby et al., 2015). Optimal compression within the IB occurs when an encoding Z X preserves only the information about X relevant to predicting Y, or when X|X minimises
Fe(p(Z|x)) -
(1)
where 3 is a trade-off parameter controlling the allowable level of distortion. When 3 approaches 0 all inputs are compressed to single point, as 3 + oo we approach the lossless case, where using X or its encoding Z tells us the same information about Y; I(Y: Z) = I(Y; The curve traced by varying / draws a bound, where the encoding p( x) is optimally compressed everything above the curve is unachievable and everything below it is suboptimal. This bound starts off with linear relationship where / [(X; Z) = Y(, until Z captures all information shared between X and Y. Intuitively, in optimal encoding each additional bit of complexity gets you an additional bit of accuracy, until all information shared by X and Y are represented; I(Y; Z) = XY: - in the cases studied here all models stay well below this saturation point, so for clarity we refer to the bound as the line I (X; Z) = I(Z; Y) (for further discussion of the bound and computing it numerically see Appendix E.8).
3

---

Published as a conference paper at ICLR 2026
Applying the Information Bottleneck to Deep Learning Tishby and Zaslavsky (2015) offered theoretical characterisation of training a multi-layered neural network as optimising an Information Bottleneck. They theorise two phases of training: first fitting phase during which representations increase mutual information with the target labels T(Y: Z); and second a compression phase, during which models compress irrelevant information about the input T(X; Z) and in the process begin to approach the optimal bound. It is this latter phase that is hypothesised to result in representations that generalise robustly.
Shwartz-Ziv and Tishby (2017) confirmed the two-phase prediction from the IB theory of deep- learning empirically in feed forward networks trained on MNIST. Subsequent work has questioned the generality of these findings, showing how - at least in linear networks the compression phase can driven by the type of non-linearity used (Saxe, Bansal, et al., 2019), or that compression not necessarily required for generalisation (Goldfeld et al., 2019). remains unclear whether deep- learning models in the general case can be expected to follow the phases of expansion and compres- sion predicted by the IB, in particular when it comes to sequence models (e.g. Transformers) trained on complex tasks.
2.3 INTERPRETING NEURAL NETWORKS
A broad literature on the theory of deep learning tries to give an accounting of learning dynamics in small multi-layer networks (e.g. Frankle & Carbin, 2018; Saxe, McClelland, & Ganguli, 2019). While there has been some extension of these kinds of representational analyses to larger models- like applying information theoretic methods to transformers (Voita et al., 2019) - much of the work on interpretability in LLMs leverages behavioural or probing evidence. Behavioural approaches treat models as akin to psycholinguistic subjects (Futrell et al., 2018, 2019), taking model outputs as behaviours (Hu et al., 2020; Marvin & Linzen, 2018; Warstadt al., 2019). Probing (Pimentel al., 2020; Veldhoen et al., 2016; Voita & Titov, 2020) trains a smaller model like a linear classifier to predict labels from a model's latent representations. as evidence that information relevant those labels is present. While valuable, these approaches are removed from the models' representations themselves characterising downstream behaviours rather than characterising the representational structures that drive them.
Mechanistic interpretability follows in a similar vein but aims to describe how circuits within a model implement the functions that solve task. These analyses have given accounts of how two layer linear and non-linear models represent features from synthetic data (Elhage et al., 2021) or how single-layer attention only transformers solve modular addition (Nanda et al., 2023). When deployed at scale, to LLMs, this work often relics on training unsupervised probes termed sparse auto-encoders (Elhage et al., 2022) to identify correspondences between parameters and different words or concepts from the training data (Bricken et al., 2023). In the general case this work often looks for semantiiatt looking for lossless, one-to-one correspondences between input fea- tures and parts of a model. More recently studies of when features emerge during pre-training have aligned with the expansion/comprrssion pattern described by the IB theory (Ge al., 2025).
To be sure, there is an abundance of methods for analysing deep-learning models. Here, however, highlight disconnect between work on the theory of learning in humans and neural networks, and work on interpretability. Interpretability methods can be deployed at scale on complex models and tasks, but lack clear relationship to existing theoretical work. In the sections that follow we operationalise Rate Distortion Theory, and related work on learning as compression, at LLM scale. This allows us to analyse training dynamies while contextualising our conclusions existent and well-studied theoretical frameworks. Our approach represents one that is theoretically motivated but can be applied to any model at any scale.
METHODS
3.1 ENTROPY ESTIMATION
Let T E R be a batch of B tokenized samples with sequence length S, drawn from a corpus of text data T. and let 8 be model with L layers and representation dimension h; the corresponding encoded representations are Z E I Bxhhx, Let X E Rx8 be feature labels for the text in T.. For example, when look optimal compression with respect to the IB bound, these labels X
4

---

Published as a conference paper at ICLR 2026
the token ids for the model inputs; however, when analysing representation information more generally. these can be other input features, such as preference label or language id. It is desirable to compute the mutual information I [X; Z) using Shannon entropy opposed to differential entropy accomplish this, previous work quantises Z into bins, get discrete encoding 2 (Shwartz-Ziv & Tishby, 2017; Voita et al., 2019)'. Unfortunately the approaches from this previous work have memory and resource requirements that make them difficult to apply LLM scale. As a result we use the soft-entropy estimator from Conklin (2025) this is an efficient differentiable relaxation of a binning-basse estimate that has been shown to converge to the true entropy of a distribution. We describe the estimation process in detail below, this estimator is not original to work but the first to apply it to analyse LLMs using rate distortion theory.
To obtain a soft quantisation 2, this approach first computes Z, which is the normalization of Z to lie on the surface of the unit sphere S* in R It then samples n points {wx} } uniformly at random from Sh. 2 Then, for each normalized representation Σ E TRA,, we compute vector whose 7th entry is the cosine between & and w,, then apply softmax to that vector softly assigning each embedding to the points in W. More formally, for each (1, b, s) E [
X [B] x [S], tensor Z (whose shape coincides with Z) is defined so that Zi,,,s: = Z. Z.bba::lll and we stack the uniform samples {w; 1 into a matrix W E RRxnn Tensor 2 E RR LxBxXXxx then defined so that for (1, [L]]
B x [S].
Ejmn Zeihi W
{w )n),
W = Wy. ZLb...: = softmax
(2)
Where e temperature parameter, which we set to enable direct comparison of representations with different dimensionalities following the procedure described in Appendix E.1.1. Each vector Z b., defined this way is a probability vector. Let 2 E RR be the matrix obtained from tensor Z by averaging over the batch and sequence dimensions, and let 2 be the I-th of this matrix, probability vector of length n by construction:
B
8
n E1, log &t.j-
2 BS
Z.b.
Z1,0
H($))
Vectors & are probability vectors for each layer E [L] describing a categorical distribution over n categories. Therefore can compute the Shannon entropy H(2) above. Due to the normalisa- tion step during quantisation, this distribution approximates the probability that a representation in layer lies along a particular angle with respect to the origin. To estimate the entropy in entire model. denoted H we average entropy across layers. Efficiency (Wilcox. 1967) normalises H by the entropy of a uniform distribution lng(n). thereby bounding the entropic quantity between 0 and 1- to aid interpretability here we convert H (Z) to an efficiency H(Z) by additionally normalising by the entropy of uniform distribution at each layer. These definitions can also be conditioned on the feature labels X.
H(Z)= L logg(n)
Σ H()), x
Log(n) Σ
H(aX = x)
(4)
This now allows us to efficiently compute the mutual information between input features X and encodings across an entire model, regardless of model size.
I(X; = H(Z) Σ =X)= xEX
(5)
3.2 MUTUAL INFORMATIONS & BACK-OFF
To look at whether or not a model is optimally compressed with respect some data we need to compute mutual informations with respect to input and output labels. LLMs are trained with in- puts as preceding context and outputs as trailing context (for discussion of this, and examples of
For discussion of Shannon entropy and why previous approaches not scalable see Appendices E.6,E.7. This is cquivalent to sampling from an isometric h-dimensional multivariate normal, th ~ (O, Id,), and scaling unit length, w0 |0,11
5

---

Published as a conference paper at ICLR 2026
the labelling procedure see Appendix E.4). Maintaining conditional estimates of token embedding given a preceding context ((Z[X) for every possible context window proves intractable, and many contexts occur only once in the training data. Accordingly, like many other works on language modelling we approximate the distribution over possible sequences using n-grams with a kind of back-off (Katz, 1987). By conditioning proximate X)) the maximum width we consider here are quad-grams by which point / [Xx; 2) begins to converge and past which point computation becomes intractable in an LLM setting, By backing off further (e.g. to trigrams, bigrams, and tokens) we can also estimate how much different context widths contribute to information level backoff, with other levels of backoff noted where they' re presented. We vary the degree of backoff equally for both the input P(Z X) and output P(Z Y)) distributions, this is because during training a model receives gradient information from the full trailing context Y due to teacher forc- ing (see Appendices E.2, E.3 for further discussion and visualisation of the labelling procedure). In comparing different models we would like to be able to determine how close given representation system is to the IB bound by extension, how optimally compressed is. When on the bound, representations preserve only the information from the input relevant predicting the output. We quantify this with a summary statistic optimality.
finite widths of preceding context we can tractably ap-
model for clarity the majority of results use token
Optimality = Expressivity 2(
(6)
Complexity I(X;Z)
Intuitively this quantity approaches 1.0 as a representation system approaches the bound, regardless of where along the bound the system is placed (i.e. which beta value along the bound the system closest to). More generally this is a relative quantity reflecting how many bits of expressivity a system has for each bit of complexity.
In addition to mutual information with input and output labels, we also consider preference data. A growing body of work stresses the importance of post-training approaches for aligning models with human preference (Bai et al., 2022; Ouyang et al., 2022; Rafailov et al., 2023). We can quantify this information in a model using preference data, where prompt has two continuations, one of which is labelled preferred by raters the other labelled as rejected. Conditioning on this label lets us compute P(Z] preferred) and [Z: preferred).
Data and Sampling Getting a true estimate of the entropy of vector space remains a major challenge, with most approaches underestimating the true entropy (Paninski, 2003). As a result we do not claim our experiments estimate the entropy of a model's true latent distribution, but rather an estimate of the entropy with respect to particular sample of data. By holding the data constant across models and experiments we can compute an estimate that useful for comparisons, even if does not exactly match the true entropy. Unless otherwise noted, token bigram, trigram, and quad- gram estimates are with respect to 10,000 samples from C4 (Raffel et al., 2020), and preference estimates are based on 10,000 samples from Tulu (Lambert et al., 2024); in both cases consider a maximum context length of 512.
4 EXPERIMENTS
In order to study training time-courses our pre-training analyses look at the OLMo2 family of models (OLMo al., 2025), which makes available intermediate checkpoints 3. We focus analysis on the 7B model unless otherwise noted, while including results for the 32B and IB variants to show where conclusions hold or differ across model scales. In addition, to show our conclusions hold outside of this particular family of models we compare wide array of open-weights LLMs (which do not make intermediate training checkpoints available), showing where they lie on the information plane at the end of training.
3 'Appendix D includes additional pre-training analyses of the Smol LM2 (Allal et al., 2025) and Pythia (Biderman et al., 2023) models which also make intermediate checkpoints available. These follow similar pattern the results presented here.

---

Published as a conference paper at ICLR 2026
Compresioon Optinalty
Exprosiiily IY: 2) Complexly II 21
Varying degree backaff shows more optimal compression context
6.
6.
Tokkn Backoff
Birron Bockol!
Trigron Bockoff
Quodgran on Bockol!
Sc
0.0
8:8
120
531
8.43
Complnry
Ghaeggen
Billoon Poraneters
Bilor Poromaners
32 Bl Faromareers
The majority of information in a model encodes
0.50
0.49
0.49
Departa 2
0.44
local context
0.30
121 101 127 12²
10P 10 101
Figure 2: Models Largely Encode Local Context. (Top) The information plane over pre-training for the different levels of backoff. By changing how many tokens we condition the mutual infor- mation on in the context window, we see how the OLMo2 7B model compresses not just token but also local context information. Across all context windows see the same two phase pattern predicted by the Information Bottleneck- with more contextual representations approaching greater optimality indicated by hue. As context increases models compress both the target and source over training, rather than compressing the target independently. We hypothesise this is because in lan- guage modelling with full context the target and source distributions are nearly identical. (Bottom) By computing the conditional mutual information for level of back-off given the others we can quantify what proportion of a model's information encodes each level of context information. Each facet shows different model size, with the horizontal axis reflecting training step and the vertical axis reflecting proportion of information hue indicates level of back-off
4.1 PRE-TRAINING APPROACUEES OPTIMAL COMPRESSION
The majority of pre-training appears to be slow compression of a model's training data. The Infor- mation Bottleneck theory of deep learning predicts two phases: a fitting phase during which output information I(Y: Z) increases, followed by compression phase during which input information T(X; Z) decreases and representations approach the bound. This transition to compression is be- lieved to occur when error on the training set saturates. Shown in figure (and reproduced in Figure 2) is the training trajectory for the OLMo2 7B model with respect to data from English C4. Strik- ingly, the 7B model closely follows the two-phase prediction from the Information Bottleneck, first increasing mutual information with outputs, before compressing input information and progressing towards the bound on optimal compression. Additionally this transition appears to happen as the model's loss on next-token prediction begins to saturate (figure I right). This shows how, even at scale, deep-learning models appear to thread a needle between representational complexity and ex- pressivity. It also demonstrates how LLMs can be effectively studied from the perspective of Rate Distortion Theory, as they try to converge to an optimal lossy compression of their training data.
Models More Optimally Compress Contextual Information By varying the degree of back-off in the source and target distributions used to compute mutual information, we can see how contextual information evolves over pre- pre-training at the token, bigram, trigram, and quad-gram levels (Figure 2 top). All cases result in similar two-phase pattern of expansion and compression, with larger conditioning context converging closer to the bound (indicated by hue). For token-level back-off late training aligns with previous work MNIST, with models compressing the source distribution
7

---

Published as a conference paper at ICLR 2026
-reducing complexity while maintaining expressivity. At higher levels of contextualisation both complexity and expressivity are reduced. We hypothesise this is because in language modelling the source and target are sampled from the same distribution, what counts an 'input' vs. an 'output' is a product of what point in the sequence the model is during generation unlike cases with an inherent difference between the distributions (e.g. MNIST classification where the source is over images and the target is over discrete labels). While individual tokens, or words can have divergent source and target distributions an adjective almost always precedes a noun and follows determiner; permuting that order will render a sentence ungrammatical at the phrase, sentence, or paragraph level the difference between preceding and trailing context between target and source becomes harder to identify. Making it difficult to compress one without also compressing the other, resulting the reduction of both complexity and expressivity the trigram and quadgram facets of figure 2. The higher degree of optimality in contextual encodings likely reflects an inherent pressure in the pre-training objective for models to not only develop token representations, but representations of a token in context.
Embeddings Largely Encode Local Context We compute the proportion of information in model explained by each level of back-off in the source distribution independently (a detailed ex- planation of this process with results related to performance can be found in Appendix E.5). As shown in figure (bottom) the majority of information in a model encodes local context (token to quadgram), likely reflecting the information locality of the natural language on which they'r trained (Gibson, 1998; Gibson et al., 2000; Hahn et al., 2022). The billion parameter model also has more token information and less contextual information than its larger counterparts. The residual informa- tion likely encodes the finer-grained contextual distinctions found in the remainder of the 512 token context window (i.e. information up to an n-gram width of 512) given the sparsity of n-grams greater than a quadgram those mutual informations are intractable for us to compute.
The Effect of Scale: Smaller Models Struggle to Compress Parameter count shows a marked effect on the degree of compression achievable by a model. Figure shows pre-training trajectories for the 1B, 7B, and 32B parameter models. The larger models both closely follow the hypothesized Information Bottleneck trajectory, exhibiting phases of expansion and compression, ultimately ap- proaching optimal compression. The 1B parameter model exhibits markedly different behaviour. While it successfully completes the initial expansion phase increasing output information [Y: 2) it fails to approach optimal compression. Instead, in the second phase the smaller model oscillates while moving slowly away from the bound (Figure bottom-left). This suggests that for a given level of data complexity, certain parameter threshold may be necessary for models to achieve an optimal compression - an observation in line with work on scaling laws (Kaplan et al., 2020).
Wide Array of Open-Weight Models Converge Along the Bound In addition to looking at the OLMo2 family of models, we compute complexity and expressivity estimates across a di- verse array of weighweight models. A striking convergence pattern emerges: across different model families, byper-parameeees, and training methodologies, representations ultimately converge to rep- resentations clustered near the bound on compression (Figure 3, Top Left; with full model names in Appendix, Figures 5 and 6). More than that models all approach the same point on the bound, suggesting they all converge to a similar information structure. This suggests that training as pro- cess of compression is not an artifact of a single LLM's training trajectory, but more fundamentally applies to deep-learning models as class, and to the data and the objectives used to train them.
4.2 RELATING REPRESENTATION STRUCTURE TO PERFORMANCE
So far we have studied how information in an LLM is structured; we now consider how that structure relates to downstream performance. Figure 4 shows correlations between representational measures and aggregate performance across six benchmarks (MMLU Pro, BBH, Math LVL55 IFEval, GPQA, MuSR) for 47 open weights models from 6 different families (Evaluations and aggregation from Fourrier et al., 2024). With token back-off lower Complexity relates significantly to performance (r -0.38, p 0.006), while expressivity alone does not (r 0.08, p 0.375) Figure 4 (top). However, the ratio between expressivity and complexity, our measure of how close a model is to optimal compression, is significant predictor of downstream performance(r 0.52, p < 0.001).
8

---

Published as a conference paper at ICLR 2026
Open weights models converge along the bound.
Expresiitty vs Complealy
Tab Preferance vL Complexity
0.5
Madi
04
-05
Expeiiity
0.3
0.4
0.0
Conpresdin Optedllyy
Expresilyy (1f Compledy 13C
Smaller models struggle compress later training.
Billion Parametert
7 Billon Porometers
32 billon Forameters
Expesilly I11
Expesbiityy IIV:
0.46
0.45
0.43
0.41
Complexity IIX:
Complextyy (0)
Complexity IIC:
Figure 3: Models Converge Along the Bound With Smaller Models Struggling to Compress. (Top Left) SpenhWeights models across 6 families at the end of training, lie along the bound on optimal compression. Hue indicates performance on MMLU Pro. (Top Right) The vertical axis indicates mutual information with preference, with models with more preference information ex- hibiting better performance (Bottom) Zooming in on later pre-training for each model size the IB model matches Phase I but struggles to achieve meaningful compression later on, oscillating for much of pre-training off the frontier. All results use back-off to the trigram level. A full legend identifying each dot, with additional levels of back-off, is given in Appendix Figures 5 and 6.
While LLMs approach optimal compression for next sequence prediction over pre-training, large body of work also tries to improve their ability to follow instructions, and generate responses hu- mans prefer (e.g. Ouyang et al., 2022). We use preference data (Lambert et al., 2024) to compute mutual information with preference. As shown in Figure 4 (bottom right), the amount of prefer- ence information in model proves significant predictor of downstream performance (r 0.76, p 0.001). This suggests that not only does the optimality of a model's compression matter, but exactly what information survives that compression does too. In Appendix C include results showing that post-training can increase the amount of preference information across different mod- els in the Llama family while minimally changing their complexity. This suggests that pre-training is responsible for the broad compression learned by a model, while post-training edits the infor- mation it contains; we leave more complete assessment of how different phases of training affect representational compression to future work.
These results also indicate how the information theoretic approach taken here could potentially be leveraged during training. Two applications could be stopping-criterion ceasing pre-training when distance to the bound no longer decreases, or as a model-selection criterion picking the checkpoint that is the most optimally compressed, or with the highest proportion of preference in- formation. Given the estimates here are computed with a Lingl-forward pass using teacher forcing, computing entropy estimate for candidate selection would be substantively less costly than eval- uating a model across a suite benchmarks. We look to experimentally validate these potential use cases of our approach in future work.

---

Published as a conference paper at ICLR 2026
Relating Model Information to Performance
Complexity
Expressivity
spearnon 0.36, 0.006 partel -.114 0.346
spvormon 0.08, 5.575 pariel 32, 0.021
Benchmorks
MMLU Pro, 9âH, Mamh LVL55, IFEval, GPQA, MuSR
Overall Accuacy
With token back-olf lower Complexity ralates significanfly
performance
Optimality
compression also relates significantly. So doas the amount of preference information
0.08
0.10
0.12
0.14
0.16
0.18
0.20
0.04
0.05
0.06
0.07
0.08
Complexly (00)
Expresevity I1Y:
Optimality
Preference
m0.52 pmc0.001 pariel m0.13, pro.373
m0..66 pms0 partol m0.71 p=40.001
Greelleed 2228
Greelmmed 2iB
Gree/bred (hamne)
theeetyhi th Dordon
Overll Ascuracy aay
kiblew
%
0.3
0.4
0.5
0.8
0.7
0.8
0.5
1.0
1.5
2,0
2.5
3.0
3.5
Optindily ZM0X3
Preference
x10
Figure 4: Representation Information Relates Significantly Performance Vertical axes, shared across all plots, show aggregate performance across benchmarks (MMLU Pro, BBH, Math LVL5, IFEval, GPQA, MuSR). Horizontal axes use token back-off to show complexity significantly corre- lates with downstream performance (Top Left), while expressivity alone does not (Top Right). The ratio how many bits expressivity model has per bit complexity does correlate significantly this quantity indicates how optimally compressed a model is (Bottom Left). The amount of prefer- ence information a model also correlates with downstream performance (Bottom Right). Above each facet results from spearman correlation between their axes, and partial spearman that treats a model's number of parameters as a covariate. Expressivity and complexity are calculated using token back-off in all four facets.
CONCLUSION
The work presented here bridges the gap hetween theoretical accounts of learning and the practical complexities of LLMs. We show that LLMs learn an optimal compression of the data on which they are trained, with a wide array of open-weights models converging along the IB bound with the optimality of model's compression predicting downstream performance. Each compression is different; we can account for the information that survives the compressive process, showing how representations encode information about different levels of local context and human preferences.
The approach interpretability we introduce here interprets a model as whole rather than fo- cussing on particular circuit, attention head - because complex distributed systems are not best understood in terms of their parts alone. Giving a holistic account of what means to train an entire model on the entire internet is challenge, but argue that LLMs are best understood as lossy compression. In doing so, we place them the context of a long history work on representation learning across the sciences.
ACKNOWLEDGEMENTS
We would like to thank Kenny Smith for his role in developing the core ideas presented here in earlier versions of this project. We also thank Julian Gold for discussion, help with formalisation, and for contributing the temperature calibration.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
All experiments reported here use publicly available datasets and pretrained models obtained under their original licenses; see Appendix B.1 for details. To our knowledge, these datasets contain no personally identifiable information, and we are in compliance with their terms of use. No additional data were collected. More generally, all authors have read and adhered the ICLR Code of Ethics. To the best of our knowledge, these results and their dissemination do not raise any ethical concerns.
8 REPRODUCIBILITY STATEMENT
All datasets and pre-trained models used in our experiments are publicly available (see Appendix B.I). All code will be released with an open source license. Appendix B.2 contains details of compute resources necessary to reproduce these findings.
REFERENCES
Aggarwal, C. C., Hinneburg, A., & Keim, D. A. (2001). On the surprising behavior of distance metrics in high dimensional space. International Conference on Database Theory (ICDT),
420-434.
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlfček, H., Lajarín, A. P., Srivastav, V., Lochner, J., Fahlgren, C., Nguyen, X.-S., Four- rier, C., Burtenshaw, B., Larcher, H., Zhao, H., Zakka, C., Morlon, M.... Wolf, T. (2025). Smollm2: When smol goes big data-centric training of small language model. modified bessel functions and their ratios. Mathematics of
Amos, D. E. (1974). Computation computation, 28( (,2)),,,22 239-251.
Anderson, P. W. (1972). More is different: Broken symmetry and the nature of the hierarchical structure of science. Science, 177(4047), 393-396.
Arimoto, S. (1972). An algorithm for computing the capacity of arbitrary discrete memoryless chan- nels. JEEE Transactions on Information Theory, 18(1), 14-20.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 2Xxi:2204.0555 058662....
Biderman, S., Schoelkopf. H., Anthony, Q. G., Bradley, H., Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., al. (2023). Pythia: A suite for analyzing large lan- guage models across training and scaling. International Conference on Machine Learning, 2397-2430.
Blahut, R. (1972). Computation of channel capacity and rate-distortion functions. IEEE transactions on Information Theory, 18(4), 460-473.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., al. (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2.
Burnham, P., Anderson, D. R. (2002). Model selection and multimodel inference: A practical cetemattionnthhoreeee approach. Springer.
Chater, N. (1997). Simplicity and the mind. The Psychologist.
Chater, N. & Vitányi, P. (2003). Simplicity: A unifying principle in cognitive science? Trends in Cognitive Sciences, 7(1), 22-
org/10. 1016 1364- (613((0))0000005 Conklin, H. C. (2025). Information structure in mappings: An approach to learning, representation and generalisation. The University of Edinburgh.
Delétang, G., Ruoss, A., Duquenne, P.-.., Catt, E.,, Genewein, T., Mattern, C., Grau-Moya, J., Wen- liang, I.. K., Aitchison, M., Orseau, I.., et al. (2023). Language modeling is compression. arXiv preprint 2rXiv:2309.70666
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language tech- nologies, volume (long and short papers), 4171 1171-4186. Edwards, A. W. F. (1972). Likelihood. Springer.

---

Published as a conference paper at ICLR 2026
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds Z., Lasenby. R., Drain, D., Chen, C., et al. (2022). Toy models of superposition. arXiv preprint
arXiv:2209.1665 10652.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A.,
Conerly, T., al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread, 1,
Feldman, J. (2000). Minimization of boolean complexity in human concept learning. Nature, 407(6804), 630-633.
Feldman, J. (2016). The simplicity principle in perception and cognition. Wiley Interdisciplinary Reviews: Cognitive Science, 7(5), 330-340.
Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., & Wolf, T. (2024). Open IIm leaderboard v2.
Frankle, J., & Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural
networks. arXiv preprint arXiv 0336533...
Futrell, R., Wilcox, E., Morita, T., & Levy, R. (2018). Rnns as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint (rXiv:/809.011
Futrell, R., Wilcox, E.,, Morita, T., Qian, P., Ballesteros, M., & Levy, R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state. arXiv preprint 1rXiv:/903.06
Gao, L.. Biderman, S., Black, S., Golding, L.. Hoppe, T., Foster, C., Phang. J., He. H., Thite, A., Nabeshima, N., et al. (2020). The pile: An 800gb dataset of diverse text for language mod- eling. arXiv preprint arXivv2101.0000
Gao, T., Yao, X., & Chen, D. (2021). Simese: Simple contrastive learning of sentence embeddings. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Process- ing (EMNLP), 6894-6910. https://doi.org org/10. 18653/v1/2021 .em2ll-mmmlpaan
Ge, X., Shu, W., Wu, J., Zhou, Y., He, Z. & Qiu, X. (2025). Evolution of concepts in language model pre-training.
Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural computation, 4(1), 1-58.
Gibson, F. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1), 1- 76.
Gibson, E., et al. (2000). The dependency locality theory: distance-based theory of linguistic
complexity. Image, language, brain, 2000, 95-126.
Goldfeld, Z., Berg. E. d., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., & Polyanskiy,
Y. (2019, May). Estimating Information Flow in Deep Neural Networks [arXiv: MarXiv:1810005:: 810.05728 [cs, stat]]. Retrieved April 18, 2024, from
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur,
A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv preprini
arXiv:2407.21783.
Griffiths, T. L., Chater, N., & Tenenbaum, J. B. (2024). Bayesian models of cognition: Reverse engineering the mind. MIT Press.
Hahn, M., Futrell, R., Levy, R., & Gibson, E. (2022). A resource- model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43), e21226021 19.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, (2020). Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.08: 000000,.330....
Hu, J., Gauthier, J., Qian, P.. Wilcox, E.. & Levy, (2020). A Systematic Assessment of Syntactic Generalization in Neural Language Models [avXiv: 2005.03692), urXiv:2005.03699 [cs). Retrieved June 23, 2020, from http://rxiv.orr/aabtee www..org////b//..ooooo 036922
Jayant, N., Johnston, J., & Safranek. R. (1993). Signal compression based models of human perception. Proceedings of the IEEE, (10)). 1385-1422. https://doi.org/ http///oi.orr/10000 1og/10.1109/5.24 Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106, 620-630.
wwwpehhoolaorroooooooeeo org/CorpuslIID (0:ppusD::1880175: 17870175
Jeffreys, H. (1939). The theory of probability. OuP Oxford.
Jurafsky, D., & Martin, J. H. (2000). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition, 2nd edition. Pren- tice Hall series in artificial intelligence.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford,
A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv.2001.08361.
12

---

Published as a conference paper at ICLR 2026
Katz, S. (1987). Estimation of probabilitiess from sparse data for the language model component of a speech recognizer. IEEE transactions on acoustics, speech, and signal processing, 35(3), 400-401.
Kirby, S., Tamariz, M., Cornish, H., & Smith, K. (2015). Compression and communication in the cultural evolution of linguistic structure. Cognition, 141, 87-102.
Lambert, N., Morrison, J. Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A.,
Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Hajishirzi, H. (2024). Tülu 3: Pushing frontiers in open language model post- training..
MacKay, D. J. (2003). Information theory, inference and learning algorithmss Cambridge university press.
Marvin, R., & Linzen, T. (2018). Targeted Syntactic Evaluation of Language Models [arXiv: 1808.09031] arXiv:1808. 09031 [cs], 1192-1202. http:://bi.org/10..8853
Mitchell, M. (2009). Complexity: A guided tour. Oxford University Press.
Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress Measures for Grokking via Mechanistic Interpretability.
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi, P.. Dziri, N., Hajishirzi, H. (2025, January). 2 OLMo 2 Furious [arXiv:2501. 000666 [cs]]. https:///oi.coo http:///ooooor100445555555 0tte:/0048550///xi...55555 21.5500000
Ouyang, L.. Wu, J.. Jiang, X., Almeida, D.. Wainwright. C., Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. & A. Oh (Eds.), Advances in neural information processing systems (pp. 27730-27744, Vol. 35). Curran Associates, Inc. https://proceedings.eemm neurips www.urrps.ccppaperr.eeeeeeeeeeee cc/paper_ http://////leeeeleeeeeeeeeeeeeee 44545333333344499944555554800000 733103350000111110 Paper--neeeeeeeeeeeeeeeeeee pdf Oyama, M., Yokoi, S., & Shimodaira, H. (2023). Norm of word embedding encodes information gain. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1, 2108-2130.
Paninski, L. (2003). Estimation of Entropy and Mutual Information. Neural Computation, 15(6).
1 1191-1253. https://doi.nnn org/ 162/08997660332178822227
Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., & Cotterell, R. (2020). Informationn-hheoret probing for linguistic structure. arXiv preprint arXiv:2004.03061.
Poggio, T., Rifkin, R. Mukherjee, S., & Niyogi, P. (2004). General conditions for predictivity in learning theory. Nature, 428(6981), 419-422.
Pothos, E. M., & Chater, N. (2001). 4 categorization by simplicity: minimum description length approach to supervised clustering.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Di- rect preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Ad- vances in neural information processing systems (pp. 53728-53741, Vol. 36). Curran Associates, Inc. https / proceedings neurips сc / paper - files paper 2023 / file / Pappe--Conference..df Raffel, C., Shazeer, N.. Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 2 2(( 2((14)) 1-67.
Reimers, N., & Gurevych, (2019). Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) and the 9th International Joint Conference on Natural Language Processing (IJCNLP), 198223992. https://dooiiooo http://doi.org/10.. www.oorg//......... 18653/v1/D19-1411
Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5), 465-471.
Sajjadi, M. S., Bachem, O., Lucie, M., Bousquet, O., & Gelly, S. (2018). Assessing generative models via precision and recall. Advances in neural information processing systems, 31. Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019). On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12), 124020. http:://6o..rr//0.1088/// 0tt.0//0000/77744448888333399999
13

---

Published as a conference paper at ICLR 2026
Saxe, A. M., McClelland, J. L.., & Ganguli, S. (2019). A mathematical theory of semantic develop- ment in deep neural networks. Proceedings of the National Academy of Sciences, 116(23),
11537-11546.
Shannon, C. F. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3), 379-423.
Shwartz-Ziv, R., & Tishby, N. (2017). Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.0000
Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information bottleneck method. arXiv preprint physics/0004057.
Tishby, N., & Zaslavsky. N. (2015). Deep learning and the information bottleneck principle. 2015 ieee information theory workshop (itw), 1-5.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A. N., Kaiser, t.,, & Polo- sukhin, I. (2017). Attention is All you Need, 11.
Veldhoen, S., Hupkes, D., & Zuidema, W. (2016). Diagnostic classifiers: Revealing how neural
networks process hierarchical structure, 10.
Vitányi, P. M., & Li, M. (2000). Minimum description length induction, bayesianism, and kol- mogorov complexity. IEEE Transactions on information theory, 46(2), 446-464.
Voita, E., Sennrich, R., & Titov, I. (2019, September). The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives [arXiv:1909.0m 380 [cs]]. Retrieved March 26. 2024, from http://axxivorr//bbb... httpaabss/99990000 0099001334 38380 Voita, E. & Titov, I. (2020, March). information-Theoretie Probing with Minimum Description Lenghh [arXiv:2003.. 12299 [cs]]. Retrieved November 21, 2023, from tttp://axxiv.org/bbb. 2003.1229
Wallace, C. S., & Boulton, D. M. (1968). An information measure for classification. The Computer Journal, 11(2), 185-194, Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo. S., Ren, W., Arulraj. A., He, X., Jiang, Z., et al. (2024). Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang. S.-F, & Bowman, S. R. (2019). BLiMP: A Benchmark of Linguistic Minimal Pairs for English [arXiv: 1912.00582]. arXiv: 101.00922 [cs]. Retrieved January 13, 2020, from http://axxiv.rr/abb//// 2.00822 Wilcox, A. (1967). Indices of qualitative variation. (tech. rep.). Oak Ridge National (ORNLL,,,)
Oak Ridge, TN (United States).
Zaslavsky, N., Kemp, C., Regier, T., Tishby, N. (2018). Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, //5(31), 7937-7942. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, (2020). Bertscore: Evaluating text generation with bert. Proceedings of the International Conference Learning Represen- tations (ICLR). http:://ppeneeeiii...eet/uummmea
14

---

Published as a conference paper at ICLR 2026
A OPEN-WEIGHTS MODELS, DETAILED VISUALS
Token Information Plane
75 Open Weights Models
Expressivily Complexity (Token Bocka!!)
0.30
0.7
0.6
0.2
0.2
Complestt/1/..21
Bigram Information Plane
75 Open Weights Models
Expressivily vs. Complexiy [Bigrem Bockoff)
145
6TM
6//
212
618
610
Comploiity XXC,2)
Figure 5: Token & Bigram Information Plane for Open Weights Models Shown here is the full, labelled token information plane for 75 different open-weights models. Overall while model lie at different levels of complexity and expressivity they broadly approach the IB Bound on optimal compression, hue indicates optimality or proximity to the bound.
15

---

Published as a conference paper at ICLR 2026
Trigram Information Plane
75 Open Weights Models
Exprestiiiy vs. Complesity (Trigram Backofi)
314
3.3
10
035
112
2.20
035
039
015
Quadgram Information Plane
75 Open Weights Models
Expresivily vs. Complexity (Guodgron on Backofi)
0.3
0.25
033
030
Compbaayy 21
Figure 6: Trigram and Quadgram Information Plane Shown here is the full, labelled trigram and quadgram information plane for 75 different open-weights models. Compared with the token case above, here models lie even closer to the frontier. The quadgram estimates are noisy due to sample sparsity, this combined with the fact that all models are close to the bound results in some estimates appearing to cross the bound.
16

---

Published as a conference paper at ICLR 2026
B DATASETS, MODELS, AND COMPUTE
B.1 LICENSES FOR MODELS AND DATASETS
As noted in section 3, we use two datasets for estimation Tulu (Lambert al., 2024) and C4 (Raffel et al., 2020) both of which fall under the Open Data Commons Attribution License (ODC- By) v1.0. Later we use MMLU Pro for behavioural evaluation (Wang et al., 2024) which falls under the Apache License (Version 2.0).
We study a wide array of models, below is license information grouped by model family:
OLMo: The code and model released under Apache 2.0.
Gemma: Released under the gemma license
stated
here:
htnps//aiigoggle.ee/ggeenee
Llama:
Released under the llama license
found
here:
Qwen:
The code and model are released under Apache 2.0.
Aya/Command: Released under the Creative Commons Attribution Non Commercial 4.0
Pythia: The code and model are released under Apache 2.0.
B.2 COMPUTE RESOURCE AND COMPLEXITY
The estimation procedure used here has low complexity for entropy estimator, requiring only dot-prou dot-product and softmax. The majority of compute expense comes from the model's forward pass required to compute the estimate. The complexity of this depends the size of the model. In experiments here estimates required encoding 10,000 samples from C4 and Tulu. This process takes approximately 10, 40, or 70 minutes either 2, 4, H100 GPUs respectively (number required depending on model size). Given this we estimate the total number of GPU hours required for the results in this paper approximately 3,600 H100 hours.
C POSTITRAINING AND PREFERENCE INFORMATION
While LLMs become optimally compressed for next sequence prediction pre-training, the final phase of the training pipeline often introduces other kinds of information. the general case, post- training is designed improve model's 's ability to follow instructions and better align it with human preferences; we look at how this changes the information content of a model, and how it affects the representations from pre-training. Figure 7 shows preference information across two different families of open weights models, Llama and Gemma, which release a checkpoint at the end of pre- training and one at the end of post-training In the Llama case post trained models consistently have higher preference information than their pre- and mid-trained counterparts. This supports framing of pre-training as imbuing the model with core semantic information, which later augmented with task-specific and preference information. With the Gemma models the picture is more complicated, with a consistent effect for the most recent Gemma 3 release post trained models have greater preference information but significant pattern across earlier models.
D ADDITIONAL MODEL TIMECOURSES
A major challenge in studying pre-training is the limited availability of checkpoints. While there are a huge number of checkpoints available for final trained models, intermediate checkpoints over the course of pre-training are relatively rare. We focus analysis in the main paper the OLMo2 models as they offer comprehensive check pointing - and comparatively strong performance. Here we look at two other families of models which make available some pre-training checkpoints. The Smol LM2 models (Allal et al., 2025) released this year are models with 1.7B parameters or smaller that achieve competitive performance. The 1.7B Smol model was trained 11 Trillion tokens and performs comparably to the IB OLMo2 model which was trained on Trillion Tokens. Broadly the 1.7B Smol model follows similar training trajectory the OLMo2 1B model having phases of
17

---

Published as a conference paper at ICLR 2026
gemmo, llama: Preference
Base vs. PosMrained
0.00040
Perrstation leu 0p=0006,mm m7]
Post-Training Increases Preference Information
0.00035
Pederente Palewnte (Prat,))
0.00030
0.00020
Across the Gemma and Llama Families a paired permutation test show significantly more preference information in the post-trained variants.
0.00015
0.00010
0.00005
Base
Instrud /
Posttrained
Preference Optimality Ratio
3.5
Pestroieed
Exead
Proference liFref (x110
2.5
2.0
1.5
Lepebttur
0.8
0.0
6.1
02
0.7
0.8
Optimolty NY: ZM)
Figure 7: Post- Post-Training and Preference Information (Left) Preference information the vertical axis against whether or not the model is post-trained on the horizontal axis, with significance values from paired permutation test above. (Right) Again, preference information the vertical axis against optimality of a model's compression on the horizontal axis axis.
expansion and compression but failing to approach the bound like the OLMo2 7B and 32B models. Pretraining timecourses for the Smol 1.7B model are shown in Figure 8 with token, bigram, trigram, and quadgram backoff. This figure also includes trajectories for the smaller 100M and 400M variants these mosels struggle to show much meaningful compression, though part of the issue may be that check-pointing starts comparatively late in the pre-trianing process compared with say the OLMo2 Models.
The other family of models analyse the Pythia models (Biderman et al., 2023). Timecourses for two Pythia models are shown in Figure 9. Included are analyses of the 1.4B and 6.9B models. In terms of parametrisation these are roughly comparable the IB and 7B OLMo2 models analysed in the main paper. However it's worth noting that the methodology for training these models is substantially different, and that their performance substantially lower than the OL.o22 models, and other more recent open- open-weights models analysed above. In terms of training, Pythia models are intended for scientific analysis, as a result they use the same amount of data, batch size, and number of training steps across model sizes. Perhaps most importantly these models are trained on the Pile dataset (L. Gao et al., 2020). This contains roughly 099,892,766,000 OLMo2 model is trained on 1,000,000,0000000 tokens mcaning the Pythia models see 7.5% of that data. Accordingly the 1.4B Pythia model appears to achieve better compression later in training than its Mo counterpart. As discussed in the main paper there may be relationship between data
by contrast the 1B
18

---

Published as a conference paper at ICLR 2026
Smol LM2 Pre Pre-Training Timecourse
100M, 400M, 1.7B parameters
Taken BockOll
Parometers
Old Foronaters
Token Foromsters
0.15
3.14
Eoominltyy 11:21
2.1
0.15
4.1
0.10
0.11
0.11
0.09
0.00
0.09
0.08
0.07
0.00
Completty
Compleily
Egran BodkOF
0.18 Paromwters
Sgrron tock-Off 0.48 Poramaters
Bigram Back-Off
1,78 Poroneters
0.30
0.000
0.29
0.000
0.20
0.21
9.296
0.27
0:50
0.286
020
0.090
6:05
6:275
6:08
0.270
0.27
0.88
0.59
0.30
0.1
0.32
5.8
5.30
0.81
0:50
0.30
0.54
0:35
0.56
0.29
0.50
0.11
0.22
6.38
Comphotty (0:
Compboityy 12: 2)
Complesity 106
Tégros Bod-Off 0.18 Porovaters
Trggen Beck 0.48 Paronolers
Tigros Bock 178 Peronaters
0.305
5/220
0.300
2.400)
Eyyrakky 21
5.405
0,375
Seriibyy
0.370
5.00
0.4129
0.413
04100
0.300
5.000
5.025
0.375
0.275
0.300
0335
0.300
0.313
0.00
0.00
0.28
0.43
0.600
2,500
5.10
0.112
0.000
2.03
0.05
Exeeloolyy 106
Geophely
Gompbelly 130
Quodgram Back-Ofi
0.18
Quodgron BackOff
Q.48
Guodgran BackOH
0.414
0.422
2.075
0.44%
Expreniitt 19:
23
0.400
6..09
D/4425
0,400
0.008
2,400
0.066
DA4775
2.00
0.405
0.438 0.410
0.41
0.14. 0.416 DJIE 0.403
0.403
0.465
0.00
0.475
0.000
0.435
0.455
5,400
0.465
0.00
0.27
Comploity 11:
Compliity 12
Complnity 133
log Training Step
522
Figure 8: Smol LM2 Timecourses
complexity and the model complexity needed in order to achieve substantive compression of it. By contrast the 6.9B Pythia model is still expanding representations late into pre-training; this would appear to indicate it is under-trained.
19

---

Published as a conference paper at ICLR 2026
Pythia Pre-Training Timecourse
1.4B, 6.9B parameters
Tokan todkOff
1.4 Parcreeter
Tobes SocbOH
A Poromaters
0.200
0.50
Pythia models are trained on orders of magnitude fewer tokens than the OLMo2 Models shown in the main paper only 7% of the
0.133
0.000
2.005
0.091
0.000
0.000
0.005
9.000
0.000
OLMo2 data. Additionally unlike other LLM families all sizes see the same amount of data, in the same batch size, for the same number steps.
Compleity US: 4]
Complexit 100
Bigror Bock-lll 1.45 Porometers
Sigoon Bo 6.98 Porameters
2.0
0.30
0.1
0.10
Accordingly here, the Pythia 6.9B model appears undertrained, still expanding representations even late in training. Having seen the same amount of data the smaller model follows the expected two phase trajectory. Here the 1.4B pythia model may achieve better compression given it is trained on only a small fraction of the data OLMo2 1B is expected to compress.
2.00
Complocky
Complexly
Trlyon BodkOfff 1.48 Fonoweters
Triroo teck 08 Formeseer
2.40
0.35
1.30
0.25
0.20
0.16
0.38
036
Compladyy
Gsodgran BockOOl 1.48 Porameters
Quodgron BackOR 6:98 Paraneter
2.00
3.26
0.30
0.50
0.22
0.28
220
0.1
0.10
Compleily XX
Log Trakning Step
Figure 9: Pythia Model Timecourses
E ENTROPY ESTIMATION
E. ESTIMATOR HYPERPARAMETERS
The estimator has two parameters that need to be set the number of bins and the temperature used in the softmax E. As shown appendix G.2 and (Conklin, 2025), the estimator is generally robust with respect the number of bins in all experiments presented here we m 100.
20

---

Published as a conference paper at ICLR 2026
E.1.1 TEMPERATURE CALIBRATION.
Naively we could use the same temperature across all models, however models broadly differ in the dimensionality of their hidden representations. Within self-attention as the dimensionality da of query and key vectors grows, the variance of their dot products scales linearly with dz. pushing the softmax function into saturated regions where gradients vanish (Vaswani et al., 2017). This specific instance of the broader concentration of measure phenomenon in high-dimensional spaces, where distance and similarity metrics become increasingly uniform and less discriminative (Aggarwal et al., 2001). As a result Vaswani et al. (2017) scales the dot product by vd to avoid saturation. The soft entropy estimator uses dot-products on the surface of a the unit hypersphere passed through softmax in order to estimate density. As a result, for fixed temperature, higher- dimensional space will begin to appear more uniformly distributed. We compute a temperature which is calibrated to prevent saturation making estimates of for different dimensionalities directly comparable.
Let Ve denote the Mises- (vMF) distribution on Sd-1 with concentration parameter 1/e; by rotational symmetry, DKL (VElU) depends only on E and d, and measures how far a single vMF kernel is from uniform. For an arbitrary data distribution P on the sphere, let P. denote the convo- lution of P with the vMF kernel at temperature E; the smoothed KL divergence Dxr, (Pi((P:)U)) is the estimation target, and satisfies 0 I DKL (IU) & DKL (Vell)).
DKL (el( um) takes values in [0, log ml, where m is the number of bins. To ensure this range is well-matched to the estimation target DRL (P: we calibrate the temperature E so that the maximum possible value of the target equals log m. The target is bounded above by DKL (VEll)) the KL divergence of single von Mises-Fisher Fisher kernel from uniform on the unit hypersphere gd-1, Direct computation of DKI. requires evaluat- ing modified Bessel functions, which is numerically unstable at large d: however, using Amos-type bounds on Bessel function ratios (Amos, 1974), one can construct upper and lower envelope func- tions dr satisfying VE,d= S DKL (VV) S 4 ee,d with gap of order O(d-a). Both bounds are strictly monotone decreasing in for d > 4, so the equation DKI. (VelU)) log m has a unique solution. To leading order in d, this solution is e (m, d) = 1/ log m, which we use throughout our experiments.
For the soft entropy estimator ((Q))
Ex (m, d) =
v 22d log
In practice this is computed once per model based on m = 100 and the model's dimensionality. The correction here bears strong resemblance to the default scaling within self attention vd.
E.2 APPROXIMATING THE INPUT DISTRIBUTION
We estimate the mutual information between model inputs and outputs. In an auto-regressive decoder-only LLM the input to a model is the preceding context up to the current token. We view the input as n-grams of tokens where the input at timestep 2; is ngram of width / containing all tokens To... Maintaining probability distributions for every possible context proves intractable due to the combinatorial complexity of natural language. Additionally, ngrams greater than tokens become sparsely distributed in the data making reliable estimation of their probabilities a challenge. As result we condition estimates on ngrams of fixed-widths 1, 2, 4 referred in the paper as token, bigram, trigram, quadgram. This is related to Backoff (Katz, 1987) which reduces n D-gram size until the n-gram has non-zero probability in corpus. Here though we do not interpolate dif- ferent n-gram widths, instead maintaining separate aggregate estimates for each width in part to be able to study how different levels of contextual information are represented in the model. Where given n- n-gram, like quadgram, does not have non-zero probability in the data it is omitted from overall quadgram mutual information estimate.
In practice this means estimates for smaller n-gram widths are more reliable a classical issue in lan- guage modelling (see Jurafsky & Martin, 2000, p.32). Token, bigram, and trigram estimates can be estimated reliably from relatively small sample of data. We judge this by looking at how estimates change as a function of the number of samples during the estimation procedure, by 5,000 samples these estimates relaibly begin to converge. Quadgrams, due their sparsity, tend to have less robust
21

---

Published as a conference paper at ICLR 2026
X
have of late but wherefore
model
sebeddings wherefore vos the
aherefore where
know not lost all mirth y, You Fnz Yuadin Fut Fnd
embeddinge
Z
iipus, and was preceded
y
Z|x
x,
[[x:Z] = H(Z) x
mutual information between trout token and representatione ve get cabsoliige correngendang This than untracted fron
points
of all
P(Z)
PlZ|x]
P(Z) x.x..
token rabeddings. he can take centext into atcount shere sepoifáo procseded
1x
flnnd resutt ne focus anslystson token, bigran, and trigram contact. Inclucing quadgrams in Fiquie
and
by quaderans the data becomss incresstnglly sparis reference
Figure 10: Illustration of conditional probability estimates. An example sentence is provided, assuming word-level tokenization for simplicity. At left are the indices for the input and output tokens when the current input word is wherefore. At right is shown the sub-setting procedure for estimating conditional probabilities. This illustrates how bigram estimates do not compute entropy of token embeddings, rather the embedding for the current token embedding conditioned on preceding context.
estimates additionally the number of labels grows quadratically with each additional ngram width making quadgrams challenging to estimate for larger models with larger vocabularies. As result our broad comparison of open-weights models uses token and bigram estimates. Additionally the pre-training model size analysis here focusses on trigram estimates (figure 3) as this widest context that still reflects reliable estimate. The analysis of how context represented over pretraining (figure 2) includes quadgram estimates for reference.
E.3 APPROXIMATING THE OUTPUT DISTRIBUTION
While during inference models predict the next token given preceding context, this is distinct from how they are trained. During training of an auto-regressive decoder-only LLM causal masking means token only attend to preceding context, not trailing context. However transformer decoders are trained using teacher forcing, where predictions are generated for the entire sequence made correctly. This is instead of having training operate
in parallel by assuming predictions one token at a time with a separate forward pass for each - which how predictions are generated during inference. The result of this is that for an embedding e, at timestep following embeddings 6x41 can attend to et. This means embeddings get gradient information from the trailing context. Put another way, the prediction for output V/+1 written in terms of e, a result the gradient information from the next token(s) in sequence V Lo (Vo+1) affect the embedding at the current timestep.
Given that our analysis computes embedding mutual informations over training with respect to model's input and outputs this fact has implications for us. It means that the output for e, is not just yt+11 1 but all following output tokens Vt+1...bh where n is the sequence length. This is because e, receives gradient information from the loss with respect to predicting all following tokens sequence. As result we consider X to be the entire preceding context in the input (as mentioned above), and Y to be the entire trailing context after the current point in the sequence. This means when compute mutual informations for different n n-gram widths match the width for X and Y conditioning the estimates on the same width of preceding and trailing context respectively.
E.4 ESTIMATING MUTUAL INFORMATIONS
To compute mutual informations between the input X and representations Z, we need two quanti- ties: the entropy of representations H(Z) and the conditional entropy given the input H(Z)X) To
22

---

Published as a conference paper at ICLR 2026
compute H((2) the quantisation procure described in section 3.1 applied to all token embed- dings which gives Z by summing over the each embedding and renormalising we get categorical distribution P(Z) that describes the embedding space. To get conditional estimate P() we simply take 2 and compute subset, containing the embeddings corresponding to the input X, X Summing and renormalising gives the distribution P(2 X).
This brings us to an important distinction, our analysis discusses mutual informations with respect to tokens, bigrams, trigrams, and quadgrams. These not computed over different widths of embeddings, but rather over single token embeddings conditioned on the preceding context. In the same way XX) ) is computed as subset of P(Z), as we condition on further context we can subset the embeddings further. Figure 10 gives high-level illustration of this process. It means that Z token is a subset of Z, and |biirrm is subset of Z |token, 2 trigram is a subset of 2 |bigram etc. This means the terms token, or bigram mutual informations refer to the width of the conditioning context, not the width of the embeddings over which entropy is computed.
E.5 CONDITIONAL MUTUAL INFORMATIONS AND THE RESIDUAL
In order to compute what proportion of a model encodes each level of context we use the chain rule for mutual information. As we increase the context width used in back-off the estimates contain each other the bigram mutual information includes the token mutual information. This means when estimating mutual information with the source distribution with token back-off we look at only the current token T( [(Z while bigram mutual information adds the preceding token (7; 2t-1). The chain rule here means:
I(x 21-1) = [ ((;
+
(8)
which allows us to separate out the information explained by the current token [Z; x2) and the preceding given the current token I(Z; 21-1|2))). For the source distribution X and given n-gram width n we can get a proportion Φ of model information by normalising by the entropy of the model.
6(I, n) 1(Z; xn (11))n)))
(9) We compute this for each level of backoff, where at the token level Φ(x, 1) ((2;z)/H((Z) The most granular label we have in the experiments here quadgram. The residual, or unexplained information, is the information in the model left after subtracting the mutual information of the most granular category.
H(Z)
Øresidual (I., Remax) = (
(10)
Results showing proportions of model information broken down in ngram can be found in figure (bottom).
E.5.1 CONDITIONAL MUTUAL INFORMATIONS AND PERFORMANCE
Shown in Figure 11, less token information, but higher proportion of local contextual information relates significantly to downstream performance. the main paper we report token level back-off in the correlation results, where lower complexity is related to performance.
E.6 ON THE USE OF SHANNON ENTROPY
In this paper compute the entropy of continuous latent variables. As result it is natural to ask why we in line with previous work (Sajadii et al., 2018; Shwartz-Ziv Tishby, 2017: Voita et al., 2019) opt instead to discretise representations and compute their Shannon entropy (Shannon, 1948). There are two major reasons for this; first, differential entropy is not the true continuous analogue of Shannon Entropy (Jaynes, 1957). This shown by the fact that differential entropy
23

---

Published as a conference paper at ICLR 2026
Context vs. Performance
More context information relates to performance
Take
(-0000)
Guddpan H837 EXI
<-: br1721
40
20
30
10
10
10
10
Complextl (Rasiduol)) vo. Evals
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.3
0.4
0.2
0.
0.3
Hall jes
0.6
0.4
0.4
62
0.2
0.2
6.0
0.1
0.2
0.05
0.10
0.15
0.20
0.05
0.10
0.15
0.02
0.04
0.06
0.5
0.6
0.7
Complaiyy
Gompbelyy
Complenly
Gemplenly
Compleiiy
Model
26
Ovent A3
Guedt mi
Ove
2-3
fdd Ovend
2.79
Ovend
Figure 11: Proportions of Information Vs. Performance: Across 47 models less token informa- tion and more local contextual information relates signifcantly performance based spearman correlation (significance indicated by hue).
D(X) unbounded -0 D(X) S oc, and variant under linear transformations. This is the main motivator for information theoretic analysis to discretise and use Shannon entropy directly. A secondary consideration is that we don't 't know how embeddings are distributed, so in order to get a differential entropy estimate we would first need to fit distribution to the data. At scale this fitting step can be expensive, and introduce topographic assumptions. While discretisation is imperfect it enables the use of Shannon entropy, and makes minimal topographic assumptions.
E.7 SCALABILITY OF PRIOR WORK
Notably Shwartz-Ziv and Tishby (2017) perform an empirical information theoretic analysis of neural-networks trained on MNIST. To do they perform dimension discretisation of model embeddings. This turns a 16-dimensional vector into a 16 character string. They then convert this to a categorical distribution over all possible strings. This gives single categorical distribution that can describe representations at particular layer in a network. This approach to discretisation works well on small problems they study feed-forward networks trained on MNIST. However the dimension- wise discretisation requires taking hidden representation with dimensions butch hidden and transforming it to batch x hidden x n bins. If using bins, in practice this means using 50 times the memory of not discretising. For the OLMo2 32B model used in this paper which has a hidden dimension of 5120 and 64 layers, and where we have a context window of 512 tokens, this would require holding in memory tensor of dimensions batch 512 x 5120 44 n bins. The memory use of this approach makes it intractable to apply to contemporary models and the problems studied here.
Voita et al. (2019) studied the transformer base model which has only 6 layers with a hidden di- mension of 512. Despite this they note the approach from Shwartz-Ziv and Tishby (2017) was not
24

---

Published as a conference paper at ICLR 2026
tractable apply to the model. They opt instead for quantising representations via clustering, based on related work from (Sajjadi et al., 2018). This method runs a clustering algorithm (Voita et al. (2019) use mini-batch k-means), then treats each cluster where density is assigned proportional to cluster membership. While this method provides robust entropy estimates, and dramatically less memory usage than the approach from Shwartz-Ziv and Tishby (2017) it still has relatively high computational complexity. It requires running a clustering algorithm to convergence, before performing quantisation prohibiting its use in an online setting you need the cluster centroids before you can assign embeddings to them. Again thinking of the OLMo2 32B model used here, this would require running a clustering algorithm on 5120 dridemi- mensional spaces, at all 44 layers separately, for each of the 150 pre-training checkpoints. This would provide the 'bins' for the quantisation, then embeddings would need to be assigned to bins, requiring a second forward pass.
event in categorical distribution
In practice an cetemmattoo--hhoretti analysis of an LLM requires an entropy estimation method that is memory efficient, fast to compute, and can be applied in online setting requiring a single forward pass and no caching of the embeddings. The only estimator we' re aware of that meets these criteria is the soft-entropy estimator (Conklin, 2025). Here the quantisation requires only cosine- similarity and a softmax making it fast and memory efficient. Additionally the normalisation step means 'bins' can be computed once the start of the analysis, rather than needing pass through the data to fit clusters or fit the support of the model's 's distribution. Conklin (2025) notes that the use of cosine similarities means this method considers only angular information in the representation space. While euclidean distances can be used instead, this would require first estimating the support of the distribution to fit the 'bins' making online estimation challenging. However use of cosine- based methods is standard practice in NLP (T. Gao et al., 2021; Reimers & Gurevych, 2019; Zhang al., 2020), with some work suggesting vector norms in LLMs predominantly encode frequency information (e.g. Oyama et al., 2023).
E.8 THE INFORMATION BOTTLENECK BOUND
The Information Bottleneck bound is the curve traced by varying the trade-off parameter 33 in:
[(x
(11)
The curve this traces is where representations are optimally compressed. Along this bound x( is an optimal encoder, from inputs to representations, preserving only the information in X relevant Y. For given dataset this optimal encoder can be found numerically via a version of the Blaut- Arimoto (Arimoto, 1972; Blahut, 1972) method for computing channel capacity. Introduced in Tishby et al. (2000), the information bottleneck method for determining channel capacity relies on three equations:
ps(z) Zz(x) exp -BDip(ll)))lsss
ps(2|x)=
(12)
p=(=)= = Σ ((x))s(2|<)
(13)
#EX
p0(0|2)
Σ aEX
ps(z(2)p(ulz)
(14)
These equations satisfied self-consistently at the bound. As these three equations rely on each other one can learn an optimal encoder by starting with a randomly initialised one. Then iteratively computing each of these equations in turn.
In the general case the shape of this bound follows a linear relationship. until all mutual information between and y is captured. At this point the curve saturates additional complexity doesn't 't result in additional accuracy, as there's 's no more predictive information in æ. This means numerical computational bound is largely important for computing where it saturates.
25

---

tractably compute Published as a conference paper at ICLR 2026
Pre-Training Loss vs. Compression Optimality
3.5
30
During pre-training models experience dramatic reductions in loss early on before this begins to saturate.
2.5
Cos
1.5
As the loss begins to saturate the representations begin to approach the bound.
1.0
0.5
0.84
Expressivity IYY; ZJ Complexity IDC
Figure 12: Cross-Entropy Loss vs. Distance to bound. Shown on the vertical axis is the OLMo2 7b model's cross-entropy loss on 10,000 examples from c4. On the horizontal axis is the ratio between /(Y; 2) and T [X; ; Z) which indicative of how close a representation the IB bound. Models hegin to compress and approach the bound as the loss saturates.
However numerical computation of the bound in our setting proves intractable. Here the optimal encoder p(z[æ) needs to map all of natural language to representations that optimally predict the next token. This an exceedingly challenging problem for an iterative numerical optimizer it's problem that ordinarily requires large language model. In experiments we are able to compute bound for tokenizers up to 50,000 tokens, however past this point convergence begins to fail. In our setting this process takes tokenizer and 300,000 sentences from c4 to get maximum likelihood estimate of P(x), P(y), , ((y(æz) on data representative of model's s training data. We can then iteratively compute (( (z)) until convergence. In experiments this iterative procedure converges to the expected saturation point- where 1(2; Y) = Y).
Given that would like have bound for problems where numerical computation of it proves intractable, we leverage this pattern by assuming the bound follows a linear relationship until the sat- uration point where Y) I(X;Y) Y). The largest tokenizer for which this quantity has a normalised / [(X; Y) of 0.7 (where 1.0 is the maximum possible value). Across all open weights models the highest token complexity converged to is 0.15, well below the saturation point. This is in line with results from (Shwartz- Ziv & Tishby, 2017), which shows FFNs on MNIST only converge near the saturation point when over-fitting.
F RELATING COMPRESSION TO TRAINING LOSS
Prior work (Shwartz-Ziv & Tishby, 2017) shows that models transition from the fitting phase where T(Y; Z) increases, to the compression phase where T(X; Z) decreases when empirical error on the training distribution saturates. Their setting is substantively different to the one studied our work the most relevant differences here are that they analyse a feed-forward model trained on MNIST for multiple epochs, meaning the model's performance can fully saturate in-distribution. In an LLM setting models are trained orders of magnitude more data, often for a single epoch passing through the data only once, meaning saturation is more graded. However worth asking the transition between fitting and compression relates to training performance in an analogous way.
We compute the cross-entropy loss for the OL.o22 model performing next token prediction on 10,000 examples from C4. C4 is substantive component of the OLMo2 pre-training data (OLMo et al., 2025) and so gives us a proxy for in-distribution performance on the model's training set. This follows previously attested dynamic, where earlier steps dramatically decrease the loss before this begins to slowly saturate. Unlike in an MNIST setting this objective never truly saturates, instead slowly flattening. Figure 12 shows this loss plotted against the ratio between expressivity I(Y; Z) and complexity [[X; Z). As noted in section 4.2 this ratio acts distance to the bound where models optimally compressed as this quantity approaches 1.0 models approach the bound. Figure 12
26

---

Published as a conference paper at ICLR 2026
Robustness to Number of Reference Points W
0.18
0.18
Halving the number of points from 100 to 50 changes the exact mutual information values but not the overall observed effect
0.16
0.16
Expressvity (16
0.14
Expresiity 11
0.14
0.32
0.12
0.10
0.10
0.58
0.00
0.10
0.12
0.14
0.16
0.8
0.08
0.10
0.12
0.14
0.16
0.88
Complexity 10:
Complexiy II
Robustness to Data Distribution
data
deta mede
0.18
0.78
0.18
0.16
0.36
(1)
Caprasviyyi I1f
Expesiiity
0.12
0:22
0.12
0.10
0.10
0.10
0.08
0.10
0.16
0.13
0.14
Comploxit
Complexiy
Figure 13: Estimator Robustness to number of bins and data distribution. Shown are trajectories through the information plane for the OLMo2 7b model. (Top) trajectories in the main paper 100 reference points Wi per layer, here 50 points are used, and show the same overall two-phase pattern. (bottom) estimates in the main paper with respect e4 given its resemblance to LLM's training distribution, here are show trajectories computed with data from tulu and MMLU pro which show the same broad two-phase pattern. Hue indicates log tokens in billions over the course of pre-training.
shows how models begin approach the bound the loss on begins saturate. This broadly aligns with Shwartz-Ziv and Tishby (2017), where saturation in-distribution relates to the transition to compression.
G ESTIMATOR ROBUSTNESS
Our work does not introduce the soft entropy estimator but is the first to apply it in this context. As result we run some robustness experiments see how the results vary under different hyper- parameters and data distributions.
G. ROBUSTNESS TO DATA DISTRIBUTION
Core results in the paper show information plane trajectories computed using C4 as this dataset forms a substantive part of the pre-training data for the OLMo2 models. To verify that the overall pattern of expansion and compression is robust across data distributions analyse the pre-training checkpoints of the OLMo2 7B model across data from C4, Tulu (Lambert et al., 2024), and MMLU (Hendrycks al., 2020). Figure 13 (bottom) shows the pre-training trajectories for the 7B model
27

---

Published as a conference paper at ICLR 2026
0.302
0.217
mean
Expresilyy lit:2
Eapeablyyl itt.
-3
0.122
0:344
0.21
15
0.317
0168
6.807
0.044
0.222
0237
0.312
0.217
6.189
0.72
0.76
Compleiity IIR
Compledyy DX
Complesity 155 4)
0.330
expectalion
0.335
(1.21
0.200
Exprebiiyy 007 21
0.100
50
Expreuiity KYC
0.330
0.156
23
0114
-10
0.700
0.200
0.122
15
0.355
1.86
0.10
0.300
0.148
0.20 0.15 0.300 0.335
9.200
0180
0,100
0.000
6.905
3.500
Complenty 136
Complesiy 1X
Compledty (R;
Figure 14: Pre-training Time-courses Computed with a Mean vs. Expectation Shown are tra- jectories through the information plane over pre-training for the OLMo2 1b, 7b, 32b models. These analyses use mean (top) or expectation (bottom) in computation of mutual information. The ex- pectation used the main paper as it reflects true mutual information. Hue indicates tokens in billions over the course of pre-training.
computed with token backoff across all three distributions. The broad two-phase pattern proves consistent across all of them with fitting phase followed by compression phase where models approach the bound. There are individual variations for each dataset, with Tulu and MMLU having higher mutual informations than C4. This may reflect that MMLU and Tulu are more domain- specific than C4 which is a broad crawl of the internet.
G.2 ROBUSTNESS TO NUMBER OF REFERENCE POINTS
The Soft Entropy estimator relies soft-quantisation of model's embedding space, whereby each representation softly assigned to n points w; which are sampled uniformly at random from the surface of the unit sphere process described in equations 2 and 3). Experiments in the paper use a n = 100. Here we show the core 7b model pre-training time-course computed for c4 with token backoff using n 100 and n 50. The results show the same overall pattern of expansion and compression with small changes to the exact mutual information values. Given this estimator resembles a differentiable relaxation of binning- based approaches increasing the number of bins reduces mutual information by assigning similar representations to an increasing number of different bins (Paninski, 2003). The results seen here are consistent with this effect with 100 points achieving slightly lower mutual informations than 50 points Conklin (2025) note when introducing this estimator that this interaction can occur, but in benchmarking experiments show the soft- asst-nssignment process makes this estimator more robust to number of 'bins' than existing clustering based approaches.
estimate, it is relevant to note that in binning
G.3 LANGUAGE IS A LONG-TAILED DISTRIBUTION - COMPUTING 'MUTUAL INFORMATION' WITH MEANS
As noted in section 3.1, the quantity estimated here is mutual information which uses an expectation over conditional entropies rather than mean.
28

---

Published as a conference paper at ICLR 2026
[(x; := = H(Z)- - Σ P(X P(X==)H(([XX = x)
(15)
mEX
Here we recompute the core pre-training analyses for the IB, 7B and 32B models using a mean detailed in equation G.3 language is known to be zipfian distributed small number of high-probability patterns likely drive worth noting when using mean the resulting quantity the true
how treating each event equiprobable effects the analysis. Given
the mutual information. mutual information, and so the information bottleneck bound does not necessarily apply.
T(x; 2)
H(Z) - - = x) #EX
Σ
(16)
X
As shown in figure 14, estimates computed with the mean, and with the expectation show the same broad two-phase pattern. with models first expanding representation phase before compressing to- wards the bound. When taken as a mean the quantity reflects the mean mutual information per label - like mean mutual information per token rather than being weighted by the exponentially distributed token representations.
29


Reviews:
[{"review_id": "Ee3hHABVAP", "reviewer": "Reviewer_niuc", "summary": "This work frames large language models as lossy compressors that retain only information essential to their training goals. The paper tries to show that that pre-training drives models toward Information Bottleneck optimality, and differences in compression across model families reflect data and training choices. Very limited and insufficient empirical studies are provided to support the claim.", "questions": "See weaknesses.", "limitations": "", "rating": null, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "None noted.  See explanation in the weaknesses.", "weaknesses": "1. The authors ignore that the understanding LLMs from the perspective of information bottleneck (IB) has been explored in the literature, such as [1-3]. These existing works understanding the compression effect of variants of LLMs by IB, and none of these works are cited and discussed in this paper. The main claimed contribution of this paper, the compression effect of LLMs from an IB persecuted, is not new and already discussed in these existing works.\n\n2. While this paper was submitted as a \"learning theory\" paper, no formal theoretical results are presented and proved, far below the standard of major machine learning venues including ICLR.\n\n3. Again, this paper lacks a key formal definitions to support its main claim, While \"optimal compression\" of LLMs are mentioned across this paper for many times, there no formal result about when \"optimal compression\" can happen (that is, minimum of Eq. (1)), and there is no result about why training LLMs can achieve such \"optimal compression\".\n\n4. The formatting and presentations are chaotic, far below the bar of the a major ML venue. A lot of sentences are ungrammatical, and there are a lot of formatting issues (large space between equations or paragraphs).\n\n5. This paper also lacks sufficient empirical justification: experiments on large-scale LLMs on standard benchmark should be used to support the claims, such as LLaMA-4 and Qwen-3.\n\n\n[1] Protecting Your LLMs with Information Bottleneck. NeurIPS 2024.\n\n[2] An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation. ACL 2024.\n\n[3] QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory. EMNLP 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0Q9GJWJss", "reviewer": "Reviewer_WLPL", "summary": "The paper investigates LLM training through the lens of Information Bottleneck (IB) theory, framing learning as a process of lossy compression of input information. It introduces an information-plane analysis that tracks the mutual information between model representations, inputs, and outputs during training, revealing a two-phase trajectory: first, expansion, then compression of representations. The authors develop a soft-entropy-based estimator to compute mutual information efficiently at LLM scale, enabling comparisons across model families and token contexts. Their empirical results show that models approaching optimal compression achieve higher accuracy on downstream benchmarks such as MMLU-Pro, and that post-training fine-tuning primarily increases human preference information rather than token-level complexity.", "questions": "Under matched compute/bit-budget/latency, how does the proposed “compression optimality” metric predict retained accuracy compared with established compression methods?\n\nHow robust are the information-plane trajectories to the choice of MI/entropy estimator? Can you benchmark your soft-entropy approach against common alternatives (e.g., MINE, CLUB)?\n\nWhat concrete theoretical insight or training principle, beyond prior claims that “language modeling is compression” and IB-style dynamic, does this paper introduce?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a clear, interpretable information-theoretic framework for understanding representational dynamics of LLMs.\n\n2. This paper evaluates multiple open-weight models, scaling classical IB analyses to billions of parameters.\n\n3. This paper provides a readable exposition with intuitive information-plane plots.", "weaknesses": "1. The core argument that LLM training is a process of lossy compression approximating the Information Bottleneck largely restates conclusions from prior work, such as Language Modeling is Compression [1] and Norm of Word Embedding Encodes Information Gain [2]. The paper extends scale but does not introduce a new theoretical insight or training principle.\n\n2. No comparison with established compression baselines. The study fails to relate its “compression optimality” metric to practical model compression methods like pruning [5, 6], quantization [7, 8], or low-rank adaptation [9, 10]. Without such baselines, it is unclear whether the proposed metric offers predictive or practical value for compression performance.\n\n3. The observed correlation between compression optimality and MMLU-Pro accuracy does not imply causality. Unlike controlled studies of feature emergence or SGD-noise-driven IB dynamics [3, 4], the paper lacks interventions demonstrating that manipulating compression affects downstream accuracy.\n\n4. The entropy estimation approach is based on a soft-entropy approximation but is not compared against common mutual-information estimators such as MINE or CLUB. Without quantitative validation, the robustness of the reported information-plane trajectories remains uncertain.\n\nReferences\n\n[1] Delétang et al., Language Modeling Is Compression, NeurIPS, 2023.\n\n[2] Oyama et al., Norm of Word Embedding Encodes Information Gain, EMNLP, 2023.\n\n[3] Tishby and Zaslavsky, Deep Learning and the Information Bottleneck Principle, ITW, 2015.\n\n[4] Saxe et al., On the Information Bottleneck Theory of Deep Learning, J. Stat. Mech., 2019.\n\n[5] Frantar and Alistarh, SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, arXiv, 2023.\n\n[6] Ma et al., LLM-Pruner: On-the-Fly Structured Pruning for Large Language Models, NeurIPS, 2023.\n\n[7] Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers, arXiv, 2022.\n\n[8] Lin et al., AWQ: Activation-Aware Weight Quantization for LLMs, MLSys, 2024 (preprint 2023).\n\n[9] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022.\n\n[10] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGNsGfHAaV", "reviewer": "Reviewer_jNtt", "summary": "The paper proposes a method to frame pre-training as lossy compression, by leveraging the representational space, i.e. the learned representations with information-theory methods. They use entropy and mutual information between input, hidden representations and output, using a scalable soft-entropy estimator, to compute complexity (compression) and expressivity (task information). The authors  demonstrate that during training LLMs, there are two phases under the Information Bottleneck theory perspective: an initial expansion of expressivity followed by a compression phase. They propose that models approaching the optimal compression bound tend to perform better on downstream tasks. The approach is novel and well-motivated, and the findings regarding compression during training are a valuable contribution.", "questions": "- Have you analyzed whether the correlation with downstream performance is primarily driven by I(Y;Z) alone, or does the expressivity/compression ratio provide additional predictive power? Could you provide results or discussion to clarify this point?\n- Are there other practical implications of your findings for LLM training? You could place here the questions from W4.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **S1** While the soft-entropy estimator is not a contribution of this paper itself, it seems to be the first to explain and implement the approach from Conklin (2025) to compute entropy and mutual information within LLMs to distinguish between complexity (compression) and expressivity (task-relevant information).\n- **S2** The use of such metrics to track the behaviour of LLMs during pretraining is, to the best of my knowledge, novel. While the results for smaller models are inconclusive, the results for olmo2 7B and 32B during training align with the Information Bottleneck theory and paired with the results at the end of training for a vast array of models constitutes a welcomed new perspective on LLM training.\n- **S3** The paper is clear in its motivation for why information-theoretic analysis is needed for understanding LLMs, and it provides a nice overview on representation learning in the context of compression.\n- **S4** The proposed methods to assess expressivity and compression during training are likely to spark future work exploring these further.", "weaknesses": "- **W1. Attribution and Contextualization of Contributions**\n   - One of the main contributions of the paper is the implementation of the theoretical framework from Conklin (2025) to estimate soft-entropy. You should better contextualize this, not to diminish your own contributions, but to clarify that much of Section 3.1 is a reinterpretation of prior work.\n   - Then, emphasize that this paper is the first to leverage the soft-entropy estimator in the context of LLMs.\n\n- **W2. Inconsistent Level of Detail**\n  - The level of detail varies across the main body, which may be challenging for readers who are not experts in compression.\n  - For example, Section 3.2 glosses over the n-gram backoff mechanisms and provides little explanation of I(Z;Y) in the main text.\n  - Readers would not know that I(Z;Y) is better explained in Appendix E.2 from reading the main body\n  - In that vein, the paper seems to expect readers to assume that the mechanisms for I(Z;X) and I(Z;Y) are identical (see last sentences of E.2), but this is only made explicit in the appendix.\n  - Meanwhile, the Introduction and Background sections are lengthy (up to page 5), and could be condensed to make room for more technical details that are currently omitted or relegated to the appendix.\n\n- **W3. Section 4.2 Results and Interpretation**\n  - The results in Section 4.2 are not fully convincing.\n  - While it is good to see other use cases for the mutual information values, this section feels lackluster in achieving its goal.\n  - It seems almost granted that I(Y;Z) will correlate with downstream performance.\n  - The paper argues that the ratio with compression (expressivity/compression) is important, suggesting that two models with the same expressivity may not generalize equally and that compression may be the key difference.\n  - However, the paper does not fully explore whether it is solely I(Y;Z) that is driving the correlation, and the claim of \"optimal compression\" may be more semantic than substantive without further analysis.\n\n- **W4. Practical implications**\n  - The paper focuses on empirical results, reporting results for a vast array of models, and checkpoints for the case of Olmo/Pythia.\n  - However, not that much emphasis is given to practical implications (beyond what is discussed in W3).\n  - This seems a missed opportunity. For instance, could these findings be used to predict early stopping? \n  - What about when to expect the change between phase1 and phase2? Are there any implications related to that in terms of performance? (i.e. does the point in which a model start compressing, predict performance?) It does seem that larger models take proportionally more steps in phase 1. \n  - Going back to W2, some content could have been condensed to leave place for more practical implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ji0udwHYsv", "reviewer": "Reviewer_FQUJ", "summary": "This work applies the idea of information bottleneck to study LLM training. The authors propose an entropy estimation method and use it to quantify the mutual information between the representations and inputs/targets. By studying these quantities, they conclude that LLM training consists of two phases: a fitting phase and a compression phase, so training can be viewed as lossy compression. They further show that their framework reveals a correlation between model performance and the optimality of the compression.", "questions": "1. In the context of the information bottleneck, there were several papers that questioned the relation between compression and generalization. Could the authors add a literature survey on that topic and elucidate what those papers got wrong?\n2. Is the method used for estimating mutual information similar to the Nystrom method in kernel methods?\n3. Is there any prior work that also measures compression of token embeddings and looks into whether such properties correlate with generalization?\n4. Do we have a reason to expect that the method would actually work once the number of points w_i is small (which is required for estimating mutual information based on the resulting softmax probabilities)?\n5. Did you conduct experiments testing the robustness of observations to the number of points w_i in eq. 2?\n6. Does expressivity alone predict performance? What is the correlation between them?\n7. How do you expect the trajectories to change as the context length used to estimate entropy increases?\n8. Does data quality impact the presence of the two phases?\n9. In the original IB paper, the \"knee\" in the curve was due to the interpolation of the training data (transition from getting zero error on ERM to compression). Could you show that the same thing occurs in your experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The work applies the interesting idea of information bottleneck to study models/datasets at much larger scale than prior works.\n2. A tractable method for estimating entropy at LLM scale is used.", "weaknesses": "1. The existence of two phases is not clear from the figures. How do we know that the change in I(Y; Z) and I(X; Z) is significant?\n2. In Figure 1, the two phases on the left and right don’t quite look similar.\n3. If the changes are indeed significant, then the compression phase seems to consist of a significant decrease in expressivity (mutual information with the label). This casts doubts on a) whether the entropy is estimated accurately enough, and b) whether the “compression phase” is really about compressing irrelevant information between X and Z.\n4. In Figure 1, the x and y axes have numbers between 0.2-0.22. Doesn’t that imply that there’s barely any difference in the mutual information throughout training? In other words, how do we know that the differences are significant?\n5. Commonly used language datasets are highly imbalanced. While I understand the authors’ desire to avoid weighing tokens like “the” and “a” too heavily, I question whether weighing everything equally skews the estimate too much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)