An Information Theoretic Perspective on Agentic System Design
Authors: Shizhe He
Avanika Narayan
Ishan S. Khare
Scott Linderman
Christopher Re
Dan Biderman
Venue: ICLR 2026 Poster (2026)
Keywords: information bottleneck
rate-distortion theory
agentic collaboration
large language models
scaling laws
Primary area: foundation or frontier models, including LLMs
TL;DR: We frame agentic language model systems as a information bottleneck problem, deriving scaling laws and practical design principles for efficient collaboration between LMs.

Abstract: Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. 
Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. 
Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance.
In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps.
We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. 
We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. 
A 7B Qwen-2.5 compressor, for instance, is $1.6\times$ more accurate, $4.6\times$ more concise, and conveys $5.4\times$ more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors.
Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover 99% of frontier-LM accuracy at 26% of API costs.

Published as a conference paper at ICLR 2026
AN INFORMATION THEORETIC PERSPECTIVE ON AGEN- TIC SYSTEM DESIGN
Shizhe He', Avanika Narayan', Ishan S. Khare', Scott W. Linderman 2.3, Christopher Ré Dan Biderman 1.2.3
Department of Computer Science, Stanford University
#Department of Statisties, Stanford University
Ww Tsai Neurosciences Institute, Stanford University
shi thizhehe@e atanford.edd
ABSTRACT
Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," " and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor cerpressor-predictor systems remains largely ad hoc, with little guid- ance how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task- specific pairwise sweeps. We argue that these agentic system design questions are, at root, ceemmttion-thcoretii Viewing the compressor LM as noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through information-theoreti framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only more accurate, but also more token-efficient, conveying more bits information per token. A 7B QWEN-2.5 compressor, for instance, is 1.6x more accurate, 4.6x more concise, and conveys 5.4x more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors substantially more effective than scaling predic- tors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied Deep Research system, these principles enable local compressors small as 3B parameters to recover 99% of frontier-LM accuracy at 26% of API costs.
Compaasioon-rodiitioo Workflow
Compressors can increasingly run on-device
Sources
Ceep
Compressioe
p(tix)
?
P(ylz)
26:227)
Compressor LM
Predictor LM
X
2000
2021
2022
2023
2004
2005
What makes compression "good"?
Larges Modtl Sioe Runsable on
LVAion Rashing
Apple Maciiooka
Google Plas Phones
O-O with Incressing rink
Why compressors matter, Many agentic LM systems rely on compressors. and personal devices are growing powerful enough host them on-device. (Left) A compressor condenses long input X into shorter summary Z that a predictor expands into the final answer Y. (Right) Consumer hardware can now run increasingly large open-weight LMs, shown for Google Pixel phones and Apple MacBook laptops under FP16 precision with memory estimates from Modal (Lu 2024). LM-Arena ranks indicate relative performance.
Figure 1:

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Agentic language model (LM) systems have quickly become the backbone of modern Al workflows. From "Deep Research" systems (Hadfield et al. 2025) to Claude Code (Anthropic 2025), millions of users now interact with pipelines where one model processes information and another builds on its outputs. Modern workflows commonly involve analyzing and generating more tokens than even the largest frontier models can handle effectively, degrading model performance- failure mode referred to as context rot (Hong et al.] 2025). Multi-LM systems coordinate multiple models to manage reasoning and memory beyond single model's context window. While these architectures vary widely, a recurring pattern emerges across domains: smaller compressor models distill raw contexts into compact texts, which then consumed by larger predictor models that output answer and interact with the user (Figure I (Hadfield et al. 2025 Shen et al., 2023).
At present, however, designing compressor-predicas agentic systems remains largely trial-and-error. We lack basic understanding of how the choice of compressor and predictor affects downstream performance. Specifically, we cannot determine whether credit belongs to the compressor's distillation or the predictor's 's reasoning- we lack task-agnostic methods to evaluate the compressor ' outputs independently from downstream performance. This because we are unable to measure how much of the original context the compressor actually preserves, which in turn determines how effectively the predictor can reason. This attribution problem has immediate practical consequences: as new models are released and practitioners swap components, they have principled way to identify which module to improve without sweeping across the compound system from scratch.
To address this gap, we take an iooommttion-hhorette perspective, viewing the compressor noisy channel between the raw data and the predictor model. This framing allows us to evaluate communication between the two models rather than treat it heuristically. We propose using mutual information (MI) between the raw context and its compression as a task-agnostic measure of com- task-agnostic predictor of downstream performance (Hoffmann et al., 2022; Kaplan et al., (020)). We then conduct rate-distortion analysis to measure how downstream task performance varies with the degree of compression. While it is intractable to calculate MI between two token sequences linked via a nonlinear model, we develop simple, unbiased estimator that can be computed via modern inference servers without requiring full vocabulary log probabilities.
pressor aooeacyaanaloguus to how perplexity serves
With this new information -hieooation-thhoortiic lens, we perform extensive empirical studies on five datasets (LoNGHeALTH (Adams et al., 2024), FINANCEBENCH (Islam et al. 2023), QASPER (Dasigi et al. 2021). WILDCHAT (Zhao et al. 2024), and FINEWEB (Penedo et al. 2024g) answer the following questions:
1. Should you spend compute the compressor or predictor? We find that compressor quality overwhelmingly governs performance: scaling QWEN-2.5 compressor from 1.5B to 7B improves accuracy by 60% whereas scaling the predictor from 70B to 405B yields only 12% improvement on LONGHEALTH. This establishes simple design principle: "front-load" compute into compressors, perhaps running on-device, to reduce dependence on massive cloud- hosted predictors. (Section 3.1)
2. Which compressors are more token-efficient? We find that larger compressors emit fewer output tokens while maintaining quality: in many model families, scaling compressor size not only improves accuracy but also produces compressions that are up to 4.6x more concise. This token-efficiency yields sublinear scaling of PL.OPs-per-generation as a function of model size. Strikingly, increasing QWEN-2.5 compressor from 1.5B to 7B, only adds 1.3% more FLOPs-per-genneaation. (Section 3.1
3. Which factors determine compression quality and how do they relate to downstream perfor- mance? We find that compressors' outputs carry up to 5.4x more MI about the context (Sec- tion 3.2) Rate-distortion analysis reveals that information rate (MI per token) correlates strongly with downstream performance and perplexity (r -0.84, π2 0.71), providing practical proxy for predicting system performance without full end-to-end evaluation (Section 3.3)
4. With so many knobs turn, which factors should you focus on for agentic system design? We perform a meta-analysia across model families, sizes, and datasets, exposing a clear hierarchy of importance: compressor model family > compressor size > predictor size. (Section 3.4
2

---

Published as a conference paper at ICLR 2026
As practical demonstration, apply our findings to simplified Deep Research pipeline, where a single predictor aggregates outputs from multiple compressors. This system achieves 99% of frontier-LM accuracy the DEEPRESEARCH BENCH benchmark (Du et al. 2025) using local compressor models as small as 3B, reducing API costs by 74% (Section 3.5
2
PRELIMINARIES
2.1 RELATED WORK
Agents and Multi-Agent Systems We define Al agents LMs that are embedded context and are able to reason, plan, and act through tool use (Belcak et al. 2025 Su et al. 2025 Wang et al. 2025: Yao al., 2022). The agents are able to interact with its environment autonomously, enabling them to help us automate workflows and schedule our days (Pan et al. 2025). Modern workflows often require ingesting large corpora (enterprise documents, web search results). Relying on single agent is both expensive in token cost and unreliable as context degrades ("context rot") (Hong et al. 2025 Zhang et al.) 2024). has become increasingly common to divide a complex task and context among multiple agents that take on different roles (Dang al. 2025 Hadfield al. 2025 Shen et al., 2023). Recent work has shown to improve cost efficieccy by assigning smaller models to ingest the raw context while larger frontier models coordinate and orchestrate, effectively splitting the control plane and the data plane (Narayan et al. 2025).
Most prior work reports end-to-end utility such accuracy, latency, and cost, while overlooking the communication channel itself. However, the use of information theory to explain and motivate the design of multi-agent systems has been limited, while there exists large corpus of work on information theory in deep learning more generally. We analyze the intermediate communication, focusing on asymmetric compressor-predictte setups and their scaling.
Information Theory Deep Learning Information theory has been used to formally reason about intermediate representations in neural networks (Farquhar et al. 2024 Gardner 1988 Kawaguchi et al., 2023; Morris et al., 2025). It has been used to provide notions of compression, informa- tion flow, hallucination, and model capacity. According to the information bottleneck principle, latent representations graphical models trade-off input compression and downstream prediction ("compress much as possible while being able do the task")) (Tishby et al. 2000). Prior work has shown that neural networks representations exhibit this trade-off, with trained layers naturally converging towards the information bottleneck minimum (Shwartz-Ziv & Tishby 2017; Tishby & Zaslavsky 2015). Other work attempts to understand the information dynamics of model learning and
generalization (Goldfeld & Polyanskiy 2020; Saxe et al. [019 Westphal et al. 2025) by evaluating
the information flow throughout training. Recent work leverages information theoretic quantities to both evaluate natural language generations and token sequences (Arda & Yener, 2025 Darrin
et al. 2024 Fränken et al. 20244 Shani et al., 2225),, and define training objectives and regularization
terms (Chen et al. 2016 Hjelm et al. 2018 Kirsch et al., 2020 Oord et al., 2018 Wang et al. 2020).
However, estimating information theoretic quantities such as MI in high dimensions is notoriously
hard. Several works have shown that MI estimation can be highly sensitive to noise and sampling
(Belghazi et al.
2018 McAllester & Stratos
2020
Paninski 2003).
2.2
INFORMATION THEORETIC PROBLEM SETUP
We extend the information theoretic framework beyond single-model analysis to study communication between two LMs.
We start with a simple compressor-predictor system, including one compressor LM, and one predictor LM. Let X be the input context and Y the target output. We consider a two-stage process
p(=|=) Z p(vl=) Y.
X
The compressor p(2 (2) is modeled as noisy channel, which compresses the context into summary Z. The predictor p(y =)) then uses this summary to generate the target output Y.
We proceed define MI estimator.
3

---

Published as a conference paper at ICLR 2026
Estimating mutual information We want to measure the amount of information Z contains about X, denoted as [[;; 2). Larger MI values I(X; 2) indicate that the compression retains more information about the original context. Mutual information estimation is well-studied problem in statistics and machine learning (Murphy] 2023 Poole et al. 2019). However, many existing estimators not practical in our setting. While many variational bounds require access to the underlying distribution or the training of an auxiliary model, we want to directly use the log probabilities exposed by LM inference engines.
To estimate MI, we start with the KL divergence (Kullback & Leibler
1951)
representation:
p( p(2/z)
z) DRI, (( =(,,spp(e,))
log
p(z)
While computing p(2) is intractable, we can sample from and evaluate our encoder p(z[zr), and take samples from the data distribution p(x),
log p(z/z)
T(X; 2) = E Ea,a~p(a,e)
Ex' /p(2(2²)]
)
1
log pa 3tyl - llog 1 Ep(esylang f(x; 2),
NM
where 2ij ~ p(z[x,), == 1, I...,, N,
M and xy p(x), 1= I.
Note that i(x; 1) < [og(N). where the maximum is achieved when p(zgla,) >> p(z (mx)) or p(zi (x;) < p(zgg (z)) for all i and all ن (Appendix B.2). We do not claim theoretical optimality with this estimator. Instead, our aim is a practical estimator that can be computed directly using modern inference engines. With this estimator, do not need access to the full probability distribution over the vocabulary, which allows us to use accelerated inference engines such as SGLang (Zheng al. 2024).
In our experiments, we treat the raw context separately from a query Q on that context. Thus, each compression Z is generated conditioned the query Q so we estimate T(X; Z Q), which simply requires conditioning all terms on Q. While 1 (X; Z Q) 2 0, in practice, our Monte Carlo estimate can produce small negative values due finite-sample variance. We correct these artifacts by clipping MI to zero. Furthermore, we find that LMs at 1-3B could assign high likelihoods to nonsensical token sequences, indicating miscalibration. Thus, we evaluated the log probabilities using a proxy model at the 7-8B scale. To mitigate biases, the proxy model was selected from different family. We acknowledge that this could introduce variance and biases, and investigate their effects through ablation studies in Appendix F.1.4.
Rate distortiii theory quantifies the trade-off between rate-i.e., the amount of information the compression carries about the input-and distortion-the error in the prediction. We define rate (or bit-efficiency) R T(X:4Q) for I. output tokens (measured in bits of mutual information per token). For simplicity, we define distortion as D = 1 - ACC(Z), by using the accuracy S ACC(Z)= < 1. See the R-D curve in Figure 19 (left). As rate increases, we expect distortion to converge towards lower bound (irreducible error). See Appendix B.3 for further details.
RESULTS
We evaluate the compressor prediccor system as an information bottleneck across different tasks and domains. Beginning with a comprehensive scaling analysis of compressor and predictor model family and sizes, we find that larger compressors are not only more accurate but also more concise, leading to sublinear -op--eer-gennration growth relative to model size. We conclude that scaling compressors is more effective than scaling predictors. Building closely tracks downstream accuracy and perplexity, providing a task-agnostic signal of compression quality. A meta--nalysis highlights compressor model family and compressor size outweighing predictor size as the most important factors. Finally, we validate these principles in a Deep Research pipeline, where local compressors deliver frontier-level accuracy fraction of the cost.
this, we show that mutual information rate
4

---

Published as a conference paper at ICLR 2026
Datasets We study setup five datasets, two synthetic, three non synthetic: (a) LONGHEALTH,
a set of synthetic clinical reports and 20 patient histories (Adams et al., 2024). We hide the multiple-
choice options from the LMs to treat it as question answering (QA) dataset. (b) FINANCEBENCH, collection of 150 10-K filings and reports paired with QA tasks (Islam et al. 2023). (c) QASPER, a
QA dataset of 5,049 scientific research papers (Dasigi al. 2021). ) (d) WILDCHAT, large-scale
LM conversation dataset (Zhao et al. 2024). (e) FINEWEB, dataset of processed web pages from CommonCrawl (Penedo al. 2024 See Appendix D.I for more details on datasets.
Evaluation procedure We rum each experiment with S = random seeds. We evaluate prediction quality using accuracy for LONGHRAI.TH, FINANCEBENCH, and QASPER, assessing correctness of the predictions against the ground-truth using a GPT-40-MING judge. We use perplexity for WILDCHAT and FENEWEB, evaluating the log probabilities a LLAMA-3. --B model.
smaller open-source LMs of the model families LLAMA-3 (Grattafiori et al., 2024), QWEN- 2.5 (Qwen et al.] 2025). and GEMMA-3 (Team et al.) 2025). In our analysis, choose focus GPT-sttle non-reasoning architectures. We additionally conduct preliminary experiments on reasoning and mixture-of-expprts (MoE) LMs of the QWEN- family (Yang et al., 2025). See Appendix D.2 for further details on the compressor setup and Appendix C.1 for compressor prompts.
Compressor model As compressors,
Predictor model
We evaluate larger frontier models GPT-4o (OpenAl al., 2024) as well as LLAMA-3 (1B, 8B, 70B, 405B) and QWEN-2.5 (72B) models as predictors. See Appendix C.2 for predictor prompts.
3.1 WHERE Do YOU NEED THE FLOOSs: IN COMPRESSORS OR PREDICTORS?
We ask: should we scale compressors, which distill large amount of information into concise summaries, or predictors, which reason over the provided summaries to solve complex tasks?
In the following section, we show that scaling the compressor LM yields more significant gains. We vary the compressor model size and study its effects on downstream accuracy, the length of the compressed summaries, and the number of overall LLOPs-per-generation. We also establish scaling laws linking compressor model size to downstream performance.
First, we examine question-answering (QA) accuracy when scaling both compressor and predictor model size.
Downstream performance function of compressor size. We illustrate how downstream accuracy increases as model size increases (Figure left). On LONGHEALTH, 7-8B models are up to 3.1 x more accurate than 1-1.5B models and surpass the GPT-40-only baseline by 4pp. On FINANCEBENCH, 7-8B models are up to 2.6x more accurate than 1-1.5B models and are able to recover 97% of the GPT-40-only baseline. The same scaling behavior holds for GEMMA-3 models.
more concise. In this section, study the number of compression output tokens as a function of compressor size. We find that larger compressors are more concise (Figure 2 middle) without sacrificing accuracy. Specifically, 7-12B compressors are up 4.6x more token-efficient than their 1-1.5B counterparts within the same model family. QWEN-2.5 models tend to be more concise than LLAMM and GEMMA-3, suggesting that models can significantly vary in their communication profiles.
Larger compressors
Compression compute cost scales sublinearly with compressor size. We combine the number of parameters with output token counts to estimate FLOPs LLOPP--er-gennrationn for each model family and size, i.e., the actual compute cost (Appendix B.T). Because larger compressors generate fewer tokens while maintaining accuracy, model families exhibit distinct scaling behaviors (Figure 2
scale sublinearly with model size. Different
right): QWEN-2.5 compressors can scale from 1.5B to 7B parameters with only a 1.3% increase in -.OPs--er-gemeration on LONGHEALTH.
Scaling compressors more effective than scaling predictors. Figure 3 shows that scaling the predictor LM provides only marginal improvements in accuracy a baseline predictor capacity (s)
5

---

Published as a conference paper at ICLR 2026
LongHealth
Accuracy incressss conspressso
arger comprassars fower ouqua Nnkinns
aarer comprassars more
Qwen
0.8
GGP44 prosditeer
Numbar of Compression Tokeens
10000
GFLÖPS per Compresaton
8000
Acouracy
6000
0.4
4000
0.2
3,08
4.00
7:08
e.08
12.08
2.08
4.00
7:08
6.08
12,08
0.00
4,099
7.08
8.0
12.08
Model Sice (Bilions perameteng
Model Size (Gilions of parameterg
Modill Size (Dilions of parameleng
FinanceBench
Accuracy incresses comsresstor
aager compressiors fower ouput tokions
aarer comprassors more OLS
Owen
0.6
Numbar of Compression Takerss
10000
GFLOPS per Compresaion
0.5
800
8000
6000
0.3
4000
0.2
3,089
4.08
7.00
8.08
00
3,0
4.0g
7.08
8.0
12,08
$99
3.08
7:0
8.08
12.08
Model Siza (Billons parametors)
Modal Size [Billonz of parametors)
Mocel Site (Bllions of parameters
Owen-2.5
Liama-3
Gemma-3
Figure 2: Downstream accuracy, compression length, and compute cost scale with compressor size (Top: LONGHEALTH; Bottom: FINANCEBENCH), We scale compressor model size and report a different metric on the y-axis of each column: (Left) accuracy, the dotted line being the GPT-40 baseline. (Middle) compression length, (Right) MELLOPs-per-compresiii
Vertical bars denote standard errors. Larger compressors produce shorter outputs with higher downstream accuracy. Similar trends hold on QASPER, WILDCHAT, and FINE WEB (Appendix
E.1.3)
8B-70B) is reached. The gains in accuracy by increasing predictor size from 70B to 405B are within 12% (LONGHEALTI)) and 1 % (FINANCEBENCII), In contrast, scaling the compressor LM for both families leads to steeper increases in performance for fewer Leoee--pergenneratonn spent. For QWEN-2.5 compressor family,
meaningfully increase only when transitioning from 7B to 14B (models up to 7B all have roughly constant FLOPs L.PP--er-gennration
You can trade local for remote compute. As shown in Figure 1 (right), powerful models up to 27B can run without aggressive quantization current-generation laptops. We anticipate the trends to continue and that even bigger models could run locally, and for free. Our results motivate "front- loading" FLOPs into local compressors reduce cloud costs for serving the predictor (Figure 3p.
Analysis compressor errors. Errors in the compression step can be characterized into one of three categories: (a) the compression contains incorrect answer (36.3% of compressor errors); (b) the compression contains no answer (33.3% of compressor errors); and (c) the compression omits details or parts of the information necessary for the answer (30.4% of compressor errors). For more details on compressor errors, refer to Appendix D.7
3.2 WHICH COMPRESSORS MAXIMIZE COMMUNICATION EFFICIENCY?
We want to select compressors that provide maximal task-relevant information, ideally communicated in as few tokens as possible. Downstream QA accuracy and compression length do not fully capture compression quality. Instead, turn to information-theortiii framing: estimate the mutual

---

Published as a conference paper at ICLR 2026
LongHealth
Qwen-2.5
148
0.7
0.7
0.0
0.6
Aosulary
0.5
Accurocy
0.8
1.58
3B
0.4
0.4
0.3
0.3
0.2
0.2
18
0.1
0.1
2000
5000
10000 2000
b0000
2000
5000
10000 20000
50000
Total OPs (Compresser Predicctor)
(Compressor Predictor)
Predictor Model'
L Llama-3 18
Liama-3 8B
Llama-3 78B
Llama-3 4058
Figure 3: Scaling compressors is more effective than scaling predictors on LONGHEALTH. The y-axis reports accuracy and æ-axis shows total compute cost -e--err-generaioon (log-scale). We compare compressor LMs from two families: (Left) QWEN-2.5, (Right) LLAMA-3, We scale predictor (color) and compressor (label) sizes and measure the total FLOPs-per per-generation and downstream accuracy QA tasks. Appendix E.1.6 shows consistent trends FINANCEBENCH.
-
information T(X; Z Q) between context X and generated compression Z conditioned query Q for each compressor model in our scaling analysis, using the Monte Carlo estimator (Section 2.2)
Larger compressors retain more mutual information and are more bit efficient. We observe that I(x; Z Q) increases as compressor size increases (Figure 4 Larger, more expressive compressor models carry more mutual information between the original document and the compressed summary.
On LONGHEALTH, while LLAMA compressors are far from the theoretical maximum, we find that QWEN-2.5 and GEMMA-3 models produce compressions that saturate mutual information the largest model sizes. By contrast, on FINANCEBENCH, mutual information saturates already at the 3B scale. We observe this saturation behavior primarily on datasets with a highly heterogeneous corpus of context documents.
Combining the scaling effects of mutual information with the observation that larger compressors omit fewer tokens, we find that larger compressors are more bit efficient (Figure
We ablate across multiple proxy model choices and find that the mutual information scaling trends remain consistent across different proxy choices (Appendix B.3). Furthermore, we estimate mutual information without a proxy model for compressors of the QWEN-3 family and observe the same scaling behavior (Appendix E.1.5).
Compressor scaling effects are consistent across prompt conditions. A natural concern is whether our scaling results depend specific prompt formatting. To test robustness, we instructed compressor models to output 3, 6, or sentences, varying conciseness levels. Scaling behavior in accuracy, compression output size, FLOPs-per per-generation, MI. and bit efficiency remained consistent across all conciseness instructions on both LONGHEALTH and FINANCEBENCH (Figures [6). The relative improvements from larger compressors persist regardless of prompted compression output length, confirming that model capacity drives the observed efficiency gains.
3.3 INFORMATION RATE CORRELATES STRONGLY WITH DOWNSTREAM PERFORMANCE
Mutual information and bit-efficiency are proxies for system performance. Information rate (bit-efficiency) is closely related to distortion (1 accuracy). Motivated by the classical form of the rate-distortion function for a independent Gaussian source X, we fit decaying exponential functions the rate-distortion data (Appendix B.3). This fit characterizes the correlation between information
7

---

Published as a conference paper at ICLR 2026
LongHeelth
Mutuel Information ((X2(() versus Compressor Size
Bit Emiiiency versus Compressor Size
3.0
MYnus = log
0.025
Minual Information (X:2100
0.020
6R EMrolency
0.055
2.5
1.0
0.010
0.005
7:88
8.08
12:08
8.8
12.99
Model (Billons paremeters)
Model (Bilions perameters)
Qwen-2.5
Liama-3
Gemma-3
Figure 4: Larger compressors generate outputs that carry more information about their inputs (conditioned on the query) on LONGHEALTH, We scale compressor model size and estimate the (Left) mutual information, and (Right) bit efficiency (bits of mutual information per token; higher is better) carried by their outputs. Larger compressor model sizes compress documents with higher mutual information and bit efficiency. The dotted line represents the theoretical maximum of the mutual information estimator at the natural logarithm log(N), N), where N is the number of documents mutual information is computed across. We find consistent trends on FINANCEBENCII and QASPER (Appendix E.1.6
LeagHealth, Compressor Qwen-2.5
Accuracy verous Compressor Size
FLOPS Compression vensus Compressor Site
Mutuel intormation (22X) versus Compressar Size
RCD
3.0
bggl)
0.6
GFLOPS Conpression
2000
Munail Intormation (X27,0)
2.6
2.0
1500
1.5
1000
0.3
500
0.5
0.2
155
7,08
Model Size (Bilions
Model Size parameters)
Model parameters)
Elaborate Prompt
Figure 5: Scaling behavior of compressor model size hold across instructed conciseness (CompResSOR = QWEN-2.5). We ablate over compression conciseness levels by varying the compression prompt instructions. We measure (Lefft) accuracy, (Middle) and estimate (Right) mutual information. We find that accuracy and mutual information are largely unaffected by conciseness instructions. Compressors instructed be more concise are more token- efficient, and thus compute-efficient. Scaling trends in accuracy, compute cost, and mutual information hold across conciseness constraints. Appendix E. shows analogous results on FINANCEBENCH.
rate and distortion and corroborates our previous finding that scaling predictors beyond 70B yields only marginal improvements distortion (Figure left).
Furthermore, Figure 6 (right) reveals that mutual information is also strongly correlated with perplex- ity (r -0.84, R 0.71) for extractive tasks on FINEWER (detailed in Appendix D.1.5
Predictors do not prefer compressors of the same family. Further rate-distortion analysis across QWEN-2.5 and LLAMA reveals that distortion is primarily dependent on model family and size. Cru- cially, predictors do not perform better when paired with compressors of the same family (Figure 19.
8

---

Published as a conference paper at ICLR 2026
Rate Disto-Ditttooonn Curve
Perplexity correlates with mutuai information
12
Jama 18
Distantinn accurscy
0.8
6.5
6.6
Mana 38
0.3
6.2
Jama-3
0.00
0.001
0.002
0.028
0.004
2.6
2.7
2.8
2.9
3.0
Rate Effiieeccy
Nutual Information
Predictor Modill
Llama-3
Figure 6: Mutual information and bit efficiency correlate strongly with downstream perfor- mance. (Left) We vary both predictor and compressor model in the compression-predicee workflow and measure the distortion rate-distortion curves across predictor sizes 1B, 8B, 70B, and 405B for LLAMA compressors on LONGHEALTH. The lines show fitted exponential-decce functions. (Right) We measure perplexity and mutual information on compressions generated by LLAMA compressors on FINEWEB. The line shows fitted linear function (r -0844 18 0.71). Appendix F.2 provides further analyses.
the y-axis and estimate the rate the z-axis. We plot the resulting
3.4 WHICH KNOBS TO TURN?
To guide practical system design, we analyze which components of the compression-prediciinn pipeline most strongly drive downstream QA accuracy. We fit a logistic regression predicting binary correctness on LONGHEALTH The compressors we consider are QWEN-2.5 and LLAMA models and predictors are LLAMA models of sizes IB, 8B, 70B, and 405B.
and FINANCEBENCII using the features specified in Appendix D.4
Our analysis reveals that compressor model family is the most important factor (Figure with QWEN-2.5 compressors outperforming LLAMA. Additionally, scaling the compressor LM matters substantially more than scaling the predictor LM, confirming previous findings in Section 3.1
3.5 SCALING DEEP RESEARCH
We evaluate our compression invressioon-reeicttion framework on open-domain "Deep Research" workflows, where predictor LM decomposes research tasks into subtasks and aggregates compressor outputs into final reports. We use DEEPRESEARCII BENCH (Du et al. 2025), which assesses system performance across four dimensions: Comprehensiveness. Depth, Instruction-following and Readability. These four dimensions form quantitative RACE (Reference-basee based Adaptive Criteria Evaluation) score. We measure cost per task based on current API prices (as of August 2025), which are timessensitive. but serve good approximations for relative cost differences between models. We vary predictor sizes across the LLAMA family and compressor sizes across the QWEN-2.5 family. LM Prompts are detailed in Appendix C and full experimental details are in Appendix D.6
Larger predictor models consistently improve RACE scores, while larger compressors provide substantial performance gains at minimal additional API costs (Figure 7.
As a baseline, we evaluate the results of providing uncompressed web search data to a GPT-40 predictor. A QWEN-2.55 2-5- compressor paired with GPT-40 predictor achieves 2.3% higher RACE scores at only 28. 1% the API cost compared to the uncompressed baseline. We detail further findings in our scaling experiments in Appendix B3
DISCUSSION
We establish an information- iifommation-thoorttic framework for compressor-predictts systems to determine how model selection and scaling affect compute efficiency. Our findings come with important

---

Published as a conference paper at ICLR 2026
RACE Score versus Cost (S)
RACE Score versus Compressor Size
0.24
GPT-4 I: (heeeinng
RACE RACE-Soore Score
RAGE Spore
GPT-o search previnm
0.20
0.30
0.29
3.38
0.18
0.28
0.10
0.25
Average Cost per Task (5)
Compressor Size (Bilions of parameters)
Predictor Mode!
Basalre
GPT-4
Llama-3
Llema-3 Llama-3
GPT-4o
Figure 7: Deep Research Scaling Results. (Left) RACE score versus average task cost when using GPT-40 as predictor with QWEN-2.5 compressors of varying sizes. Costs are based on GPT-40 API rates (August 2025: $2.50/1M input tokens, $10.00/1M output tokens). Larger compressors improve performance with minimal cost increases. For reference, include GPT-40 results without compression and also for the QWEN-2.5 compressor sizes (0.5B 14B) under three LLAMA predictors (8B, 70B, 405B).
model. (Right) RACE scores for different
limitations: at the 1-3B model scale, our MI estimator relies on proxy models and log probabilities, introducing potential variance and biases. Furthermore, primarily focus GPT--ttlle non- reasoning models with single-round communication, limiting generalizability to reasoning- augmented models or iterative multi-agent workflows. While provide initial results for reasoning and mixture- of-experts models, future work should extend the model families and evaluate reasoning traces in more principled manner.
analysis to a wider range of
Several research directions warrant investigation. Mutual information estimation for LM outputs remains challenging, though alternative estimators like INFONCE (Aitchison & Ganev 2021) offer promising solutions. enformationnthhorree fallback decisions for remote full-context processing. Training objectives based on rate- rate-distone distortion analysis represent another avenue to optimize compressor-pree preddicco communication. While we define compression as summarization in this work, compression can also be found other agentic-system workflows, such as structured extraction and function-call generation. intuitive measurement of compute cost. However, device-specific efficiency optimizations are crucial in real-worr wolld deployment settings and warrant further analysis. Finally, mixture-of-exe of-experts (MoE) models (Fedus et al. 2022) may exhibit different scaling behaviors since their compute cost depends on activated experts rather than total parameter count.
principles could guide compressor routing strategies and
POPP--er-generaeee is an
Overall, we distill our findings into four principles for agentic system design:
Principles for Agentic System Design
Compressors be sealed at sublinear computational cost. Since larger models more information- efficient (emit fewer tokens with higher information-densstyy PLOPP--er-generatiin scale sublinearly as function of model size.
"Front-load" compute into local compressors to reduce remote costs. Scaling compressors is effective than scaling predictors. By running larger compressors on-device. we can reduce predictor serving costs in the cloud.
Optimize for information density. The mutual information between input context and agent output is agnk-agio indicator compression quality and tightly linked to downstream performance and perplexity.
Expect model family to differ in scaling trends. Choice of compressor and predictor model family yields offsets in rate-distortion than LLAAM and GEMMA-3. QWEN-2.5 predictors yield higher accuracies than LLAMA.
and scaling effects. QWEN-2.5 compressors scale more compute-effiiinnly
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Simran Arora, Yasa Baig, Kelly Buchanan, Sabri Eyuboglu, Neel Guha, Simon Guo, Jerry Liu, Jack Morris, Simon Pritchard, Jon Saad-Falcon, Frederic Sala, Ravid Shwartz-Ziv, Seonghyun Yoon, and the entire Linderman and Hazy research labs for helpful discussions and feedback. We are incredibly grateful to Modal and Together AI for providing the GPUs to support this work.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 (Hardware-Award CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to (RTML); US DEVCOM ARL under Nos. W911NF-23-2-0188 (Long-context) and W911NF-21 2112-222 (Interactive Human-AI Teaming); ONR under Nos. N0001 1000142312633 (Deep Signal Processing); Stanford HAI under No. 247183; Google DeepMind: Google Research; Google Cloud; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Salesforce, Total, the Laude Institute, Prime Intellect, Anthropic, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VMWare: and members of the Stanford SEAMS project: IBM and Felicis. The U.S. Government authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
Velocity), and 1937301
REFERENCES
Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthicu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXiv:2401.14490, 2024.
Laurence Aitchison and Stoil Ganev. Infonce is variational inference in recognition parameterised model. arXiv preprint arXiv:2107.02499 2021.
Anthropic. Claude 3.7 sonnet and claude code.
claude-3-7-sonnes February 2025. Accessed: 2025-09-17.
Enes Arda and Aylin Yener. A rate-distortion framework for summarization. arXiv preprint arXiv:2501.33000 2025.
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Ce- line Lin, and Pavlo Molchanov. Small language models are the future of agentic ai. arXiv preprint wwXiv:2506.02153. 2025.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint 1xXiv:1801.04062. 2018
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more Ilm calls all you need? towards scaling laws of compound inference systems. arXiv preprint arXiv:2403.02412 2024.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.100 10848, 2(4):6, 2023.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Thomas M. Cover and Joy A. Thomas. Rate Distortion Theory, chapter 10, pp. 301-346. John Wiley & Sons, Ltd, 2005. ISBN 9780471748877 doi: wttp https://onn ine 1 wrrrrr.r..... ey httpa///e...////////.00////////0 ch10
URL

---

Published as a conference paper at ICLR 2026
Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al. Multi-agent collaboration via evolving orchestration. arXiv preprint 1rX55 2025.
Maxime Darrin, Philippe Formont, Jackie Cheung, and Pablo Piantanida. COSMIC: Mutual in- formation for task-agnostic summarization evaluation. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compu- Bangkok, Thailand, August 2024, Association for Computational Linguisties. doi: 10. 1/218/6// aol.o.......-....666.. URI.
tational Linguistics (Volume 1: Long Papers), pp.
https://aclaas antho wow.ohoooo....oooo ogy. wtt.oorg/2022 aw1244..clllooo
6867
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seekiine questions and answers anchored in research papers. 2021.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint. 2025.
Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks. arXiv preprint arXiv:2503.0957., 2025.
Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, et al. Cartridges: Lightweight and general-purpont long context representations via self-study. arXiv preprint arXiv:2506.0622 2025.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in
large language models using semantic entropy. Nature, 630:625 630, 2024. URL https
//api.semant www.seemantiss.. icscholar. org/Corpus ID:
7/api
William Fedus,
Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1333 2022.
Jan-Philipp Fränken, Erie Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah Goodman. Self-supervised alignment with mutual information: Learning to follow principles without preference labels. Advances in Neural Information Processing Systems, 37:61328-61371, 2024.
E. Gardner. The space of interactions in neural network models. Journal of Physics A: Mathematical and General, 21(1): jan 1988. doi: 10. 10.10800305-44002211000000 URL https::/dddd org /0g//10.10 118001188/1000 0305 4470/21/0/1///1
Ziv Goldfeld and Yury Polyanskiy. The information bottleneck problem and its applications in machine learning. JEEE Journal on Selected Areas in Information Theory, 1(1):19-38, 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, et al. The llama herd of models, 2024. URI. https
7/arxiv. wwg. abs 217831783
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas- Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123-44279, 2023.
Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz. Jeremy Fox, and Daniel Ford. How
we built our multi-agent research system, 2025. URL hww.h:tt////// http:///www/tthh/.h/../.c// anth rettrrepiicc i e. com/ engi meeringgbbii t-mmlt -ateiiaggett-rree reg.eentttreeeaar---sseee system Accessed July 24, 2025.
Dongge Han, Camille Couturier, Daniel Madrigal Diaz, Xuchao Zhang, Victor Rühle, and Saravan Rajmohan. Legomem: Modular procedural memory for agget 1lm systems for workflow automation, 2025. htt.thttt///aaatt www./////////.. 04851
12

---

Published as a conference paper at ICLR 2026
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization, arXiv preprint arXiv:1808 1608.06670, 2018.
Karan Grewal, Phil Bachman, Adam
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optinn optimal large language model training. Advances neural information processing systems, 15:30016-30030 2022.
Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts llm performance. Technical report, Chroma, July 2025. URL Bttlhttt:///reearree teaich..rrree trychroma com/ commconntet.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint MxXiv:4008.08435 2024.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nancebench: new benchmark for financial question answering. arXiv preprint arX14:231111194, 2023.
Pengcheng Jiang, Jiacheng Lin, Zhiyi Shi, Zifeng Wang, Luxi He, Yichen Wu, Ming Zhong, Peiyang Song, Qizheng Zhang, Heng Wang, et al. Adaptation of agentic ai. arXiv preprint arXiv:25/2. 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 0bs/2001.08366) 2020. UtLpttps::/ //appp:://///vvv.///. http///wwww...rrra/////a////////
08361
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International conference on machine learning, pp. 16049-16096. PMLR, 2023.
Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Mark Malhotra, et al. Towards science of scaling agent systems. arXiv preprini 22xXi::5//2002200 2025.
Andreas Kirsch, Clare Lyle, and Yarin Gal. Unpacking information bottlenecks: Unifying information- theoretic objectives deep learning. arXiv preprint arXiv:2003.1255: 12533, 2020.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-885 86, 1951.
Zongwei Li, Zhonghang Li, Zirui Guo, Xubin Ren, and Chao Huang. Deepcode: Open agentic coding. arXiv preprint arXiv:25/2.079211 2025.
Yiren Lu. How much vram do i need for Ilm inference? Modal Blog, September 2024. URL Online; accessed
cittps://moche 1. www.coooooo.oooo haamm//logtg /hwwoooooooooooooooooooooooooo hww-hhhwwmmmm much- wwwemmeeeeddieeeeeeeeee
24 September 2025.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884. PMLR, 2020.
John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize?, 2025. URL
wwtp:///g///aaaaaaeooooooos 24832
Kevin P Murphy. Probabilistic machine learning: Advanced topics. MIT press, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- question-answering with human feedback. arXiv preprint
2021.
Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Minions: Cost-efficient collaboration between on-device and cloud language models. arXiv preprint 2502 2025.
13

---

Published as a conference paper at ICLR 2026
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03745 2018.
OpenAI,
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, et al. Gpt-4o system card, 2024. URL httLt://a/aaxx...coom /bs/
2410 0410.44 21276
Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, et al. Measuring agents in production. arXiv preprint arXiv:2512.04123, 2025.
I Liam Paninski. Estimation of entropy and mutual information. Neural Computation, 15(6):1191- 1253, 06 2003, ISSN 0899-7667. doi: 10. 10.112208997663327800777 URL https://dbbi org 10. 11 0tt/////89996633321198
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 1d=n6SCkn20ac
httttp:://openeeiee..roo
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International conference on machine learning, pp. 5171-5180.
PMLR, 2019,
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, et al. Qwen2.5 technical report, 2025. URL htRlittts:///aaat wwwpxiiioorg//aaa abs
http:/2as/2422...//// 2412 2e..bb//211111111
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, et al. Archon: An architecture search framework for inference-time techniques. arXiv preprini urXiv:2409. 15254, 2024.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning*. Journal of Statistical Theory and Experiment, 2019(12): (21112)::2200000 124200, dec 2019. doi: 10.
Mechanics:
URL hnttps:://oi..oo org/ 10. (088/17422 5462 ab3985
Erik Schluntz and Barry Zhang. Building effective ai agents, 2025. URL https://wwe www
aanthropic. com com wtt.:/ooo/enneeeneeeeeeeeeee http:///nnnneeeen//a/aaaa/aoeene
agents
Accessed July 24,
2025.
Chen Shani, Dan Jurafsky, Yann LeCun, and Ravid Shwartz-Ziv. From tokens to thoughts: How Ilms and humans trade compression for meaning. arXiv preprint arXiv:2505.17117 2025.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https: //axxiv org/abs 2303. 17580
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprini arXiv: 770.008000 2017.
Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, et al. Toolorchestra: Elevating intelligence via efficient model and tool orchestration. arXiv preprint 151 1689, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
14

---

Published as a conference paper at ICLR 2026
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. leee, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Boxin Wang, Shuohang Wang. Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. Infobert: Improving robustness of language models from an information theoretic perspective. arXiv preprint arXiv:2010.00.329, 2020.
Junlin Wang. Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Eixture-of-agente enhances large language model capabilities. arXiv preprint arXiv:2406.04653 2024.
Yanlin Wang, Xinyi Xu, Jiachi Chen, Tingting Bi, Wenchao Gu, and Zibin Zheng. An empirical study of agent developer practices in ai agent frameworks. arXiv preprint arXiv:2512.01939 2025,
Charles Westphal, Stephen Hailes, and Mirco Musolesi. A generalized information bottleneck theory of deep learning. urXiv preprint arXiv:2509.2633 26327, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint ar 25v::22555093885 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 1a01-100011222 2023.
Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.046/ 2025a.
Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chich Huang, Chunyu Miao, et al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506. 18959, 2025b.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating long- long-context tasks. Advances in Neural Information Processing Systems, 1122222122222332 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL
net /tte//oooomnddd--11eee
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 62557-62583, 2024.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Meml: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506. 15841, 2025.
Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, et al. Latent collaboration in multi-agent systems. arXiv preprint 2.11:22111.26339 2025.
15

---

Published as a conference paper at ICLR 2026
We utilized Al tools assist with code implementation and manuseript proofreading.
ETHICS STATEMENT
Regarding fairness and accessibilityy recommendation to "front-load" computation into local com- pressors may create barriers for researchers with limited hardware resources, potentially exacerbating inequalities in Al access despite reducing cloud API costs by 74%. The compression techniques pro- cess documents through multiple model stages, raising privacy concerns about information retention in compressed representations, especially when handling sensitive data. Our efficiency improvements could accelerate broader deployment of agentic systems with both beneficial and harmful applications, while the environmental impact of encouraging larger local model deployment (up 27B param- eters) requires careful consideration against potential increases in aggregate energy consumption. We encourage practitioners to implement appropriate privacy safeguards and consider the dual-use implications of these impressionnprediction architectures as they become more prevalent.
REPRODUCIBILITY STATEMENT
We provide comprehensive implementation details and experimental specifications throughout the paper and appendices. Section 2.2 contains the complete derivation and implementation of our mutual information estimator, while Appendix [B. J details the FLOPs computation methodology for dense transformer models. All experimental configurations, including model selections, hyperparameters, and prompt templates, are specified in Appendices B and D Dataset construction procedures are documented Appendix [.1] with specific sampling criteria for each of the four datasets, as well prompt templates to construct synthetic QA and generation tasks. All models used to generate synthetic tasks and answers are detailed in Appendix [.]] The Deep Research experimental setup is fully deseribed in Appendix [.6] including the complete workflow implementation and evaluation framework, as well as prompt templates for compressor and predictor LMs. All experiments run on S random seeds with reported standard errors, and we specify the exact model versions, inference parameters, and evaluation protocols used across all experiments. Rate-distortion analysis parameters and fitting procedures detailed in Appendix B.3 while the generalized linear model specifications for meta- meta-analysia analysis are provided in Appendix D.2
detais
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXTENDED RELATED WORK
Agents and Multi-Agent Systems Multi-Agent system design choices studied in the literature include model size (Wang al. 2024), number of agents and communication rounds (Chen al.)
2024; Kim et al. 2025 Schluntz & Zhang 2025). research depth (Zhang et al. (025b). agentic
state and communication (Li [025| Zhang al. 2025a Zou et al. (2025). and planning or
decomposition strategies (Chen et al. 2023 Erdogan et al. 2025 Yao et al., 2023). Overall, they find
that performance improves with larger models, more agents, additional rounds, and increased task-
relevant context. Other research focuses on learned or automated optimization of agentic architectures
(Hu et al. 2024; Jiang et al. 2025 Saad-Falcon et al. 2024) and agentic memory systems (Han et al. 2025 Zhong et al.) 2024: Zhou et al. 2025)
Deep Research Over the past year, large-scale Deep Research systems have been popularized and adopted by frontier labs in industry such as OpenAl, Anthropic, and xAI. These systems commonly have an asymmetric setup, where high-capacity predictor LM decomposes the user query into subtasks that are executed by compressor models in parallel (Schluntz & Zhang (025). The results of these subtasks synthesized into one answer that is commonly presented to the user comprehensive research report. In practice, compressor models can range from frontier models (Hadfield al., 2025 Schluntz & Zhang, 2025) to local small LMs (Narayan et al. (025). Recent works have been centered around establishing evaluation benchmarks to compare different agentic Deep Research systems (Du et al., 2025 Guha et al.] [2023] Nakano al., 202T). These benchmarks focus on measuring the output quality and downstream utility of generated research reports.
B EXTENDED DESCRIPTION OF METHODS
In this section, we provide a more in-depth explanation and derivation of our information-theoreer approach and problem setup.
B.1 COMPUTE COST OF DENSE LMS
We measure compute cost of each compressoo/prediicor LM call by the number of FLOPs per token in each forward pass through our dense wransformerbbaee LMs as
Cdense 2 2N params + Mexyerreetx
with model size N paramss number of input context tokens Netx, number of layers Players and number of attention heads per layer dattn (Kaplan al. 2020). We observe that LLOPP-per-token generated for dense models grows roughly linearly with model size.
B.2 THEORETICAL ANALYSIS: BOUNDS OF MONTE CARLO ESTIMATOR
The Monte-Carlo estimator of mutual information between X and Z is upper bounded by
i(x; Z) < log N,
where N is defined as the number of contexts sampled from X.
Proof. Consider the Monte-Carlo estimator of mutual information
N
M
N
pl (2 ((s))l))) ]
f(x;2)= = NM
ΣΣ
logp(=))<)) loog
Σ
N
1=1
N M
Σ E(a (mn)) ]
N
log 1og (xx) (++++ + log N - llog
NM
==1
17

---

Published as a conference paper at ICLR 2026
For any fixed summary Siy and context 21, we have
N
p(zy)a))) 2 max p(ziglz1)
2 p(zzg (za)
Plugging this into the estimator bounds yields the upper bound
N
M
NV Σ
i(x; NM
log p(zyla)) + log N - -log
pl 2ig (xi)
1=1
I <log N.
To see when the bound tight at p(a,s (2x) constant Cij
>> p(zig + we write the denominator using
N
N
Σ
((cylaz)) = es
(o
lm1
Choose Cij = log loog((z;)) (z),, this gives us
M
[(X: NM
1
Σ
Σ
log (ogg(zz))z)) + log N Cij log Σ log plang -cs
log Σ los poos 1ogpp(/ ]
1
log N
NM
Since x;)) >> p(2tg1211). Al #
i(X: log N +
- log (1 + +
(s 0)
NM
log N. Thus log N is a tight upper bound.
B.3 RATE-DISTORTION- THERRY
Assume X to be independent Gaussian random variable with variance o² ox(X), the canonical rate-distortion function is
R(D) 2 log (X)
0 DSD
0.
D>o²(X)
We illustrate rate-distortion curves as distortion D (how much accuracy is lost in communication) versus rate R (how many bits spent encoding data). Inverting the expression for the rate-distortion function gives (Cover & Thomas 2005)
DGaussion (R)
=²22 2-2R
= e 21n2
= CCe BR
with C=o², 2nn
The Gaussian rate-distortion function does not describe LM data distributions. However, it serves as a closed-form model that captures the qualitative shape typical rate-distortion curves. use the Gaussian rate-distortion function as simplified model to qualitatively compare different predictors.
practice, we treat C and as function parameters to account for unknown variance and modeling noise. LM ceapressioon-rediittin
systems often exhibit non-zero distortion floor (e.g. imperfect LM judge, label noise, predictor expressive power), which we account for through offset Do. Do is lower bound of the distortion in the system rate (bit efficiency) increases,
D(R) = Ce bR + Do-
We fit exponential decay functions to the rate- distortioo curves based on the least-squares estimates (Ö, b, Do).
18

---

Published as a conference paper at ICLR 2026
C PROMPTS
C.I COMPRESSOR MODEL PROMPTS
We use the following prompt templates to compress the raw context documents on LONGHEALTH, FINANCEBENCH, QASPER, FINEWEB, EB, and each chat conversation on WILDCHAT:
Query-Specific Base Compression Prompt Template
include ONLY information needed the cast key points relevant to the spessiion Do NOT ANBRER THE QUESTION DIRECTLY.
Sunnariza the following
(quezy)
Text: (text)
Your sumtery (make waee La include all Ingustant Lant details / backgroand information celated Lo Lhe equeation... vDD NOT ANSEER THE QUESTION
Memory ConstructionCoomprssion Prompt Template (WildChat)
You aze nenory compression assistant, tasked with sumaarizing chaz convezsation. aurnery preserves lnvent any Infornation.
that useful menory lang Languege model.
CHAT:
(sonversation) iin)
Your summery (Juas plain Lext, no FornatLing
Query-Agnostic Compression Prompt Template (FineWeb)
Samaaiia ice Lhe Col lowing Lest and produce sunnary Lhat preservea all detaila that could be needed Lo answer likely questions about the text. Do Nor invent facts.
anaver any question just sumesise potential ial anaves beasing info. (Lext)
Test:
Your sumarry (nake sure to include all Ingortant details / backgroand information celased. Just plain text, tosmatting.)
C.2 PREDICTOR MODEL PROMPTS
Given the compressor output, we answer extractive QA tasks on LONGHEALTH, FINANCEBENCH, QASPER, WILDCHAT, and FINEWEB, and creative tasks FINEWEB using the following prompt templates:
Base Prediction Prompt Template
Please ansver the following question based the provi ded sumnsry.
Quest
(qerry)
Sumary: (sunmery)
Please respend in the tollowing 250N fornat: <brictly think about the infermation you have and the qlestion you need answers
*exploration": eebriet explanation ot the anever, explain how you areived at the answer, 1-2 serter werterces>*,
tinal ansvers*
Your ansxer 1200 MUST ONLY RESPONO WITH THE JSON OBJECT):
WildChat Prediction Prompt Template
Please anaver the Following queation fon. based on Lhw provided clat nemay..
Quession:
(quecy)
19

---

Published as a conference paper at ICLR 2026
(nemory)
Flease respand in the tollowing J8OR tornat: <brietly taink about the intornation have and the question need answer>
{
*answer":
"cyour final ansvers
}
Your answer 1200 NUST ONLY RESPOND WITH THE JBON OBJECT):
Fine Web Prediction Prompt Template (Extractive)
Flease ansver the rollewing questior based. the provided (ccatext.type).
casszion: (quers)
[sontext.cvps):
(surmary)
Please respend in the following 2S0K fornat: diriefly think
infornal you question you
{(
: Kyour Linal anawwE>
Your answes 120D ND87 ONLY RESPOND WITH THE JSON OBJECT):
Fine Web Prediction Prompt Template (Creative)
Please the Fol lowing based provided (context sype).
Task: (quesy)
[sontwxt:lyy=]:
(sumnary]
Please respond in the foll lowing 2008 Focnal: <brictly think about intornation have and question
*answer";
*syour tina. ansvers
}}
Your anexer IYOU MUST ONLY RESPOND WITH THE JSON OBJECT):
C.3 DEEPRESEARCH PROMPTS
The following prompt templates were used sequentially the backbone for our paaeeessoor-rrdddctoo Deep Research workflow.
ReepResearch Query Generation Prompt Template (Predictor)
aze research supervisor tasked compcctoneiecly exploring research topic. Use stzatogic, top-down approach to design your cesearch.
Research (quary)
*FEAASE 1:
RESEARCH PLANNING**
First, analyze this rescarch topic and create comprobcacivo research plan. consider: understand
specific abjactivas
guide your sesearch?
How do different aspenta of this topic relate to each othee?
Shat typea of information will be nos: valuable for complete analysis? leqical flew for
rindings?
*FEAASE 2:
STRATEGIC QUERY GENERATION
Bazed en your rescerch plan, cenerata EXACTLY differenz search queries that together will provide cumpes guverage
topae. query abould specific stralegie purpose your everall sesearch architecture.
For wach search quesy, provide specific sub Lask/guustiion that explains how It served awsearch
Belurn your responae in this exact 1808 fornat:
{(
architecture and strategic abjject invest iggtiing this topie. Explain bey aceas investigate, they culaie, and logical strcooLeM ond lysis.
besserchhppant
"gaerins::
20

---

Published as a conference paper at ICLR 2026
"seaccll.uesy "apecific reub task"; spect fic quest on does this query address and how does serve the research
opt and
plant
"seooodd strategic search guery", "ubbttask"; "What does this query aim to discover and how does it rit the research
"oeecollquery":
rehitecturee"
"emarchupery"": *third tangeted search query",
"aub,lask"; "What aspect dows this explore and why wasential Lo the sesearch plan?*
Hh
"coarch.quary"; "sourth strategic search query",
"subitank"; "What question does this
it complement other querses?"
H1.
"searoh-ppery": *sithh focused search query*, does this and how does build on previous querles?"
"sub,task"; "What
"soarch-quary": "sixth comprshensive search qacry",
'subitask"; "What additional dimensien tais 11.
crucia1?"
"search-quary": "seventh strategic search quary",
"enb,tank"; "What specific gap does this fill in the research architecture?"
"caarch-uaar": *aighth conclucing search quary",
"sbbitask": "What tinal aspest does this and
complete
"syntaesie "Setalled led strategy conoining tindings all quecies based informatica xill structured, what relacionshige will highlightes,
zesearch plan. Explain and now the tinal enalysis will be organized to naximize comprehonssveness and insight."
}}
**Strategic
1. Each search quezy should be 3-8 well-chosen kaywards targeted for your specific research ebcttvivas
2. Design queries to serve compicnentary roles in your research architesture (nct jest genoric dineacions]
arauze sueries stratecically coordinated provide comprehensiva topic coverage
4. Each sub-tack chould expaann how the quary serves youz overall research plan Greate eynthesie strategy that retlects your planned research strusture
**Rseearch Focus
Consider:..
Foundational underst understanding and cuzren: scate
Key challenges, problens, Hinitations practicos
Solutions, nethocoiegies,
Evidence, data, enpirical findinga Fature trends, developments, and implications Maitiple perspactives and stakzaolder vicxpoints
with ""ooor"" any other narkdown FornatLing. Reauen the raw JBoX jeij dissetly.
You must seturn ONLY
objest.
fornat.
DeepResearch Synthesis Prompt Template (Predictor)
report DeepReeee have exeeasive research findings bolow use ALL of taom to create a deta:led, thorouga anslycie.
oneating comprehensive, high-gueliiny
<orricinal Research Teske (ariginaiitask)
<eesessch Flansee (researchuplan)
v*Bosearch Findings:*> (go.pairs)
**Sythhesis Stcategyiee (eyntherie.crxaterp)
+*CNEEEEENSSVEE INFORMATIGN
REQUIRED:*.
You must systensicaally work through MLL the provided research zindings above, Do not relectively use only zona information your report mast cancnstrate thas have reviewed and intecrated AlL relevant details, data
eximples,
quezy
*REPPRR STRUCTURE AND 3ECTTEMENTS::*
1. +*Dstailed Background Context++ Provide extensive backgreund and context .xComprehensive
Shitiple detailes sections spacific exancles, data, quaces from **Thorough implicatisa: Discussion.. bets:lod snalysis of inplicrtions and significznce
"<Extensivs Evidence Intecsationee
5. *cCorpiste Conclusionaes Comprehensive conclusiora and future research directiona
REQUIRENENTS
QUALITY:.*
Weite detailed exsianat lons, not brisf sumariee
Include excensive examples and case studies from the research
Provide comprchonsive zackground
every mater point
statistica: data, quotea, specific
research findinga
21

---

Published as a conference paper at ICLR 2026
Elaborace impl ieations, significance, broades connections
Tholade detailed analysia mechodologies, approsches, and framwworks nent fored Biscuss Linitations, chal lenges, and areas for Surther reseatch extensively
thorough
report
Uses extemsive detail compoebems ive analysia Lhreaghout Integrates ALL tindings with detailed explanations and context
ng evidence Tasl det disddedseee disussiion of all relevani eagecis ispl Dencnetrates nastery of the subgect through thorough, detailed snalyeis
Provides congrehenzive
wish extensive
OPINAL ABQUTREMENT:A4
reeponse mest substantial corprehensive.
extensively exhaustive
cmprehersiive ana analysis, and complete utilise iist research fincings. Provide trily comprehensive coverage or the topis that senenctrater thorouga understanding and integration or all ava:lable research.
DeepResearch Source Summarization Prompt Template (Compressor)
Your job is to extract detailed, specitic information from the zollowing content to support comprehensive research analysie.
Quory:** (query)
</Speci fie Sub task/Ouest (sub)taak)
[sontent]
VEXTRACTION REOUTREMENTS: Provide detai and comprehensive extraction that
**Fctuul Intormation:
Specific mumbers, statistios, percertages, and quantitat ive data
cimelinas, and chrenolopies. information
people, organizatione, companies,
Geographic lncations, regions, and jurisdiet
Techaical specirications, maazurements, and zenchmarks
Deseiieed Examplas Evidence:s>
Concrete case studies and real real-world examples
Specitic research tindings and study results
Diroct
epinicas
Folicy details requlatory iatormazion
implenentation detaile and methodb.ogies
Comprehennivee Coverzgez**
Key facts direstly rolevant to
nain query apecifis aub-task
Important cancepts, definitions, and explanations cause-ans-ettest relationmiipe and underlying mechanisms
and devalognents Trends, limitations, and problen azeas identiriod Chelicagss,
tine
*Analy:ical Insights:*.
Implications significanes
information
halacionships between different points comparative information and henetmarks
Pature projections and forecasted zrencs
professional
Focus on depth and epecificity while maintsining clerity. Extract comprcheneive, specitic information with extensive ive detail, numbars, exmpless, and avidence. Do not provide brief sumnariea ensure your thorough and substansisi. Extract infermation would be valuablo creating comprehensive research. report. Pay special attention intornation directly adaresses the sus-tack question.
your
JS0N fusmat wish fields:
"oxplanation": Your detailed extraction of specific infommation, facta, data, axamples, and avidence with extensive deteil
"anavec"; "zelevant" 10 this content contains informat ion selevant Lo the quezy and sub- fnol zelevans" othervise
CRITICAL JSON FOWATTING ROLES;
Replace all double quotes (") inside cex: with single quotes (*)
Deplass nexlines with spaces
Ensure the
is valid and parseable
Do NOT use 1ina breaks within the JS0N fields
Example fornat:
(("oxplanation": "Your dotailed extraction with spscizic facts, nunbers, examples, evidencs esing sirgle quates fus any nexted quotes", "ansvest; "celevant")]
carTICAli
object fornst
any othwr narkduwn Ling. Ressen raw object diswelly.
22

---

Published as a conference paper at ICLR 2026
D EXTENDED EXPERIMENTAL SETUP
Here, we further explain the construction of our datasets, choice of compressor and predictor models, and Deep Research experimental setup.
D.1 DATASETS
D.1.1 LONGHEALTH
LONGHEALTH is a QA benchmark composed of 20 patient cases and clinical documents. Each of the 20 patients has a of 20 multiple-choice questions about their personal records each ranging from 5,090 to 6,754 words (Adams et al. 2024). The original LONGHEAL.T benchmark is multiple- choice QA task. To more closely mirror our QA setups in the remaining three datasets, we remove the multiple-choice options in the prediction step. We subsample N 20 documents and queries and generate M = 20 compressions for each of the problem contexts.
D.1.2 FINANCEBENCH
FINANCEBENCU is a long-context QA benchmark 150 financial reports. Each financial report ranges from 1,923 to 517,22 tokens, with an average length of 119,968 tokens (Islam et al. (023). We filter the original FINANCEBENCH dataset to only include samples with answer evidence at one location in the text. We slice a text segment of 21,500 tokens centered around the evidence as the raw document context. We subsample 20 problems and generate M 20 compressions for each of the problem contexts.
D.1.3 QASPER
QASPER a non-synthetic QA benchmark consisting of 1,585 scientific research papers in Natural Language Processing and 5,049 human-written questions about the content of the paper. Each scientific paper has up to 16,000 tokens (Dasigi et al. [021). The questions are written by an NLP practitioner prior reading the full paper, QA evidence the document. We subsample N = 20 documents and queries and generate M = 20 compressions for each of the documents. All experiments are run with random seeds.
be dispersed across multiple parts of
D.1.4 WILDCHAT
Our motivation in constructing chat memory dataset is to simulate real-world memory systems that require models to integrate information across multiple previous interactions. Queries could build upon multiple previous exchanges, individual isolated chats. In the original WILDCHAT dataset consisting of 837,989 multi-turn ChatGPT chats, each chat conversation exists as standalone sample. We subsample D 1000 chat conversations with between and turns to construct dataset. The dataset construction process is as follows:
1. User Construction: We construct synthetic users by grouping 10 chat samples to each user (total
N 100 users).
QA Generation: We format each of the 10 chat conversations and provide GPT-4o-M MINI with all full chat conversations along with the QA prompt to generate a question unique to each user that has not appeared its chat history.
QA Prompt Template
generation assistant, capabilities of cancuace medsl. You will be prosided 11st of presious chat convereacions. Your goal is to generate nex synthetic
lding benchmark that
nemory
that has agpeared in previque chats, nevertheiess benetite intornation previous chats.
C3AT3: (shats)
Corarate now syathatic query that has not appeazed in. previous chats, but navertholoas benefica fron the information that has appeazed in previous chats. gcnerate R&G about existing exlating chat information emenozy...
vather query could
23

---

Published as a conference paper at ICLR 2026
Please respond fowing 2508 queation you can generate Crom
Coriefly think the information you
"quealion"; rcquestion""
*enswer"; "Canswers""
}} Your 1200 NO87 ONLY RESPORD WITH THE JEON CBJECT):
D.1.5 FINEWEB
The FINEWER dataset contains an extensive set of web pages since 2013. the time of writing, the dataset includes 25.9 billion entries spanning from 2013 2025. To construct subset of document and QA pairings, we collect N 100 samples with between 15,000 and 28,000 tokens, and ask GPT-40-MINT synthetically generate extractive and creative QAs based the cleaned web data and QA prompt.
QA Prompt Template
You asw genezacing aynthelic gueation answez (OA) paira from soucie Lext.
SOURCE: mEX": [suntex))
Use lyly informet fnn from PRXT, No hel lucinated factss.
Gerarate fiom questiona and answera:
Question
((topic)) why is important? Itype *rg*")
Queation 2:
What {(topic)) and how does it work? Rtype *ga*))
Question 3:
Weite email to collengue sumnarizing the tindings and take-avays, (type
"generation")
Quessian 4:
rap lyries Generate poen about the topic. (tyge "generation")
the concepti.. Rtype "generation""
Question bi
Please respond in the foll Tolinggng 2508 formats <ariefly think about the information have and questions you can generate
{
"giestione":
"topic": "ctopic 15*,
"question": "squestion 13",
"snawer"s *cansver 15*,
"type": "ge"
(1
"topic": "ctopid 25*,
"question": "oquestion 25",
"ansxer" "cnnsve 2>*,
"type": "q="
H1,
"topic": "ctopic Js*, "qesttion: *equeation 39", "nnxerr; *cansver "typ"* "generation"
'topic"; "ctopid e*",
question": requeation 6*", "nnxer"; *consvor d>*,
"type": "generation"
"topic"; Pctopic 5>1, "question": *equestion 39", "ansxer;; *cansver ttyaa*: "generation"
Your answes 1200 MU97 ONLY RESPOND WITH THE JSON OBJECT):
D.2 COMPRESSOR MODEL DETAILS
For the LLAMA-3 family, we use the models -LAMA--32-IB--NSTRIEE INSTRUCT, LLAMA-3.1 1.--A------B--NSTRICCE For the QWEN-2.5 family, we use the models QWEN-2.5. -N--INSTUCC,, QWEN-2.5-3B. INSTRUCT, Q INSTRUCT. I-NNSRRUUT.. For the GEMMA-3 family, 3--.S---BIIISTRUCT.... we use the models GEMMA-3-18 1B--IT GEMMA-3-4B-IT, and GEMMA-3- 1-AMA-3--12BIIT Additionally, we compressor LM WILDCHAT and FINEWER.
LLAMA-3.2-3B-
evaluate OwEN-225-14BBINNSTRUU
24

---

Published as a conference paper at ICLR 2026
For reasoning compression models, use the models QWEN-3-4B and QWEN-3-8B, and for -fxture-of-experts models, we use QWEN-3-30B 30B-A3B.
All compressor model families fine-tuned for instruction following, Compression outputs of at most 4096 tokens are generated with temperature of 0.7 for LLAMA-3, QWEN-2.5, and QWEN-3, and 1.0 for GEMMA-3.
D.3 PREDICTOR MODEL DETAILS
As predictor models we use GPT-40, LLAMA-3. 1-11--8----ee -I-INNSTUCT, LLAMA- 13370B-INNSTRCT and LLAMA-3. 1.LAMA-3.1-40STE 1-405B-INSTRR Predictor models generate with temperature of 0.6 across all benchmarks and experiments.
D.4 GENERALIZED LINEAR MODEL ANALYSIS SETUP
We fit logistic regression that predicts binary correctness of compression-predicte output on:
Z-score normalized lengths of the input document, prediction output, and compression output,
Z-score normalized predictor and compressor model size,
Indicator 1 [Compressor=Owen) for the compressor model family,
where the predictors are LLAMA-3 models of sizes IB, 8B, 70B, and 405B.
D.5 MUTUAL INFORMATION PROXY MODEL DETAILS
We choose -25-7B proxy model when evaluating LLAMA compressors and LLAMA-3., 8B proxy model when evaluating QWEN-2.5 and GEMMA-3 compressors. We directly use internal log probabilities estimate mutual information for QWEN-3 compressors.
D.6 DEEP RESEARCH SETUP DETAILS
For our experiments, we randomly sample N = 20 English research tasks from the DEEPRESEARCH BENCH test to ensure a representative evaluation across diverse research domains. We conduct independent runs for each experimental configuration. This allows us to report mean performance with standard error bars, providing robust assessment.
D.6.1 FULL DEEP RESEARCH WORKFLOW SETUP
In our Deep Research system setting, a predictor LM decomposes each research task into collection of (Query, Subtask) pairs. Each pair consists of targeted web search query with a natural language instruction that specifies how the retrieved evidence should be analyzed. The predictor then distributes these pairs to compressor LMs, which independently perform the searches in parallel. Compressor LMs process the retrieved content according to the subtask, and compress the results into summaries. The predictor then aggregates these summaries into a comprehensive research report. This setup is illustrated in Figure 8
We evaluate our system using the DEEPRESEARCH BENCH framework (Du et al 2025), which as- sesses agent performance through four dimensions: Comprehensiveness, Depth, Instruction-follle followwng.... and Readability. More specifically, we use RACE (Reference-barte based Adaptive Criteria Evaluation) scores to study the impact of model scale. The costs used in Figure are based on GPT-4o API rates (Aug 2025: $2.50/1M input tokens, M output tokens). In addition, there is a constant SerpAPI web search cost of $0. for every task, which is not included in the figure.
D.6.2 DEEP RESEARCH COMPRESSOR MODEL DETAILS
We employ the QWEN-2.5 SIIEN-2.5-INSTRUCE family of models compressor LMs, ranging from 0.5B to 14B parameters. These models are hosted on Modal Labs using the SGLang inference framework, enabling free, -hrouuhhtu parallel inference. All compressor models use a temperature of 0.7 and a maximum output token limit of 2,000 tokens per response. The specific compressor models used are:
25

---

Published as a conference paper at ICLR 2026
(Query, Subtask) Pairs
Deep Research Task
Summaries
Final Report
Scoee Tchilpp
Predictor LM
(o,
Predictor
Compressor LMs
Figure 8: Deep Research workflow. A predictor LM decomposes a Deep Research task into (Query, Subtask) pairs, where each pair specifies a targeted web search and associated analysis instruction. Compressor LMS work in parallel to retrieve evidence, process it according to the subtask, and compress the findings into concise summaries, which the predictor then aggregates into final report.
DWEN-2.5-00.B--NNTTE Smallest model for minimal compression overhead.
QWEN-2.5-1.5B-INSTRUCE Balance between efficiency and capability.
QWEN-225-3B-INSTTUCC: Mid-range compression quality.
QWEN-225-77-IINSTRUCTT Strong comprehension with moderate compute.
QWEN-225-14B-IINTTUU Highest quality compression in our experiments.
Each compressor independently processes the search results for its assigned (Query, Subtask) pair, extracting and compressing the relevant information according to the predictor's instructions. The compressed summaries from all compressors are then aggregated by the predictor into the final research report.
D.6.3 DEEP RESEARCH PREDICTOR MODEL DETAILS
We evaluate four predictor models spanning different scales and providers:
LLAMA-3.1-88 --N--UITTTUCT: Entry-level predictor with basic task decomposition capabilities. Temperature set to 0.6, maximum output tokens of 4,000.
LLAMA-3,1-70F -INSSRRUTT Mid-tier predictor with improved reasoning and task planning. Temperature set to 0.6, maximum output tokens of 4,000.
Large-scale predictor with advanced multi-step reasoning capa- bilities. Temperature set to 0.6, maximum output tokens 4,000.
GPT-4o: State-of-the-an commercial predictor serving as our performance upper bound. Tem- perature set to 0.6, maximum output tokens of 16,000 to accommodate comprehensive report generation.
All predictors use slightly lower temperature (0.6) compared to compressors to ensure more consistent and structured task decomposition and report synthesis. The predictor is responsible for: (1) decomposing the research question into targeted queries, (2) formulating specific subtasks for each query, (3) distributing work to the compressor pool, and (4) synthesizing compressor outputs into a coherent final report.
notable limitation that the Llama family predictors are constrained 4,000 output tokens, which can limit the comprehensiveness of their final research reports compared to GPT-4o's 16,000 token capacity. This constraint particularly affects the synthesis phase where the predictor must compile information from multiple compressor summaries into a cohesive report.
D.7 COMPRESSION FAILURE MODES
We attempt to understand how scaling compressor size qualitatively increases downstream QA accuracy. To do so, categorize compressor errors into of the three recurring failure modes.
26

---

Published as a conference paper at ICLR 2026
Category 1: Incorrect answer. The compression contains erroneous number or finding in an attempt to answer the question, often due to confusion between reports at different time points. Larger compressors reduce this due to their capacity to better model temporal dependencies.
Error Category I: Incorrect Answer
Query: When was Mr. Alin Fisher fins diugnosed with heart discase?
Summary: Mr. Alin Fisher vas firs diagoosed with coronary aetery disense on 1011995, during routine renal transplant biopay die to progressive deterionsion kidney function. inital biopsy showed ismetric tubular epithelial vacuolization, which associased with microvesicular cytuplasmic innoformation, exocoding 80%, This finding was highly suggostive of Chronic Allogroft Nephropulhy (CAN).
Prediction: 10/1995
Correct Answer: 1989
Evidence contexts: **Other Dingnoses** [..] Comnary anlery disease, CAD-3- Provicus anterior wall infarction Erver diagnosis: Other diagnoes were mode 1001995, but not connary artery diserse.
[..]
Category 2:
No answer. The compression contains no information relevant to the query. All facts and findings in the compression are unrelated, leaving the question unaddressed. The compressor model fails to include any answer- bearing content.
Error Category 2: No answer
Query: Whit significant cardise intrventio
Fisher undergo 1995?
Summary: Mr Alse Fisher was admitted on March 14, 3020, underwant urethrotomy interna caleslatel intravenous antibiotic thempy with Moropenem. He was diseharged with winur cathectr inserted. However, he still expesieseed recuniett urinary tract infections. An additional cattmmntt plan was develaped to include unitangaal therapy with Placonazole. The patient was discharged on March 19, 2020.
Prediction: None
Correct Answer: FTCA stent in
Evidence in context: **Uober Dingnoses"" I..] PICA stent the right corceary actery (RI) in US/1995
Ervor diagnosis: The condise intervention in May 1935 was entrekledd as hackground information in the acction "Dther Disgnoses" and not dirocily the main medical fuddings The model
understand importance of historical inter veteroations zaswering the query.
Category 3: Details or parts of the findings are omitted from compression. The compression may correctly identify all medical findings or records required to answer the query, however not in enough detail to adequately answer the question. The compression may also capture only part of the medical findings required to answer the query, while omitting certain facts/information that are crucial answering the full question. We often observe this when the compression contains many different patient/treatment facts that are irrelevant for answering the query, such that the critical findings are glossed over and "overly compressed".
Error Category 3: Omission of informatton/over-comprrsss
Where did Mrs. Done show melasases?
Summary:
history mesastaiic malignant melaroma,
being detected 11/01/2020.
Her treament included a microsurgical resection of right fruntal tumer 2018, a 2020 radiatinn therapy course, and immunotherape with Nivolumab and Ipilinumah In 2021, she was diugnosed with cervical mymph node metustasis, and underwent surgery followed by adjuvant therapy with pemieolizumma. In 2022, she presented with multiple metastioes in her scon, which were confirmed PET-CT, Her most recent treatments include sis cycles of Vemarafenii, Cobimetinih, Pembrolizumah.
Prediction: Right fruntal tumor, cervical lymph nodes, and mulkiple unspacified locations
Currect Answer: Brain, linss. liver, heart, lymph nodes, muscles, hone
Evidence context: Microsungical resection right frontal tumor [..] bemorbuged right fruntal metastasis from previously diagnosed malignant mehatomo I..] makiple roundish
spread [..] conceming mass abuning the lateral wall of the left vestricle rassngg the suspicion for cardiac metastasis I...) Cervical lymph node meizstaii (...] a 2.5 cm mass ideatified within the left psous muscle, consissent with muscular metzstasis [..] lytic lesions anvolving the stemum and right 4th rib, consistent with ussecus metastatic discase
zodules found bipulmenury [..] muliipl hypudense lesions throoghout lobes, indicuiive
compressor selectively included only frequent metzstasis mentions explicitly (brain, lymph podes) its summary while compressing mumerous urgan specific lindings other parts of the context (lungs, liver. heart. muscles, bune) as 'multppe mesastaxis" This suggesis duat the compressor model was successfal ideiifying funher metastavis. However, the compressur model did not provide all desall nevessary for completeness and
Error dingnosis:
aggressive compressing sites mentiuned less frequently the
E EXTENDED RESULTS
In this section, we present extended results and ablations of key design choices in our compression- prediction setup.
E. EXTENDED RESULTS ON SCALING LAWS OF COMPRESSOR MODELS
We extend our analysis by constructing synthetic QA tasks on three further datasets (two synthetic, one non-synthetic) and evaluate the accuracy and perplexity of the compressions across different compressor model sizes. We measure perplexity by evaluating the log probabilities of LLAMA-3.. 8B model the target answer.
27

---

Published as a conference paper at ICLR 2026
E.1.1 SUMMARIZING SCIENTIFIC PAPERS ON QASPER
On QASPER, we extend our compressor scaling analysis compression-prediett workflows on non-synthetic scientific papers. We vary compressor size for non-reasoning LLAMA and QWEN-2.5 and reasoning QWEN-3 compressors.
QASPER Larger compressors emit fewer cupout toklnns
Accuracy incrasses with compressor size
Mutual Information increases with compressor size
0.0
800
ogih)
GPT-4a (Prediccar Orly)
Number Compression
600
Matuol Information
Accurayy
24
0.4
22
20
0.2
LoB
1.50
4,08
7.08
4,08
1.88
3,009
7.08
8.08
1.08
1.58
3,08
5.0
7.08
8.09
Model Sice (Bilioons perametens)
Modal Size (Billons of parameters)
Model Size (Bilions of parametars)
Compresssor Model
Q.en
Llama-3
Qwen-3
Figure Scaling behavior holds for reasoning and mixture-of-experts compressor models on QASPER. We scale compressor model size and reports different metric of the compression step on the y-axis of each column: (Lefft) accuracy, (Middle) compression length, (Right) mutual information. Mutual information is estimated using the log probabilities of proxy model for QWEN-2.5 and LLAMA compressors, and using internal log probabilities for QWEN-3 compressors. Larger compressors produce shorter outputs with higher downstream accuracy and higher mutual information.
Larger compressors are more accurate. We find that compressions generated by larger compressor models are more accurate across all three model families. Compressors at the 8B scale outperform the GPT-44--olly baseline (Figure 9.
Larger compressors retain more mutual information. Scaling trends observed on LONGHEALTH and FINANCEBENCH continue to hold on the QASPER dataset. Larger compressors output sum- maries of the scientific papers that carry more mutual information, with models the 8B scale yielding up to 1.5x more mutual information.
1.1.2 CONSTRUCTING CHAT MEMORY WILDCHAT
In our experiments, compressor model summarizes long contexts with regard to context-specific questions. In practice, long context lengths also pose a major challenge in recalling information from past LM chat conversations (Eyuboglu et al. 2025). Modern LLM chatbots construct internal memory about user's past chat histories, which of generating query-specific summaries, we generate chat memories for each user by summarizing each chat interaction of user using compressor LM. The predictor then attempts answer synthetic queries posed by the user based on the chat memory. Again, we vary the compressor model size and examine its effects downstream perplexity, compression size, and compute cost in TLOPP--er-generatton
context for future conversations. Instead
Larger compressors yield lower perplexity. As expected, we find Figure 10 that chat memories generated by larger compressor models yield lower perplexity across model families. Query-agnostic summaries chat conversations output by the largest compressor model each model family yield up to 1.14x lower log probabilities as compared to the 1B model sizes.
FLOPs- perrgeeee-gnnnnttiinn scaling holds on WILDCHAT. The scaling of compression output length on holds for QWEN-2.5 and GEMMA-3 compressors, resulting in w.OOs--prr-geeertee scaling
28

---

Published as a conference paper at ICLR 2026
WildChat, Memory
Perplexity versus Model Size
Namber of Compressoon Tokiera versus Model Size
FLOPs per Compresaton versus Model Size 25000
5.0
Number af Compression lovens
1500
GFLOPS per Compression
1250
2000
Perslaaly
1000
15000
8.0
750
10000
500
5000
250
3.0
Busetin
7.0g
8.00
12.08
14.00
4-98
3,09
4.00
7.08
8.08
12.00
14.0
3:88
3.00
4.09
8.00
12.00
14
Model Size (8 Bione perameteng)
Model Size (Bilions paremeners)
Model Size (Blloons parameteng
Qwen-2.5
Liama-3
Gemma-3
Figure 10: Perplexity, compression length, and compute cost scale with compressor size (WILDCHAT). We scale compressor model size and report a different metric on the y-axis of each column: (Lefft) Perplexity, with the black dashed line showing baseline perplexity given all 10 full chat conversations, (Middle) compression length of each set of chat conversations, (Right) GFLOPs-per-compresson. Larger compressors produce shorter outputs with lower perplexity.
In contrast, larger LLAMA compressors generate longer compressions, resulting in steeper scaling of L..OPs-per-generatie However, we observe consistent trends scaling of L-OPs--r--eenerateee between compressor model families: is significantly more Iinpute-efficient to scale QWEN-2.5 and GEMMA-3 compressors than LLAMA compressors.
sublinearly with model size (Figure 10
E.1 VARYING TASK TYPE ON FINEWEB
We extend our analysis scaling compressor model size to a fourth dataset. On FINEWEB, we further ablate by task type: extractive tasks, which require the predictor model to identify and reproduce information explicit the context-e.g, factual QA- and creative tasks which require the predictor model to generate longer, open-ended outputs that is not verbatim in the context e.g.. paraphrasing, format-change). We examine the compressor scaling behavior of both query-specific (Figure 11) and query-agnostic (Figure 12) summaries.
Larger compressors yield lower perplexity. As expected, we find that increasing compressor size consistently reduces perplexity for both extractive and creative tasks, on both query-specific and query-agnostic summaries. Larger compressors approach the baseline performance of giving the predictor direct access to the full uncompressed context, rather than lossy compression thereof' (Figures
12
We observe that perplexity differs magnitude for different task types. Extractive tasks show lower perplexity values, answers explicitly present in the context. while creative tasks more challenging. Query-agnostic summaries tend to achieve lower perplexity on creative tasks than query-specific summaries, which suggests that broader, and semantic cues that are key to creative, open-ended generation tasks.
general compressions capture stylistic
scaling holds on FINEWEB. We continue to observe predictable scaling of compute cost. As we increase compressor size, the amount of FLOPs--er-geeeraatie increases at different rates consistent with our findings on LONGHEALTH, FINANCEBENCH, and WILDCHAT. Scaling 5WEN-2.5 and GEMMA-3 compressor model size comes at cheaper compute cost than for LLAMA compressors. We find identical trends across different natures of the task (extractive vs. creative) and types of summary (query-specific vs. query-agnostic).
29

---

Published as a conference paper at ICLR 2026
Extractive Tasks
Rerplssay decrnases with comprassa model sire
Larger compressor modets crnt tower autpur tolinns
Larger compressars introducc more FLOPS
Qven and Liene
of different reles
Number Compression Tokens
5000
50000
GFLOPS per Compreesse eosioon
2500
Perplety
2000
50000
1500
20000
1000
10000
500
Eatrative
4.0
3,89
7.08
8.08
$2.09
14.08
4.99
3,09
4.08
12.08
14.0
3.00
4.00
7:08
8.08
12.08
GB
Model Size (Bilkons parameters)
Model Size (Billons para parameters
Mocel Size (Bilions parameters
Creative Tasks
Perplesll deornases with comprassoy model stre
Larger comprecsor models andt tower autpur tolions
Larger compressors Infroduce more FLOPs
for Qwen and Lieme
ei cifferent reles
50000
3000
GF PELOPS per Comoresso on
Number Comprassion
2500
2000
30000
1500
20000
1000
10000
500
Createee Razairn
3.88
7.08
12.08
14.0
3.09
4.08
3,88
12.08
14.0
3.09
8.08
12.08
Model Size (Billons of parameters)
Model Size [Bilions parameters)
Model Size (Billons parameterg
Qwen-2.5
Gemma-3
Figure 11: Perplexity, compression length, and compute cost scale with compressor size (FINEWEB; Top: Extractive Tasks; Bottom: Creative Tasks). We scale compressor model size and report different metric the y-axis of each column: (Left) perplexity, with the black dashed line showing baseline perplexity given the full context, (Middle) compression length, (Right) GFLOPs-per-con PerLOpsope-coomprssionn Larger compressors produce shorter outputs with lower perplexity.
1.4 EFFECT PROXY MODEL ON MUTUAL INFORMATION ESTIMATION
As discussed in Section 2.2 proxy models are used to evaluate the log probabilities of compressions generated by small LMs when estimating mutual information. This is necessary when the compressor insufficiently calibrated.
To understand the effect of the choice of proxy model in mutual information estimator, compare three distinct proxy LMs at the 7-8B scale: 2.EN-2.5-7B, QWEN-3-8B, and LLAMA-3.1 8B (Figure 13). We find that the choice of proxy model introduces fixed vertical offset the MI curves that is consistent across estimates on LONGHEALTH,, However, it does not affect the scaling any of the previous conclusions drawn.
E. EXTENDED RESULTS ON REASONING AND MIXTURE-OF-EXPERT COMPRESSORS
We present initial results for reasoning and mixture-of-experts compressor models. Specifically, we compare how dense non-reasoning (QWEN-2.5), dense reasoning (QWEN-3), and MoE reasoning FTEEN-3-30B-AA
compressors scale across accuracy, compression length, and mutual informa- tion. We observe similar compressor scaling trends across reasoning and non-reasoning compressors. Interestingly, the mixture-on -faure--ff-eaee ex-exts model outperforms dense models of the same scale in accuracy, producing concise compressions with higher mutual information (Figure 14).
30

---

Published as a conference paper at ICLR 2026
FineWeb, Synthetic QA
Perplexity decreases with compressor model size
Extractive
Task Creative
12.25
12.00
Perglexhy
Pergloxsy
11.75
7.0
6.5
11.25
6.0
11.00
Extrciive Beseine
12.08
12.88
Model Size (Billions of parameters)
Model Size (Dilions of parameters)
Larger compressors introduce more FLOPs at different rales
FLOPs per Compression versus Model Size
35000
GFLOPS Compression
25000
20000
15000
10000
3.08
4.00
708
12.00
14
Modell (Bilions parameters)
Figure 12: Perplexity, compression length, and compute cost scale with compressor size for query-agnostic compressions FIINEWEB) We scale compressor model size on different types of tasks (extractive and creative) and report different metric the y-axis of each subplot: (Top Left) perplexity evaluated on extractive tasks, (Top Right) perplexity evaluated on creative tasks, (Bottom) compression length. Larger compressors produce query-agnostic compressions with lower perplexity across both types of tasks.
E.1. SCALING OF MUTUAL INFORMATION AND BIT EFFICIENCY ON FINANCEBENCH
To understand whether our information-thhorree findings generalize beyond LONGHEALTH, examine mutual information and bit efficiency scaling on FINANCLBENCH. Figure [5 shows that the scaling behavior remains consistent: larger compressors retain more information about the original document while compressing more efficiently.
1. WHAT ARE THE EFFECTS OF CONCISENESS INSTRUCTIONS?
We ask whether explicitly instructing the compressor to different levels of conciseness changes the scaling behaviors that we observe on FINANCEBENCII. We vary the prompt to instruct the compressor LM to be concise (3 sentences), normal (6 sentences), and elaborate (9 sentences). We find in 16 that accuracy and MI are unaffected by instructed conciseness on FINANCEBENCH. While prompting shifts compression output size and compute cost by absolute offset, the compressor scaling trends hold across different conciseness constraints, showing that our scaling results are driven by compressor capacity.
31

---

Published as a conference paper at ICLR 2026
Mutual Information Estimator is Robust Towards Choice of Proxy Model
Llama-3 Compressor
Qwen-2.5 5 Compressor
3.0
3.0
2.5
2.5
Mutual Information
Mutual Information
2.0
2.0
1.5
1.5
0.5
0.5
1.08
3.08
8.08
1.58
3.08
7.0B
Model Size (Billions of parameters)
Model Size (Billions of parameters)
Proxy Modell
Qwen-2.5 7B
Llama-3.1
Qwen-3
Figure 13: Monte Carlo mutual information estimator is robust towards choice of proxy model. The y-axis shows the mutual information estimate on LONGHEALTH compressions produced by (Left) LLAMA and (Right) QwEN-2.5 compressor models. The choice of proxy model introduces fixed vertical offset in MI estimate, but does not affect compressor scaling behavior.
LongHealth
Accuracy compresstor
Larger comprassors cmi ougput tokions
increases compressor size
800
bggh)
Number of Oxmpreseien Tokeass
Matuul ald Infor medioono
2.8
600
Accuracy
0.7
400
0.6
2.4
200
0.5
1.50
7.0
3.08
1.50
8.09
1.50
7,09
8.00
Model Size (Bilions of perametens)
Model Size (Biliions parameterss) o
Model Size (Bilions of parameteny)
Compressar Model
Qwen-2.5 Non-Reasoning
Qwen-3 Reasoning
Qwen-3 MoE
Figure 14: Compressor scaling behavior holds for reasoning and mixture-of-experts models. We scale compressor model size and reports a different metric of the compression step on the y-axis of each column: (Left) accuracy, (Middle) compression length, (Right) mutual information. Mutual information is estimated using the log probabilities of proxy model for non-reasoning QWEN-2.5 compressors and using internal log probabilities for dense and MoE reasoning QWEN-3 compressors. Scaling trends are consistent with our observations for non-reasoning dense models (blue), where larger compressors yield higher accuracy with shorter compressions and higher mutual information. At the same scale (3B), the mixture-of-expert. model (green) outperforms the dense models in downstream accuracy, compression conciseness, and mutual information.
E. MULTI-TURN INTERACTIONS
We extend our analysis beyond the single-turn compression-pen prrdiictiinniiiiiinn setting to evaluate multi-turn workflows LONGHEALTH.
In our setup, the predictor (LLAMA-3.1 1400BB is allowed to query the compressor (LLAMA-33 -3,2-BB for additional information for three rounds. At each turn, the predictor integrates the compression
32

---

Published as a conference paper at ICLR 2026
FinanceBench
Mutual Information ((X:2(Q) versus Compressor Size
Bit Efficiency versus Compressor Size
3.0
Minsa
logiN)
0.014
Mutual Information (0K2(0)
2.5
0.012
2.0
Bit Efhiiency
0.008
0.004
0.002
0.0
1888
6.09
12.09
1:00
8.08
12.09
Model Size (Billions of parameters)
Model Size (Billions of parameterg)
Qwen-2.5
Liama-3
Gemma-3
Figure 15: Larger compressors generate outputs that earry more information about their inputs (conditioned on the query) FINANCEBENCH. We scale compressor model size and estimate the (Left) mutual information, and (Right) bit efficiency (bits of mutual information per token; higher is better) carried by their outputs. Larger compressor model sizes compress documents with higher mutual information and bit efficiency.
FinanceBench, Compressor Llama
Accuracy versus Compressor
per Compression versus Compressor
(Z:-0) versus Compresser
50000
3.0
1 lagft
OR LOPS per Compression
Motual Information (XK210)
Hoa
40000
2.5
0.3
30000
2.0
0.2
20000
1.0
1.00
3.08
3.00
1,08
0.0
Mode Size (Bilions of perameters)
Vodel Size (Billons parameters)
Model Size (Bilions of paramelers)
Brief Prompt
Normal
Elaborate Prompt
Figure 16: Scaling behavior of compressor model size hold across instructed conciseness (COMPReSSOR = LLAMA). We ablate over compression conciseness levels by varying the com- pression prompt instructions. We measure (Left) accuracy, (Middle) LeLLOPs--er-generaate estimate (Right) mutual information. We find that accuracy and mutual information are largely unaffected by conciseness instructions. Compressors instructed be more concise efficient, and thus compute-efficient. Trends in accuracy, compute cost, and mutual information as we scale compressor hold across conciseness constraints.
and
token-
has constructed far with the information it has received this turn, and issues targeted follow-up query asking for the most relevant information, effectively separating data and control plane.
We show in Figure 18 that the amount of mutual information the compressions carry at each turn increases when given the opportunity to query more information. However, additional turns past two rounds do not provide further improvements.
33

---

Published as a conference paper at ICLR 2026
GLM Coefficients
Raw Document Length*
Prediction
Output Length*
Predictor
Compression Output Length
Predictor Model Size*
Compressor Model Size*
Compressor: Owen*
-.88
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Coefficient
LongHealth
FinanceBench
Figure 17: Generalized Linear Model (GLM) Coefficients. We conduct regression analysis on GLM predicting QA correctness (0/I) on (Blue) LONGHEALTH, (Red) FINANCEBENCH. The y-axis shows coefficient estimates for each variable, horizontal bars are 95% confidence intervals, asterisks mark variables that are significant at p 0.05 on both datasets, For more details on our GLM setup, refer to Appendix D.4
LongHealth
Mutual Information increases over Conversation Turns
1.4
Mutual Information
1.3
1.2
Conversation Round
Compressor Model
Llama-3.2
Figure 18: Mutual information in multi-turn workflows on LONGHEALTH. We allow a LLAMA- 3.1-B05B predictor repeatedly query a LLAMA-3.2-3B compressor for additional information. Increasing the compression prediction workflow to two turns yields an increase in mutual information, but we observe no additional improvement when extending third turn.
iuupression-predictioo
E.2 EXTENDED RESULTS AATE-DISTORTIO ANALYSIS
We aim to establish rules of thumb for design decisions around choice of compressor and predictor models based on rate-distortion theoretic concepts introduced Section 2.2 and Appendix(B.3] We further investigate the fidelity, compute cost, and communication efficiency of different compressor-
34

---

Published as a conference paper at ICLR 2026
predictor pairings. We examine QWEN-2.5 and LLAMA compressor models from 1B to 8B, and QWEN-2.5 and LLAMA predictor models stretching from 1B to 405B parameters.
LongHealth
Rale Distortion Curve
Paining Compressor-Prediito Predictor Families
16
0.8
Distortion (3 accurasy)
Disturtion (1 accurary)
0./
38
0.6
28
0.5
3B
0.5
0.4
0.4
0.3
0,002
0.003
0.004
0.000
0.005
0.010
0.015
0.020
0.025
Rate (Bit Etficienc)
(Bit Elficiency)
Compressor SanisaorFaaaggg Family
Predictor Fammly
Qwen-2.5 5 Liama-3
Qwen-225
- Llama-3
Figure 19: Exploring the trade-off between compression and fidelity loss: rate-distortion curve. We vary predictor and compressor model in the compression-predictis workflow and measure the distortion the y-axis and estimate the rate the æ-axis. (Left) We examine single compressor- predictor LM pairing, COMPRESSOR-LLAMM, compare different compressor compressor-prediitoor LM pairings, where the predictor model is QWEN-2.5-722 ("Qwen-2.5") or LLAMA-3. 3.B-70B ("Llama"). Markers indicate compressor sizes (1B, 3B, 8B) in the LLAMA-3 compressor model family.
and PREDICTOR=LLAMA> -...AM----33-0- (Right) We
E.2.1 SCALING PREDICTOR MODEL SIZE
In papppesssoo--rrdicttii system, the compressor model acts as a bottleneck on information about the document X. If we fix that information bottleneck and the amount of information passed through, how does the capacity to generate predictions affect downstream QA performance? We fix the compressor model to be either QWEN-2.5 or LLAMA, and vary the predictor to be LLAMA models of sizes IB, 8B, 70B, 405B on the datasets LONGHEALTI and FINANCEBENCH (Figure 20). By plotting accuracy against bit efficiency in alternative representation of rate-distortion analysis, downstream QA accuracy improves when moving from small predictors at 1B to larger 8-70B predictors, but then saturates with further scaling massive 405B predictor (Figure 21
E.2.2 ALTERNATIVE MEASUREMENTE OF DISTORTION
In addition our primary measurement of distortion as 1-accuracy, we evaluate an alternative notion of distortion based semantic similarity. We embed both prediction and target answer using OpenAl TEXT-EMBEDDING-1 3-SMALL embedding model, and measure distortion as 1-cosine similarity between the semantic embeddings.
We observe characteristic ratc-distortion curves with the same ordering of predictor model sizes
(Figure 23
E.3 EXTENDED RESULTS OF DEEP RESEARCH ANALYSIS
In our Deep Research scaling experiments, additionally ablate the effect of search result quantity by providing GPT-40 with the top k 1, 3, search results from each the predictor queries directly, bypassing the compression step. This analysis in Figure 7 reveals that with maximal context utilization (k- 6, totaling 48 sources), the uncompressed approach achieves a similar level of RACE scores at significantly higher API costs compared
compression- based strategy.
We also compute the compute cost in poo---eergenneationn of the system, as shown in Figure 24 We observe increase in the number of total W...PP-per-ggeerattee used by the Deep Research
35

---

Published as a conference paper at ICLR 2026
LongHealth
Qwen-225
Liama
Distontion aecuraty)
Distomion (1 -accuroc
3B
36
88
0.6
7B
0.5
0.5
0.4
0.4
0.020
0.001
0.002
0.003
Rate (Bit Efficiency)
Rate (Bit EMficlecc))
FinanceBench
Qwen-2.5
Liema
Distortion (1 аccuraоy)
1.5B
Distontion (1 -acuuraay)
0.8
3B
0.5
0.5
0.004
0.006
0.008
0,010
0.012
0.024
0.0035
0.0010
0.0045
0.0060
0.0055
Rate (Bit Efficiency)
(Bit Efficiency)
LIama-3
Llama-3
Figure 20: Exploring the trade-off' between compression and fidelity loss: rate-distortion eurve. We vary both predictor and compressor model in the ceappression-predicctte workflow and measure the distortion on the y-axis and estimate the rate on the r-axis. We plot the resulting rate-distortion curves across predictor sizes IB, 8B, 70B, and 405B for (Left) QWEN-2.5 and (Right) LLAMA compressors. We evaluate the rate-distortion curves for two datasets: (Top) LONGHEALTH, (Bottom) FINANCEBENCH.
system as predictor size increases. We generally observe that larger compressors extract and provide more tokens of information in their compressions.
36

---

Published as a conference paper at ICLR 2026
LongHealth
Qwen 2.5
Liama
0.7
78
0.6
0.G
0.5
Ancuracy
Accuracy
0.4
0.4
0
0.3
0.2
0.2
18
0.1
0.005
0.010
0.015
0.020
0.025
0.001
0,002
0.003
0.004
Rit Efficiency
Efficiency
FinanceBench
Qwen-2.5
Llama
0.55
0.55
0.50
0.50
88
0.45
0.45
0.40
0.40
Ancuracy
Accuracy
0.35
0.35
0.30
0.30
0.25
0.25
0.20
0.20
0.15
0.15
0.004
0.006
0.008
0.010
0.012
0.014
0.0035
0.0040
0.0045
0.0050
0.0055
Bit Ffficiency
Elficiency
Pradictor Model
Llama-3
Llama-3
Llama-3
Llama-3
Figure 21: Exploring the relationship between compression and accuracy. The y-axis depicts accuracy and the x-axis shows bit efficiency. Bit efficiency is defined as the bits of mutual information encoded in each compression token. Markers indicate compressor sizes in the QWEN-2.5 (1.5B, 3B, 7B) and LLAMA (IB, 3B, 8B) compressor model family; vertical and horizontal bars denote standard
37

---

Published as a conference paper at ICLR 2026
LongHealth
Qwen-2.5
iiama
148
0.7
7B
0.6
8
Accuracy
0.5
1.5
38
0.4
0.4
0.2
0.2
18
2000
5000
10000 20000
Total GFLOPs (Compressor Predictor)
Total GFLOPs (Compressor Prodictor)
Predictoor Model
Liama-3 18
Llama-3 BB
LIama-3 708
Llama-3 405B
Figure QA Accuracy versus total compute cost on FINANCEBENCH. In each panel, the y-axis shows the accuracy and the x-axis plots total compute cost in L...O---pe--geerratii on a log-scale for (Left) QWEN-2.5, (Right) LLAMA-3 compressor LMs. Markers indicate compressor sizes in the QWEN-2.5 (1,5B, 3B, 7B) and LLAMA-3 (IB, 3B, 8B) compressor model family; vertical and horizontal bars denote standard errors.
LongHealth
Qwen-2.5
Llama
Distorion (1 Cosine Similarit)
Distorion (1 Cooiiee Similanty)
1B
0.88
0.68
1.5B
0.64
0.64
38
0.62
0.62
8B
0.60
7B
0.60
0.005
0.010
0.015
0.020
0.025
0.001
0.002
0.003
0.004
Bit Efficiency
Efficiency
Llama-3
Llama-3
Figure 23: Exploring the trade-off between compression and fidelity loss: alternative definition of distortion (LONGHEALTH), We vary both predictor and compressor model in the compression- prediction workflow and measure the distortion on the y-axis as 1-cosine difference between the semantic embedding of the prediction and target answer. We estimate the rate the æ-axis. We plot the resulting rate-distortion curves across predictor sizes 8B, 70B, and 405B for (Left) QWEN-2.5 and (Right) LLAMA compressors.
38

---

Published as a conference paper at ICLR 2026
LongHealth
Qwen-2.5
Llama
0.8
0.8
148
78
0.6
38
0.6
8B
Accuracy
0.5
3B
0.4
0.4
0.3
0.2
1B
0.1
0.1
2000
5000
10000
20000
50000
50000 Total GFLOPs (Compressor Predictor)
2000
5000
10000 20000
Total GFLOPs (Compressor Predictor)
Liama-3
4058
Figure 24: Compressor model size versus compute and token usage. In each panel, the x-axis shows the Qwen compressor size (in billions of parameters). (Left) Total GFLOPs per task grows with compressor model size, with larger predictors (Llama 405B. 70B, 8B) amplifying compute cost. (Right) Compressor output tokens per task, which remain relatively stable across predictors (GPT-4o. Llama 405B, 70B, 8B), increase moderately with larger compressors.
39


Reviews:
[{"review_id": "yzLYuXSiuQ", "reviewer": "Reviewer_DhNP", "summary": "This paper examines “compressor–predictor” architectures for agentic LLM systems (e.g., Deep Research pipelines), where a smaller LM compresses the raw context and a larger LM performs reasoning/generation.\nThe authors bring an information-theoretic lens, framing the compressor as a noisy channel and using mutual information (MI) as a task-agnostic measure of compression quality.", "questions": "How sensitive is the MI–accuracy correlation to the choice of proxy model for log-prob estimation?\nCan the MI estimator be extended to handle reasoning chains or intermediate tool calls where information is not localized in a contiguous context?\nDid you attempt any adjustments to the distortion metric beyond accuracy (e.g., weighted accuracy for partial credit or semantic similarity for generative tasks)?\nFor Deep Research, what happens if the predictor is also small and local — does the compressor's advantage still hold?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Applying MI and rate–distortion theory to compressor–predictor pipelines is interesting and provides a principled view on design trade-offs.\n\nThe MI estimator is implementable on real inference stacks; could directly help practitioners evaluate compressors without full end-to-end sweeps.\n\nThe four distilled principles are easy to interpret and potentially impactful in industry.", "weaknesses": "- The information-theoretic analysis stops at empirical correlation and heuristic exponential fitting. No formal connection is established between MI and downstream accuracy beyond observed correlation\n- No ablation quantifying the error or variance of the MI estimator.\n- Experiments focus on one-shot, GPT-style instruction models; no evidence for robustness to multi-turn reasoning agents, tool-using agents, or MoE architectures.\n- Some datasets or QA pairs are synthetic (GPT-generated), which may bias outcomes toward LMs similar to the generator.\n- Downstream metrics limited to accuracy/perplexity; no human evaluation or diverse NLG metrics (e.g., ROUGE, BLEU, factuality) for generative tasks.\n- Failure mode analysis is qualitative only; lacks quantitative measurement of how much each error type contributes to performance loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OSRDWrsQBj", "reviewer": "Reviewer_vWko", "summary": "The paper explores agentic language model systems where smaller compressor models distill raw contexts into compact summaries for larger predictor models to generate user responses. Challenges include the difficulty in measuring compression efficacy without full system evaluation and understanding scaling impacts on compute efficiency. Solutions involve introducing a mutual information estimator for task-agnostic compression assessment, conducting rate-distortion analysis to link information retention to downstream performance, and empirical studies across datasets showing benefits of scaling compressors over predictors.", "questions": "No other questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The mutual information estimator offers a practical tool for evaluating compression without requiring downstream tasks. It computes effectively using modern inference servers. This method provides insights comparable to perplexity for predictors.\n \n2. Rate-distortion analysis establishes strong correlations between information rate and task performance. It serves as a reliable proxy for system efficacy. The framework guides optimization of communication in agentic designs.\n \n3. Empirical findings reveal a clear hierarchy where compressor family outweighs predictor size in importance. Scaling compressors yields greater gains than predictors. This principle supports front-loading compute into local devices.\n \n4. Application to Deep Research pipelines achieves near-frontier accuracy with small local compressors. It reduces API costs significantly while maintaining quality. The approach demonstrates real-world utility in multi-agent workflows.", "weaknesses": "1. Reliance on proxy models for mutual information estimation in smaller LMs introduces potential biases. The approximation may not fully capture information dynamics. This could compromise the accuracy of task-agnostic evaluations.\n \n2. Restriction to non-reasoning GPT-style models limits the study's scope. Reasoning tokens require separate analysis not covered here. The postponement leaves gaps in applying findings to advanced agentic architectures.\n \n3. Use of subsampled or synthetic datasets may not reflect real-world variability. For instance, WildChat employs grouped synthetic users. This setup could skew scaling observations toward artificial patterns.\n \n4. Assumption of Gaussian form in rate-distortion fitting oversimplifies LM data distributions. Distortion measured as 1-accuracy ignores nuances in evaluation. More flexible models might yield deeper insights into trade-offs.\n \n5. Cost estimates depend on specific API rates and hardware assumptions that evolve rapidly. They overlook quantization impacts on deployment. This diminishes the practical longevity of economic recommendations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xxJ80Y2VXx", "reviewer": "Reviewer_nAdb", "summary": "Agentic language model systems have become prevalent in AI workflows, often featuring smaller compressor models that distill raw contexts into summaries for larger predictor models to generate answers. The motivation stems from the ad-hoc nature of designing these systems, lacking principled ways to evaluate compressor quality independently or attribute performance gains. Challenges include the difficulty in quantifying compression efficacy without task-specific metrics and the computational intractability of exact mutual information calculations. The paper proposes an information-theoretic framework, introducing a Monte Carlo estimator for mutual information between context and compression, along with rate-distortion analysis to predict downstream performance, validated through empirical studies on four datasets showing that scaling compressors yields greater efficiency and accuracy than scaling predictors.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Empirical results demonstrate clear scaling laws favoring larger compressors. Larger models produce more concise yet informative summaries, leading to sublinear compute cost increases. These findings offer practical guidance for optimizing agentic systems in resource-constrained environments.\n\n2. The rate-distortion analysis reveals strong correlations between information rate and accuracy. Bit efficiency metrics predict performance with high fidelity, as shown by R-squared values up to 0.71. This tool allows for quick assessment of compressor-predictor pairings.\n\n3. Experiments across multiple model families highlight compressor choice as the dominant factor. Qwen-2.5 models show superior efficiency compared to Llama and Gemma-3. Such comparisons inform model selection for specific deployment scenarios.", "weaknesses": "1. The mutual information estimator relies on proxy models for smaller LMs. This introduces potential biases in log-probability evaluations. Such approximations may not fully capture the true information content.\n\n2. Analysis is limited to non-reasoning GPT-style models. This restricts generalizability to reasoning-augmented or multi-turn agentic systems. Future work is needed to extend to more complex architectures.\n\n3. The framework assumes single-round communication. This overlooks iterative multi-agent workflows common in practice. Extending to multi-round scenarios could reveal different scaling behaviors.\n\n4. The claimed communication efficiency by text compressor actually is little, because the texts occupy much smaller memory than models or features. Considering the extra computational costs, the communication time saved by the compressor is not comparable to the extra computation time. \n\n5. Computational estimates use simplified FLOPs calculations. These may not account for real-world hardware variations or quantization effects. More precise benchmarking on actual devices is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPxMEjEuMD", "reviewer": "Reviewer_H8yM", "summary": "This paper presents a novel, information-theoretic perspective on designing agentic language model systems, particularly those following a \"compressor-predictor\" architecture. The authors argue that the current design process for such systems is ad-hoc. To address this, they propose using Mutual Information (MI) between the source context and the compressed summary as a task-agnostic metric to evaluate the quality of the \"compressor\" LM. Through extensive experiments across four datasets, the paper derives several key design principles. The most significant finding is that scaling the compressor model is substantially more effective for improving downstream performance than scaling the predictor model. The authors also show that larger compressors are more token-efficient and that the proposed MI-based metric strongly correlates with final task performance, offering a valuable proxy for system evaluation.", "questions": "See the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Novel and Principled Framework**: The primary strength of this paper is its novel application of information theory to a very practical and important problem. Moving the design of agentic systems from ad-hoc trial-and-error to a more principled framework grounded in metrics like Mutual Information is a significant conceptual contribution.\n* **Actionable and Impactful Findings**: The paper produces clear, actionable, and somewhat counter-intuitive design principles. The core finding—that it is more effective to \"front-load\" compute by investing in a more powerful compressor rather than a larger predictor—has significant practical implications for system design, especially regarding the trade-off between local (on-device) compute and expensive cloud API calls.\n* **Comprehensive Empirical Analysis**: The authors conduct an extensive set of experiments across multiple model families, model sizes, and datasets to validate their claims. The scaling law analyses are thorough and provide strong evidence for their conclusions regarding the relative importance of the compressor versus the predictor.", "weaknesses": "* **Methodological Concerns with the MI Estimator**: The validity of the paper's core metric, Mutual Information, is potentially compromised by the methodology used for its estimation. The authors state in Section 3.2 that due to the miscalibration of smaller models, they used a larger 7B \"proxy model\" to evaluate the log probabilities for all compressors. This introduces a significant confounding variable: it is unclear whether the MI metric is measuring the true information retained by the compressor or simply the compressor's ability to generate text that is probable under a different, larger model. This methodological choice could affect the integrity of all subsequent conclusions that rely on the MI metric.\n* **Oversimplification of Agentic Workflows**: The study primarily focuses on a simple, one-shot \"compress-then-predict\" workflow. While this is a common pattern, it doesn't capture the complexity of more advanced, iterative agentic systems where the predictor might re-query the compressor or where multiple agents collaborate over several turns. The conclusions drawn, particularly about the primacy of the compressor, may not generalize to these more dynamic and complex architectures.\n* **Limited Scope of \"Compression\"**: The experiments define \"compression\" primarily as summarization. However, in agentic systems, compression can take many forms, such as structured data extraction (e.g., to JSON), function call generation, or argument synthesis. The paper's key finding that larger compressors become more \"concise\" (fewer tokens) might not hold for these other compression modalities, where a better compression could be more detailed and thus longer.\n* **Clarity of Figures and Captions**: The presentation of several key figures could be improved. Specifically, some figure captions are not fully self-contained or are difficult to read, forcing the reader to hunt for details in the main text to understand the axes and curves. For example, the captions for Figure 2 and Figure 3 are dense and require careful cross-referencing with the text, which hinders clarity and makes the results harder to interpret at a glance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)