Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoning
Authors: Adnan Oomerjee
Zafeirios Fountas
Haitham Bou Ammar
Jun Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Information bottleneck
Generalisation
Large Language models
Latent space reasoning
Representation learning
Memory consolidation
KV-cache compression
Predictive encoding
Reasoning
Information theory
Primary area: generative models

Abstract: Transformer LLMs have been shown to exhibit strong reasoning ability that scales with inference-time compute, most prominently through token-space “thinking” (i.e., chains of thought). A growing line of work pushes this extra computation into the model’s latent space (adjacent to standard decoding) which we term Auxiliary Latent-Space Computation (ALSC). Existing ALSC methods largely fall into three buckets: (i) token-mediated latent or special-token rollouts, (ii) residual/activation steering, and (iii) memory compression via cache pruning, merging, or summarization. An underexplored alternative is memory consolidation and reconsolidation, two processes in the brain that are responsible for stabilising newly formed memory traces, and, upon recall, transiently rendering established traces plastic such they can integrate new contextual information before restabilising. In a Transformer LLM, this can be seen as analogous to performing in-place global rewrites of incoming KV segments, and rewrites of past segments conditioned on newly observed tokens. In this work, we give a theoretical justification as to why memory (re)consolidation via KV cache rewrites is beneficial for improved reasoning. We do this through the lens of Information Bottleneck (IB) theory, which posits that model generalisation emerges from an optimal balance between input information compression and retention of predictive information in latent representations. We prove using IB theory that Vanilla decoder-only Transformers are inherently constrained in their ability to form task-optimal sequence representations. We then introduce the Bottlenecked Transformer, which augments a decoder-only backbone LLM with a lightweight Cache Processor, an auxiliary Transformer that performs periodic, non-causal, in-place KV rewrites at newline-delimited reasoning step boundaries. The processor consolidates recently written KV entries and reconsolidates a small, top-$k$ attention-selected set of prior entries, conditioned on recent context. We evaluate our Bottlenecked Transformer architecture on seven mathematical reasoning benchmarks, with four backbone LLMs. Our model sees consistent performance gains over vanilla Transformers and pause-token augmented Transformer baselines, with gains of up to +6.6pp for selected tasks and backbones.

Published as a conference paper at ICLR 2026
BOTTLENECKED TRANSFORMERS: PERIODIC KV CACHE CONSOLIDATION FOR GENERALISED REASONING
Adnan Oomerjee
Zafeirios Fountas
University College London
Huawei Noah's Ark, UK zafeirics. www.taauttntttttt.aaaaa com
Huawei Noah's Ark, UK adnan. oomer 220uc1. acc...lac.... uk
Haitham Bou-Ammar
Jun Wang
Huawei Noah's Ark, UK University College London
University College London Jun wangeucl. ac
haitham. a wwwwweirwwwwwwaaa..aaeaa com
ABSTRACT
Transformer LLMs have been shown to exhibit strong reasoning ability that scales with inference- time compute, most prominently through token-space "thinking" chains of thought. A growing line of work pushes extra computation into the model's latent space, which we term Auxiliary Satent-Space Computation (ALSC). Existing ALSC methods largely fall into three buckets: (i) moken-mediated latent rollouts, (ii) residual/activation steering, and (iii) memory (KV) compression. An underexplored alternative is memory ouusolidation/reonnooiitttoooo two processes in the brain that are responsible for stabilising newly formed memory traces, and, upon recall, transiently rendering established traces plastic such they can integrate new contextual information before restabilising. In Transformer LLMs, this can seen as analogous to performing in-place rewrites of new KV segments, and rewrites of recalled past segments. In this work, we give theoretical justifica- tion as to why memory (re)consolidation via KV cache rewrites beneficial for improved reasoning. We do this through the lens of Information Bottleneck (IB) theory, which posits that model generalisation emerges from an optimal balance between input information compression and retention of predictive information in latent representations. We then introduce the Bottlenecked Transformer, which augments backbone LLM with Cache Processor, an auxiliary Transformer that performs periodic, non-causal, in-place KV rewrites at newline-delimited reason- ing step boundaries. The Processor consolidates recently written KV entries and reconsolidates small, top-k attention-selecter set of prior entries. We evaluate our Bottlenecked Transformer architecture on math reasoning benchmarks. Our model sees consistent performance gains over vanilla Transformers and pause-token augmented baselines, with gains of up to +6.6pp for selected tasks/backbones.
INTRODUCTION
Transformer-baie based large language models (LLMs) have achieved strong results in retrieval, pattern recognition, and knowledge extraction (Brown et al. 2020 Chowdhery et al., (202)). With carefully engineered prompts/post-trannngg
they can also display nontrivial reasoning behaviours (Wei et 2023; Shao et al. 2024, DeepSeek-Al et al. 2025). A critical development that has significantly advanced Transformer LLMs is the discovery that reasoning performance scales strongly with inference-time compute. The most widely applied example of this has been seen in "reasoning" models, which generate verbal chains of thought before giving final answer (Wei al. 2023).
A growing body of work extends this idea to algorithms that allow LLMs to perform additional compute during generation directly in a latent space rather than the token space. We refer to these as Auxiliary Latent-Space Computation (ALSC) methods, which facilitate computation over internal continuous states during inference without emitting intermediate natural- natural-language tokens, doing so in addition to (or in place of) the standard one-forward foooward-pass-eee pass-per-token decoding strategy. In

---

Published as a conference paper at ICLR 2026
this work we focus sequence-level ALSC: operators that intervene between decoding steps to transform the model's KV cache and/or final hidden state before the LM head, which constitute the model's 's internal representation of processed sequence. Auxiliary Latent-Space Computation is potentially more efficient than strict autoregressive decoding as latent embeddings can encode semantics more compactly than token sequences. Additionally, such processes align more closely with human cognition, in which thought does not occur as an endless verbal monologue, but contains nonverbal stretches of conceptual processing that proceeds without recruiting the language system (Alderson-Day & Fernyhough 2015 Fedorenko & Varley 2016 Monti et al. 2012).
Prior sequence-level ALSC approaches primarily fall into three categories: token-space latent stepping, activation-space edits, and cache-operators (most commonly compressive schemes such as pruning, merging, or summarising KV entries). An underexplored direction is incorporating processes for memory consolidation and reconsolidation in the neuroscientific sense. Consolidation is process in the brain where new memory traces are stabilised upon formation. Reconsolidation refers to rewrites of recalled memories: when a stored memory is reactivated, it can briefly enter a plastic state in which it can be modified before restabilising, allowing it to be updated and recontextualised with new salient information (Lee 2009 Hupbach et al. (2007).
In this paper, we explore consolidation and reconsolidation in Transformer LLMs from both a theoretical and architectural standpoint. We adopt a working interpretation in which the KV cache serves as the model's memory and (re)consolidation is realised through periodic in place edits to that memory during generation. We first offer an information-theortiic justification for why periodically reprocessing the model's working memory (KV cache) should aid generalisation from the lens of Information Bottleneck theory; concretely, we show that in autoregressively trained models, the KV cache is incentivised to preserve information from the sequence history that is unnecessary for future sequence-lewe prediction, potentially hindering generalisation. We then introduce the Bottlenecked Transformer, which augments a pretrained backbone with a Cache Processor, small Transformer that periodically rewrites recent memories (consolidation) and selectively recalled K entries (reconsolidation) in- in-place, without dimensional compression. Our architecture is shown in Figure [] Note that whilst our aim is to implement a mechanism functionally analogous to consolidation and reconsolidation in the brain, the underlying biological processes are richer and more complex than our computational abstraction. Empirically, we yield consistent gains over vanilla Transformers across seven mathematical reasoning benchmarks and multiple backbones.
2 PRELIMINARIES
In all following sections, we denote random variables by uppercase letters (e.g. X, Y, Z) and realisa- tions by the lowercase letters (e.g. z, y, 2).
2.1 STATE-SPACE FORMULATION OF DECODING
Autoregressive decoding in Transformer can be viewed as state-space process, where the model's vey-value (KV) cache its memory state. Formally, let x; be the input token at decoding step t. For a model with L layers, we let h,, denote the KV cache (covering tokens 0 (), E Rd the final-layer residual stream. We then model the next-token decoding process as:
h4 {(hoo 1
(ht, 01) = (h. x2), p(re+1 he, 0z) softmax (fhend
This view treats ht as the sequence-level latent state that mediates future predictions, while o, summarizes the current step's computation. Under vanilla LLM, updating ht amounts to per-layer appends of the key-value vectors produced for the incoming token Tt.
2.2 SEQUENCE-LEVEL AUXILIARY LATENT-SPACE COMPUTATION
We call an auxiliary latent-space computation (ALSC) method any inference-tinee procedure that performs extra computation over internal continuous states adjacent the standard forward pass of backbone L.M. We focus on sequence-level ALSC that acts on (h,, o.) via
(h', o') T(h,, ot);
2

---

Published as a conference paper at ICLR 2026
invoked according to schedule s(t) C N (e.g., periodic every m steps event-triggered) After applying T, decoding resumes from the transformed state:
p(aet1 ,',o') softmax((nend(
RELATED WORK
We classify sequence-level ALSC works into three execution pathways: (i) Token-mediated, (ii) residual-operator, and (iii) cache-operator.
ii Token-mediated. These methods instantiate T to be an LLM (often the backbone LLM itself), and operate via an internal micro-sequence of latent tokens, thereby lengthening the cache and updating (be, 0x) via standard forward pass. Basic variants inject pause or filler tokens during decoding (Goyal et al. 2024; Pfau et al. 2024). Cache Deliberation uses an external coprocessor (often initialized from the backbone) to produce latent embeddings conditioned on hy and append them to the cache (Liu et al. 2024a). Other approaches recycle the model's last hidden state as a continuous latent fed back as the next input embedding. taking multiple latent steps without emitting
text until termination (Hao et al. 2024 Shen et al., 2025 Su et al., 2025).
(ii) Residual-operator: A complementary line defines T to only modify the current hidden represen- tation or before the LMM head, leaving h, unchanged. Activation steering adds structured directions to ot (or selected layers) to influence style, stance, safety (Turner al., (024), including Contrastive Activation Addition (Panickssery et al. (024) and exemplar-derived "style vectors" (Konen et al.)
2024). Recent work targets sparse features for precision and interpretability, e.g., SAE-targeted
steering (Chalnev et al. 2024). operations directly in SAE latent space (FGAA) (Soo et al. 2025).
and broader SAE-based frameworks (He al. 2025).
(iii) Cache-operator. In these methods, T operates solely to transform the memory he Here, the cache transformed directly between decoding steps to control what information remains accessible. Existing methods are predominantly centred on memory compression for long context tasks. Evic- tion/pruning methods retain high-utility entries using importance or heavy-hitter policies, preserving pivotal tokens and stable "sink" anchors (Zhang et al. 2023 Xiao et al. 2024). Merging/aagrreaation mechanisms fuse entries into representatives under a memory budget (Zhang et al.) 2024; Wang et al.) 2024). Recurrent architectures summarize older activations into compact memories memory banks (Transformer-XL. Compressive Transformers, RMT) (Dai et al. 2019 Rae et al. 2019 Bulatov et al.,
2022). Selective recall architectures externalise long histories memory banks and re-inject salient
slices on demand (Fountas et al. 2024).
interpret it belongs to the cache-operator family, but differs from the predominantly compression-oriented approaches above. In-place memory rewrites under (re)consolidation do not necessarily entail reduction in memory footprint. Rather, our goal is to demonstrate how KV rewrites may improve reasoning performance in Transformer LLMs.
Positioning. Memory (re)consolidation
4 MOTIVATION AND THEORY
In this section, we give a theoretical analysis as why a mechanism for (re)consolidation via KV rewrites is likely to improve on performance reasoning tasks in Transformer decoder-only LLMs, from the perspective of Information Bottleneck Theory. All proofs are given in Appendix
4.1 THE INFORMATION BOTTLENECK METHOD
The Information Bottleneck (IB) is framework for optimising some latent variable Z to be maximally informative of some output variable Y and minimally informative of an input variable X, via the objective:
[[p(z|z))= 1(x; - ((2;Y)
(I)
subject to Y ++ X ++ Z. Here 3 0 balances information compression via lowering T(X: Z) with relevance T(Z; Y) (Tishby et al. 2000). Controlling T(X; Z) has been proven to bound test set generalization eтror e < ((v(I Z) 1)/n) for i...d. data (Kawaguchi et al. 2023 with strong empirical results that suggests this extends to non-i.i.d. time senies data (Feng et al. 2024
al. 2024b Ullmann al. 2023 Choi & Lee 2024).
3

---

Published as a conference paper at ICLR 2026
Consolidation Recent tokens
Re-consolidation Retrieved tokens (k)
kv= (kv)
KV cache
RSW
Figure 1: Bottlenecked Transformer architecture, consisting a backbone LLM process- ing/generating tokens, and Transformer Cache Processor that rewrites KV entries. The Cache Processor is invoked each time a newline token is generated (marking the end of a reasoning step). When invoked, recent tokens (from the recent step window in grey) and retrieved tokens beyond the RSW (in blue) are passed in parallel to the Cache Processor, and rewritten in-place.
4.2 INFORMATION BOTTLENECKS AND DEEP LEARNING
In practice, one can approximate true distributions p(=[z) and p(yla) by parameterized distributions (2(2] æ) and ( 2). The joint model (a y, 2) now factorizes as:
Pe (z, y, 2) = p(a po p (v/=)
and we optimize & with respect to (ø, vb) To ground these ideas, we now introduce three formal definitions that capture the essential properties and ordering of information bottlenecks in neural networks.
Definition 4.1 (Neural Information Bottleneck). Let Mg be a neural network parameterised by 0, with input/output variables (X, Y), and let Z be latent variable within the model. Then, Z is an information bottleneck in Ma if and only if Z satisfies the Markov chain X + Z Y.
Definition 4.2 (Ordering of Bottlenecks in Neural Networks). Let [Zifier be a set of distinct information bottlenecks Ma. We say that a bottleneck Zj is deeper than another bottleneck Zü, denoted by Z, < Zj. if and only if + Z: + Z,
Definition 4.3 (Terminal Bottleneck). Let ZMMu be the set of all information bottlenecks Ma. Then 2 max Me is denoted the terminal bottleneck in Ma,
The notion of ordering of bottlenecks allows for the observation that the complexity [(X; Z) of any arbitrary bottleneck bounded by I(X; 2), the complexity of the terminal bottleneck, formalised in Lemma 4.1
Lemma 4.1. Let Mg be model parameterized by 0, with input/output variables (X,Y), and let M be the set of information bottlenecks in Mo. with 2 defining the terminal bottleneck in Mo- Then I(x; I(X; 2) for any bottleneck Z E Ma.
Implicit Information Compression During SGD Even without explicit IB loss, noise inherent to stochastic gradient descent (SGD) has been shown to implicitly minimise [(X; Z) in neural networks. During training, after an initial "fitting" phase, SGD hase been shown enter low-signal- to-noise "diffusion" regime in which gradient noise dominates, systematically compressing input information in hidden representations (Shwartz-Ziv & Tishby 2017
Butakov et al. 2024).
4.3
IB OBJECTIVE FOR GENERALISED LANGUAGE REASONERS
The problem of learning generalised Linguage-basee reasoner can be formulated as one of learning generalised sequence sequence model via the IB objective. Given an input X So.n (a reasoning history of transitions), and Y Su+1 (a subsequent reasoning step), we seek to train model Ma aiming to predict S,+1 given Sun. Under the IB method, given a neural information bottleneck Z in My that sequentially mediates (So.,, Sn+1)
1), we define the IB objective:
8" = arg min I (S Z) ((
Under this objective, a realised latent z should act as abstraction of the reasoning history to infer a generalised state of partial solution, from which some logical rule of inference be applied.
4

---

Published as a conference paper at ICLR 2026
4.4 ANALYSIS OF INFORMATION BOTTLENECKS IN DECODER- ONLY TRANSFORMER LLMs
Our first main result is given in Theorems 1.1 and 4.2 We show that in a decoder-only Transformer, given a imput sequence So:n and output sequence Sn+1, the KV cache and last hidden state computed from Sun forms the terminal bottleneck 2 mediating these sequences, and autoregressive training maximises both T(S::; 2) and T(2: S2+1)-
Theorem 4.1 (KV-Cache and Final Hidden State as Seq-to-Seq Terminal Bottleneck). Let M be a decoder-only Transformer language model parameterized by 9. with input/output sequence variables S.+1). We define the information bottleneck Con as:
Coan (Kun: Vo:,; On)
where Kom and Vo,n are represent keys and values computed from So: across all heads/layers, and On is the final hidden-state vector of the last token of Son prior to the model's (eleesssssa final logit projection. Then Co. Z, the terminal bottleneck in M...M.
Theorem 4.2 (Autoregressive Training Encourages high ISSoni 2) and I (2; (, Let So.N be a complete reasoning trace drawn from p( For some n. (where O S n < N), we define (80mn; 8n+1) to be an input/output pair corresponding to incomplete reasoning history and ground- truth next reasoning step. Let M be a decoder-only Transformer that maps input Som to KV cache/final hidden state co.n. (komms Vom: On)) via a determinstic mapping f where & C 0:
CD. (S())))
Let L(0 be the expected next step log-likelihood to maximise (computed via negative cross entropy) with respect to parameters 0:
L(0)= = Ep(sor))
Σ
log po (8n+1 s0:n)
(5)
n=0
Then we can show two bounds on L(0):
N
L(0) Σ ISSoni Co.)
Σ
H(S Sp..)
(6)
N-1
L(0) -
Σ n=0
I(Comi ((Com;SStt) H(S (S
(7)
Under Theorem 4.2, since IL(O) acts bound each term (where the entropy terms fixed under a particular dataset), maximisation of L(f) thus acts to encourage raising both mutual information components I(Co.n S,+1) and / (Soon.; Co.n). When we consider the tokenwise view, with Co. representing tokens at timesteps to I, we can see that Co. contains, as sub-states, each earlier cache Co: for i and so contains sufficient information to recover the full collection of next-token predictors [pe(S.+1 Co:s) realised along the sequence. Consequently, the final cache encodes a high-fidelity, step-by-step predictive trace of the right-shifted tokens (Si,..., St), rather than single compressed summary of the past.
Combined with Lemma 4.1 and Theorem 4.1 this implies that autoregressive training encourages internal sequence representations that are minimally compressive of their inputs as well as maximally predictive of future outputs.
4.5 COMPARISONS WITH RNNS AND CACHE COMPRESSION METHODS
Transformers excel at eetrieval-style tasks, due to their effectively unbounded memory, while RNNs and structured state-space models often outperform on problems requiring systematic rule application OOD generalisation (Deletang al. 2023 Liu et al. 2023 Wen al. 2025). Due the hard sequence-level bottleneck imposed by RNNs via their fixed-size hidden state, latent representations are forced to be reprocessed and compressed at every time step, whereas standard Transformers'
ever-grov weerggrowing cache removes this constraint completely. This leads to mutual information between given inputs (X) and these compressed latents (Z) that is reduced relatively to the one between latents Z and predicted outputs (Y), compared to Transformers. See Fig. B for a conceptual illustration of this comparison.
5

---

Published as a conference paper at ICLR 2026
A
High generalisation
Pattern Interpolation
Existing cache-operator methods have largely been explored from the perspective of compression via memory footprint reduction (see Section B). As illustrated conceptually in Fig. A, these algorithms tend to reduce not only the infor- mation retained about the input (T(X; Z)), but also indis- criminately reduce predictive information (Z; )), thereby moving towards region of lower generalised performance on benchmark tasks. Crucially, these methods lack repro- cessing step designed selectively compress I [X;; ; Z) while preserving or enhancing Y): Consequently, without a mechanism to substantially improve predictive efficiency
Bottlenecked Transformer
Transformers
((Z.)
Houristic KV Compression
Low generalisation
I(x: Z)
B
(1Z,)) as depicted in Fig. B, these techniques offer
Bottlenacked Transformer
T(X:2
limited improvements in generalisability.
4.6 ARCHITECTURAL SOLUTIONS
I(Z 1 Z)
RNNs
Our analysis shows that the KV cache (together with the final hidden state) forms the terminal bottleneck 2 and, in practice, carries extraneous detail from processed sequences. This motivates an inference-time mechanism that rewrites KV entries in place, producing new bottleneck 2' ( with an increase in predictive efficiency I(2'; ( 2*). By the data-processoo
Houristic Compression
Transformers
Information capacity
inequality, any such transformation results in (X; 2) 2 (X; Z"). By training T to minimise future prediction error, we preserve improve T(2';)) Conceptually, we interpret this as analogous to consolida- tion/reconsolidation:: selectively reprocessing working mem- ory to discard irrelevant information and maintain salient information. We focus on rewriting KVs rather than the final hidden state as the cache is the component principally re- sponsible for retaining the extraneous sequence information.
Figure 2:
Conceptual illustration.
(A) Bottlenecked Transformers bal- ance input compression T(X:
with predictive information T(Z:Y)
for high generalisation. (B) This
achieves superior predictive efficiency TZ Z) vs. capacity over other methods.
We apply no dimensionality reduction in rewritten KVs so as to avoid indiscriminate reduction predictive information that plagues compression methods.
BOTTLENECKED TRANSFORMERS
Overview. Here we introduce the Bottlenecked Transformer. We augment a pretrained decoder-only Transformer M with an external Cache Processor T.rroc, a neural module (smaller than the backbone) that periodically rewrites KV cache entries in-place during autoregressive generation. illustration of our architecture can be seen in Figure
Processor Invocation and Mechanism. During generation, immediately after some reasoning step 8,, completes (detected by emission of newline token), the Processor is invoked to rewrite cache entries. Decoding then resumes conditioned on the rewritten cache. We design our rewrite mechanism to be analogous to memory consolidaatio/reeoossiiaaaoooooo in the brain, and implement a selective mechanism for rewriting cache entries. When invoked, the Processor rewrites (i) cache entries corresponding to the most recent segment 8a, and (ii) the top A entries from the prior step history Su:u-1 by attention mass with the recent segment są- These components realise mechanisms that are respectively analogous to consolidation and reconsolidation: new memories within recent step window (RSW) of variable length R undergo a stabilisation process, and recalled memories rewritten in light of new information. Formally, we designate the set of recalled and recent KV entries as (k(s), P(s)). All other KV entries are left unchanged. A more detailed formulation of our selection mechanism can be found in Appendix B
Cache Processor Architecture. For backbone MLLM with L layers and H heads, the Cache Processor consists of small Transformer blocks proe,, I.
Jz=1 one aligned to each backbone layer.
Block l operates only the corresponding layer's selected KV entries (M (.)) We first convert the selected key-value pairs into "K V-tokens" by concatenating across all heads for that layer, and

---

Published as a conference paper at ICLR 2026
Base L.M
Method
Task TheoremQA LogiQA
GSM8K
MATH
SVAMP
Gaokao- MathQA
GSM-
Hard
SFT
11.76
15.36
3.70
7.13
Llama 3.2 IB
SFT w/ pause tokens
30.02
11.34
41.6
7.22
13.36
1.71
7.96
SFT w/ latent rollout
24.41
9.66
32.8
8.97
15.36
1.13
5.31
Bottlenecked Transformer (ours)
32,97
12.72
44.6
10.84
19,05
3,99
7.96
SFT
70.28
31.88
77.7
15.26
20.74
4.84
19.41
Llama 3.1
pause tokens
15.93
20.12
5.98
SPT w/ lotent rollout
4.02
2.80
7,80
5,89
0.00
0.00
0.61
Bottleecked Transformer (ours)
71.87
31.96
78.4
15.93
23.81
3.99
19,93
SFT
53.75
26.68
14.32
23.04
5.70
19.56
Qwen 0.6B
SFT w/ pause tokens
52.92
26.76
60.3
14.73
21,50
5.13
19.26
latent rollout
20.28
57.70
12.32
24.27
3.42
17.66
Bottlenecked Transformer [ours)
57.01
29.08
65.4
14.73
26.57
5.41
20.55
SFT
46.78
18.40
55.5
10.71
22.12
3.13
11.45
Llama
SFT w/ lokens
48.07
18.00
55.9
12.05
17.67
4.84
11.6
3.2 3B
SFT w/ latent rollout
42.46
15.28
52.0
11.65
12,90
2.56
10.61
Bottlenscked Trunsformer (ours)
51.33
20.90
59.4
14.73
20.12
3.99
12.28
Table Accuracy (%) on seven mathematical reasoning benchmarks across four backbones and three configurations: SFT, SFT+pause (16 pause tokens after the prefix), SFT+latent rollout (16 rollout tokens after the prefix), and Bottlenecked Transformer (ours; frozen SFT backbone augmented with Cache Processor). Scores pass @ under greedy decoding. Bold indicates the best result within each backbone.
project them via learnable matrix into the Processor's hidden state space:
(1) (1:0) (0)) E BR(++))x2dd
(8)
( x (((()) in
l!l ER2Hd.
The sequence (d) (consisting of recalled and recently cached memories) is then processed in parallel by a small Transformer block without causal masking, such that selected KV entries may be updated with globally available information. The block's 's output is projected back via learnable matrix to the KV dimensionality. Finally, we apply gated, in-place residual rewrite the selected KV entries.
Â (E) Tproc, (2)())
(10)
(A (8)
A = A (4) (
w
out E 12dx2Hd2
(11)
out
& k +
(1) - +g
(12)
Here (0) R is learnable, layer-wise scalar gate initialized small and denotes the logistic function. The gate mitigates early drift in model capabilities, i.e., large, destabilizing cache changes before the Processor has learned useful updates.
Training. Learning proceeds in two stages. In the first stage, the backbone M undergoes SFT on reasoning trajectories with the standard next-token cross-entropy objective. In the second stage, the backbone is frozen and only the processor parameters w are updated. Each training sequence So:N is split into individual reasoning steps (sg..... sN). For step sn, the backbone first processes the tokens in that step to append new KVs to the cache. The Processor is then invoked, selecting (R)))) (x(())
each layer and applying the in-place rewrite. Cross entropy loss for the next reasoning step 8 is then computed, conditioned on the rewritten cache, and backpropagated through the Processor. We truncate BPTT across step boundaries, such that the Processor is trained solely to rewrite the cache a way that improves prediction the next reasoning step.
Note that we do not implement an IB-style loss function; our goal is realise plausible mechanism for (re)consolidatioon in Transformer LLMs, supported by our theoretical findings that periodic memory rewrites may improve generalisation. The rewritten cache realises new sequence-level terminal bottleneck Z. Training the Processor minimise the cross entropy loss of the entire next reasoning step is equivalent to maximising (S,,1i Z). In other words, Ž is trained solely to improve prediction of future sequences, with no requirement for the rewritten entries to retain unnecessary information for reconstructing their input sequence. Moreover, whilst do not explicitly include compression term for minimisation of I(Se:n 2), removal of pressure maximise this quantity (as we showed to occur in vanilla Transformers) opens a pathway for implicit minimisation of this term via noise injection from SGD (as described in Section 4.2
7

---

Published as a conference paper at ICLR 2026
GSMBK
MATH
SVAMP
Theorem@A
LagiQA
GSM-Hard
20
8.
11
7.2
Soctlenecc
7.0
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 3: Epoch-matched comparison of SFT@ N and Bottleneck@1 N across seven tasks. The backbone is SFT-trained for 8 epochs with per-epoch checkpoints; Bottleneck@ N uses checkpoint N-1 plus one Processor epoch, and curves plot accuracy versus total epochs N. The red x marks the highest score for each task across both model variants and all N.
6 EXPERIMENTS
6.1 PERFORMANCE ON MATHEMATICAL REASONING TASKS
We evaluate the Bottlenecked Transformer set of mathematical reasoning tasks, choosing this domain as it offers an easily verifiable testbed for for observing improved reasoning generalisation. We compare four settings: (i) vanilla LLM fine-tuned for epoch math dataset (SFT), (ii) pause-toker baseline trained identically but includes 16 pause tokens appended after each question prefix pause, following prior convention). (iii) latent rollout model (inspired by Coconut (Hao et al. 2024)) which performs an n-step latent rollout directly in the token space by feeding the final hidden state back into the model without decoding to tokens, and (iv) our Bottlenecked Transformer, which freezes the one-epoch SFT model as backbone and trains the Cache Processor for one epoch using the procedure in Section 5 We use SFT a standard Transformer baseline and SFT with pause and SFT with latent rollout as token-mediated ALSC baselines. We omit other ALSCC variants (residual/cache operators) as these primarily target style/behavior control or memory footprint reduction rather than generalisation. All models are trained on 128k examples from OpenMatthnsttrutt2.. large synthetic mix GSMSK/MATH--st/ questions (Tohniwal et al. 2024). We evaluate on seven benchmarks: GSM8K, MATH, SVAMP, TheoremQA, LogiQA. Gaokao-Mai and GSM-Hard. Six are mathematical reasoning tasks; LogiQA is a logical reasoning task included to test transfer beyond mathematics. For all experiments, we fix Processor hidden size dp 512, intermediate size 2240, 16 heads per Processor block, selective reconsolidation uses = 32. Detailed hyperparameters can be found in Appendix [..I]
Results are given in Table Across backbones and tasks, the Bottlenecked Transformer improves over both baselines in almost all cases. Gains are strongest in-distribution math benchmarks (GSM8K, MATH, SVAMP, GSM-Hard): c.g., Llama-3.2 IB on SVAMP (46.6 points, 38.0-44.6)), Llama-3.2 3B on GSM8K (+4.6, 46.78 16.77-+511 -51333,, Qwen-3 0.6B on MATH (+2.4, 26.68-22.008) and Llama-3.1 8B on LogiQA (+3.1, 20.74-23.811, On the more out- -iitr--distribttion QA-style tasks, improvements generally persist (e.g., TheoremQA matches or exceeds baselines on all backbones), with one notable exception: LogiQA on Llama-3.2 3B where plain SFT is slightly higher (22. vs. 20.1 The main underperformance Gaokao-MathQA, where baselines often win (e.g.. Qwen- 0.6B and Llama-3.1 8B), consistent with a distribution/languuaee shift (Chinese) beyond the Cache Processor's 's training exposure. By contrast, the pause-token baseline shows variable and often lower performance than plain SFT when used only at fine-tuning (c.g.. consistent drops on Llama-3.1 8B and Qwen-3 0.6B), with only occasional wins such as TheoremQA at 8B or Gaokao-Math on some backbones. This mirrors findings from the original pause token paper, which showed reliable gains only when paired with continued pretraining before SFT. Additionally, the latent rollout baseline typically underperforms even the pause token baseline, which is consistent with results seen in the original Coconut paper, wherein the model performed slightly worse than a Vanilla model but saw improved efficiency (fewer tokens needed per answer). Performance degradation is especially bad for the Llama 3.1 8B model, which sees severe model destabilisation under continuous latent rollsuts.
6.2 EPOCH- TRAINING BUDGET ABLATION
To compare extra SFT with cache (re)consolidation under the same training budget. align models by the total number training epochs seen. We first train a backbone with SFT for 8 epochs, saving checkpoint after each epoch. For every checkpoint, we freeze the backbone and train
8

---

Published as a conference paper at ICLR 2026
k
GSM8K
MATH
SVAMP
TheoremQA
LogiQA
Gaokao-Math
GSM-Hard
16
32.07
13.12
43.20
10.01
19.05
3.70
7.58
32
32.97
12.72
44.60
10.84
19.05
3.90
7.96
64
33.43
12.94
44,20
10.04
19.20
2:28
7.96
128
33.05
13.20
43.30
9.64
19.05
3.13
7.73
256
33.05
13.34
43.30
9.50
17.81
2.28
7.58
Table 2: Top-k ablation of the bottleneck model across tasks (backbone: Llama 3.2 IB). Each column
is color-scaled from red (lowest) through yellow (middle) to green (highest), with softened tones.
GSM8K
MATH
SVAMP
TheoremQA
Le
Gaokao-Mall
GSM-Hard
16
31.69
12.92
42.70
9.77
18.89
4.27
7.58
32
32. 15
12.23
43.20
10.98
17.97
2.85
7.81
48
31.99
12.84
42.30
11.38
18.13
3.13
7.96i
07
12.15
43.40
12.05
17.97
3.13
7.58
10.58
7.96
Table 3: Ablation over recent-step window size R for the Bottlenecked Transformer (backbone: Llama 3.2 IB). The Cache Processor invoked once the end of the prompt and then every R tokens, so R controls the length of the local segment consolidated at each update. Performance is broadly stable across R. with slight gains for moderate windows.
Cache Processor for one additional epoch. We then compare SFT@ N (pure SFT for N epochs) against Bottleneck N built from checkpoint N plus one Processor epoch (both variants have seen N epochs). We use Llama 3.2 IB backbone, with same Cache Processor configuration as in Section 6.1 Across all seven tasks (Fig. 3. Bottleneck @N outperforms SFT@N on most N for GSM8K, GSM-Hard, SVAMP, and LogiQA, and the best score attained these tasks over any N is achieved by Bottleneck model. Two consistent exceptions are MATH and, to a lesser extent, TheoremQA, where SFT@N tends to be higher; additionally, Gaokao-MathQA mostly favors SFT@N at given N, although the single best score over all N is still achieved by Bottleneck model. plausible reason is that these settings require sustained access precise symbolic/theorem or language-specitng details, and step-boundar top-k reconsolidation (k=32) may down- weight earlier formula tokens or non-English cues that remain predictive.
6.3 RECONSOLIDATION BUDGET (k)) ABLATION
We ablate the Processor's 's ivvention-guided selection budget by varying the number of prior positions that are reconsolidated per layer at each Processor invocation, holding all other settings fixed (backhone: Ilama 3.2 BB identical training/evaluation protocol as Section 6. For each k: we train a separate Processor and report accuracy on the seven benchmarks. Table 6.2 summarizes results. Across all tasks except MATH, moderate budgets (k s 32 to k 64) are generally optimal. In contrast, MATH benefits from larger budgets, with best scores at k RJ 128 or 256. This likely reflects that MATH contains harder problems with longer solutions and stronger long-range dependencies, also offers a plausible explanation for the Bottleneck model's weaker MATH performance in the training budget experiment (Section 6.2 where the reconsolidation window was fixed 32.
6.4 RECENT STEP WINDOW (R) ABLATION
We also ablate the size of the recent-step window R by invoking the Cache Processor once at the end of the prompt and then at every fixed R tokens during generation, so that R directly controls how many of the most recent tokens consolidated at each call (Table 6.2). Across benchmarks, performance remains relatively stable over a broad range of Rt with mild gains for moderate to larger windows (e.g., A 64-96) and small drops when consolidation is restricted to very short windows. This suggests that the Processor benefits from access to reasonably sized local context, but does not require fine- mnninrdd per-token updates to yield gains. Together with the top-k reconsolidation ablation, these results indicate that our memory (re (re)ooosoiiittion mechanism is robust to the precise update schedule, so long as it can periodically reshape a medium-horizen segment of the working memory rather than attempting to track every token verbatim.
6.5 PROCESSOR REWRITE MAGNITUDES
We measure how strongly the Cache Processor modifies the KV cache by tracking cosine distances between entries before and after each rewrite. On GSM8K with the Llama 3.2 1B Bottlenecked

---

Published as a conference paper at ICLR 2026
Transformer, we compute mean cosine distance at every processor invocation for (i) the top-k recalled tokens. (ii) the recent-step window (RSW), and (iii) all rewritten entries. Results are shown in Figure 4 Across all three groups, value vectors undergo nontrivial updates, while key vectors remain almost unchanged, indicating that the Processor mainly edits the contents of memory rather than its addressing. Rewrite magnitudes are largest at early processing steps and then settle into stable plateau after roughly ten invocations, showing that the Processor does not collapse to the identity map but applies consistent moderate adjustments throughout generation. Layer-head heatmaps of value-vector distances show that edits are concentrated in the earliest layers, with only small changes in middle and later layers. This suggests that the Processor learns to reshape low-level representations that then propagate forward through the backbone, rather than rewriting deep layers directly.
Estimating [(;; Z) for a high-dimensional, variable-length KV cache is intractable in our setting, so we use rewrite mag- nitudes as a qualitative proxy for how the Processor reshapes the working memory state. Our observation of systematic shifts in value vectors away from their teacher-forced encod- ings indicates that the model is restructuring the content of selected memories. Because the Processor is trained solely through next-step prediction loss, these local, persistent ed- its suggest that past information is being reorganised while maintaining what is useful for future tokens. From the In- formation Bottleneck perspective, such prediction-preserving preserving, non-identity updates are naturally associated with reducing redundant input detail and making more efficient use of the bottleneck; here we treat rewrite magnitudes as an indirect, qualitative signal of this process rather than direct estimate of citemattoo-thhoretti quantities.
(Mererproiee
7 DISCUSSION AND FUTURE WORK
Figure 4: Cache Processor rewrite magnitudes on GSM8K. Left: per- invocation mean distances for top-k, recent-step window, and all rewritten tokens. Right: layer-head heatmaps of mean cosine distance between pre- and post-Processe value vectors.
Our work has explored the gap in cache-operator ALSC systems that pertains to our interpretation of memory (re)consolidation, giving both theoretical justification as to why this beneficial in decoder-only Transformer LLMs and empirical verification via an architecture that improves mathe- matical reasoning performance. Here we discuss limitations of our method.
Training the Processor solely through next-step cross- cross-entropy can produce variancr,, poorly localized credit assignment, providing weak supervision for cache rewrites, making it challenging for the model to escape its strong local optimum. Training a model from scratch may alleviate this issue. Additionally, we do not include an explicit infommation-thooeee reduction of [(X; Z): any information compression can only arise from the data processing inequality or SGD noise. Whilst direct MI estimation in high-dimension cache is challenging, promising route is controlled noise injection into selected KV entries followed by iterative denoising/reefnement, which constitutes a mapping that reduces I(X; Z) while preserving predictive structure [(Z; Y) (by the data processin inequality and denoising as regularization). Such a mechanism would essentially constitute iterative latent reasoning in the model's memory space; past works exploring this idea in non-L_LM-based frameworks have yielded promising results (Du et al., 2024).
objective for compression via
Regarding our interpreation//mpleeenniii literature indicates that these are related but partially distinct processes: consolidation unfolds over hours days with systems-level reorganization and sleep-driven replay, whereas reconsolidation is a brief, attrieval-induced window in which a reactivated trace becomes labile and then re-stabilises (Dudai et al.) 2015) Stickgolld Walker 2007). In light of this, our single, online Processor collapses two modes that in biology differ in triggers and timescales; a closer analogue would pair an offline, replay-style consolidator with an online, retrieval-contingent reconsolidator. Additionally, reconsolidation appears to depend on prediction error at retrieval, i.e., mismatch is often required to open the plastic window, suggesting that surprise/PE gating (rather than a fixed newline trigger) would be more suitable for determining when reconsolidation should occur (Exton-MeGuinee McGuinness et al.. 2013) Fernández et al. 2016). More closely aligning future (re conooossoiit architectures with these biological mechanisms may yield substantial gains over our current models.
of consolidation and reconsolidation, neuroscientific
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics. Our study uses publicly available, licensed datasets and synthetic math corpora: no human subjects or private data were involved.
REPRODUCIBILITY STATEMENT
We detail all training and evaluation settings (datasets, preprocessing, hyperparameters, model sizes, and decoding) in Appendix D and provide proofs for theoretical claims in Appendix A
REFERENCES
Ben Alderson and Charles Femyhough. Inner speech: Development, cognitive functions, phenomenology, and neurobiology. Psychological Bulletin, 141(5):931-965, September 2015. doi: 10. 10.1037/bu0000001. URL http:::: wwwp///wwww///////////////ww ncbi nlm. nih. gww/pmc/aaticllls.....
PMC45389547
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https
//arxiv. httpvvorg/abb..
///.xii..rg//a///2000000 2005. 14165
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. Recurrent memory transformer, 2022. URL a22/
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression, 2024.
hwg. wwwp:///////////////a/////eeeee/ 220.. 08013
Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy, Improving steering vectors by targeting sparse autoencoder features, 2024. URL ////////////////a...oo//////////
02193
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma. Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem- driven question answering dataset, 2023. URL https
hbblg//bbs/200 12524
MinGyu Choi and Changhee Lee. Conditional information bottleneck approach for time series imputation. In The Twelfth International Conference Learning Representations, 2024. URL
atttps://cpenxeviee net/ forum? 1d=immmmdddmmiiii-mmmm iDdOJ]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https::/arxiis org/ abs /ww./22/2222 2204 02311
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110. 14168, 2021. URL
wrw. https://arxx http:///wxxiv.oo////////////////
14168

---

Published as a conference paper at ICLR 2026
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdi- nov. Transformer-XLL Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum. and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653//1//19-1122 URL https? wwclantholooy.orr//330
1285/
DeepSeek-A,, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai,, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yunfan Xiong. Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. /btp//bb////2222/2 12948
URL https:
Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, 2023. URL
Yilun Du, Jiayuan Mao, and Joshua B. Tenenbaum. Learning iterative reasoning through energy diffusion, 2024. https:::: arxiv. www.:iiw..g/aaaaa..eopeee wtgg.abs///....11///// 11179
Yadin Dudai, Avi Karni, and Jan Born. The consolidation and transformation of memory. Neuron, 88 (1):20-32, October 2015. ISSN 0896-6273, doi: 10.1016/j neuron.2015.00
Mare T. Axton-McGuinness. Jonathan L. C. Lee, and Amy C. Reichelt. Updating memories-th the role of prediction errors in memory reconsolidation. Behavioural Brain Research, 278:375-384, February 2015. ISSN 0166-4328. doi: 10. (0.10166).b::2014.100 10.0011 Epub 2014 Oct 22.
Evelina Fedorenko and Rosemary Varley. Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369 (1): April 2016. doi: 10. 1 1 13046. URL htthhttss:/ www ncbi. n m.m. nih Epub 2016 Apr 20.
Ninghui Feng, Songning Lai, Jiayu Yang, Fobao Zhou, Zhenxiao Yin, and Hang Zhao. Timesieve: Extracting temporal dynamics through information bottlenecks, 2024. URL hhtttt:: //axxivv brg//bb/ /406 05036
12

---

Published as a conference paper at ICLR 2026
Rodrigo S. Fernández, Mariano M. Boccia, and Maria E. Pedreira. The fate of memory: Recon- solidation and the case of prediction error. Neuroscience & Biobehavioral Reviews, 68:423-44 441, September 2016. ISSN 0149-7634. doi: 1e11016f..uubborre.00666...00000
Epub 2016 Jun 7.
Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lam- pouras, Haitham Bou-Ammar, and Jun Wang. Human- episodic memory for infinite context llms, 2024. URL htt.:::ttt:/aaaaa
org/ abs//b/////// 2407. .0..0.0.099050
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:221.10043,, 2022.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL
https::/aaaa www./r/gsss.....oo atcp//bs/2233 2310 02226
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2024. URL https
Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models, 2025. URL https: wwwp:xwww..gggaa.ooooooo abs /2502. 11356
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Almut Hupbach, Roberto Gomez, Oliver Hardt, and Lynn Nadel. Reconsolidation of episodic memories: subtle reminder triggers integration of new information. Learning & &Memory, 14 (1-2):47-53, January 2007. doi: 10.1101/m..65700 Published 2007 Jan 3.
Kenji Kawaguchi, Ziwei Ji, and Leslie Pack Kaelbling. How does information bottleneck help deep learning? arXiv preprint wrXiv:2305.18885 2023.
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language model, 2024. URL
Jonathan L. C. Lee. Reconsolidation: maintaining memory relevance. Trends in Neurosciences, 32 (8):413-420, August 2009. doi: tins. 00009.005.022... Epub 2009 Jul 27.
Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shorteuts to automata. In The Eleventh International Conference Learning Representations, 2023. URL
ewreerrvview.... net. wttt/fooruue forum? FeD.emm/e2eddeeeFFFFeeee
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08/ 2020.
Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, and Arthur Szlam. Deliberation in latent space via differentiable cache augmentation, 2024a. /tt//tttp:://ccx..o///////
Zichuan Liu, Tianchun Wang, Jimeng Shi, Xu Zheng, Zhuomin Chen, Lei Song. Wenqian Dong, Jayantha Obeysekera, Farhad Shirani, and Dongsheng Luo. Timex++: Learning time-series explanations with information bottleneck, 2024b. http:tpps//xxvv.o/////aaaaa
09308
Martin M. Monti, Lawrence M. Parsons, and Daniel N. Osherson. Thought beyond language: neural dissociation of algebra and natural language. Psychological Science, 23(8):914 922, August 2012. doi: 10. 00.1777095677762443427. URL 227608837 Epub 2012 Jul 3.
ncbi. nlm. nih. gov/
13

---

Published as a conference paper at ICLR 2026
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Tumer. Steering laaaa via contrastive activation addition, 2024. URLattas:/arxiv.or
abs 06681
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? CoRR, abs/2103.07191, 2021. URL https://arxiv> org
wwtp/////////////////////////aa/ abs /2103
91
Jacob Pfau, William Merrill, and Samuel R. Bowman. Let's think dot by dot: Hidden computation in transformer language models, 2024.
15758
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019. URL httpp::/aaxxxi..r///////
1911.05507
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li. Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL /tRLittts///arxiioorrgg /abssaaawworrogsss//
2402. 03300
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation, 2025. URL https:://arxiv
3bs/2502.21074
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, abs/1703.00810 2017. URL http:://txxxxy wttpiiiwwwrr//b////00000 00810
Samuel Soo, Chen Guang, Wesley Teng, Chandrasekaran Balaganesh, Tan Guoxian, and Yan Ming. Interpretable steering of large language models with feature guided activation additions, 2025.
URL httts: org/ abs. 09929
Robert Stickgold and Matthew Walker. Sleep-dependent memory consolidation and reconsolidation. Sleep Medicine, 8(4):331-343, June 2007. ISSN 1389-9457. doi: 10. ....0666sseeep22000......1. Epub 2007 Apr 30.
DiJia Su. Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning, 2025. URL
hitts:://ariis / aas.///////////o//aa/a 250222502.0 03275
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference Communication, Control, and Computing. 2000.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislay Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstrutt-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL
https://arxxx: www./oggg///.oooooo 2230 1003008.11225
Denis Ullmann, Olga Taran, and Slava Voloshynovskiy. Multivariate time series information bot- tleneck. Entropy, 25(5), 2023. ISSN 1099-4300.
doi: 10.3390/e25050831. URL https
www. www.mmdp..co wwc.mm1090 00110099990000000 4300/25/5/
Zheng Wang. Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for lims on long-com context tasks, 2024. URL httpa::/axxiv.ozg///
2407. 08454
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL U htt.:httts///xxiiiiee httppp:://aaii....m 2201 11903
14

---

Published as a conference paper at ICLR 2026
Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. RNNs not transformers (yet): The key bottleneck on in-context retrieval. In The Thirteenth International Conference on Learning Representations, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks, 2024. URL arwai.iiiv....... org/ otg/abs/////ooooooo..t...//...tr abs abwaaabs////// 17453
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. CaM: Cache merging for memory-efficient LLMSs inference. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of PMLR, 21-27 Jul 2024. URL mlr. www.srrsspseee.ee v235/2235222/// zhang html
Proceedings of Machine Learning Research, pp. /t0/ess//33/////
https:
Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song,
Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference large language models, 2023. URL https?
14048
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299-2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 122553/////224... 1000:/.iin///. andings-naacl. 149.. URL httess//aeelaacboooo.o.o///22222 fine:24.iiinngggg htt.:/22002....ndnngn--maaaaeae. 149/ 1497
APPENDIX
A PROOFS
A.1 PROOF OF LEMMA 4.1
Proof. Under the non-trivial case where Z 2, by Definition
4.2
we have that:
x -
Under the Data Processing Inequality, we thus have:
1(X:2) < 1(x:)).
A.2 PROOF OF THEOREM
Proof. Assume for contradiction that there exists some bottleneck Z' strictly deeper than Cosn (i.c., Con < Zz).. By definition of the partial order, we could discard Co.. when predicting S,41,
ps Co... z') pe (S.+1 IZ').
However, under decoder-only Transformer, under arbitrary input/output sequence variables (Son,, Sn41), decoding of 5,41 must be conditioned on Co. as Con is a projection of the in- sequence (So.. and is trained to contain all information necessary predict S,+1- Furthermore, under this architecture, no further processing occurs on Cuen once has been constructed. Hence, Con in its entirety cannot be discarded and replaced by some variable Z', contradicting Z' X Co.n- Therefore, Corn is the maximal element the set of bottlenecks, and thus it is the terminal bottle- neck.
15

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 4.2
Proof. Since Cu.n. = (8on)) ) is a deterministic mapping under a vanilla decoder-only Transformer, we can write our objective function as:
T())) Ep(so:n (B:N)
Σ
log po (sn+1 s0:n)
(13)
Ep(so: No):) co:
Σ
log po (sn+1
co:n)
(14)
N-1
Σ E [.
log po po com)
(15)
n=
For two random variables A, B with samples (a, b) from joint distribution p(A, B) and approximate distribution (A, B), we see that:
pp) [log q[a|b]]] = Bp(a,b) [log pogg(())) + log q(a(b)
(16)
p(alb)
B) Ep(s) [Dxc [I
(17)
[(A;B) - (A) - [xx [(cal)
(18)
S B) - (A)
(19)
Which implies that:
N-1
L(O) S Σ
M(Coin Sm+1) HSS++1)
(20)
We can also write L(0) as:
L(0)= Pl No.N Ma.N)
Σ
log po (Sn-1 80:ny hon+11, Monnt)))))))))
(21)
N-1
[log po (Su+1 80:n 1oo+1 M0x++1)
Bep(sum 8aon pom+t))
(22)
H(S++1|Som Komtt,, Vo:++1)
(23)
n=0
N-1
I(S Kom+1, (on41|So.n) H(n (Som.)
(24)
n=0
((S.41:Contal)) (S SS:.)
(25)
N-1
T(Soa+1; Con+1) (S
(26)
n=0
N
N-1
Σ
I(Sani Co:)) Σ
H(Sa+1|Som))
(27)
m0
Thus, combining Equations 20 and 27
we have:
N-1
2L(0) <
(Sani Co.n) Σ
[I(ConiSh43) -H(S -
N
N-1
L(0) <
(SS... Co..) + Σ
[[Coom Sa41) - (Sx+1) - 11 [|
n=0
16

---

Published as a conference paper at ICLR 2026
GGMBC
16TH
SUAMP
TheonmmOA
14.0
12.0
345
11.
415
11.
64.5
11.
11.0
333
12.4
41.
10.3
43.
100
023
523
23
2.25
134M
Opctha
Epocts
Epodts
Figure 5: Size ablation of the Cache Processor on frozen Llama 3.2 1B backbone, showing per- epoch performance of each variant each task.
B CONSOLDATIOVRECONNLLDDDDTT MECHANISM
Let Jn Id,(s,) denote the token indices of just-completed step and Pcn Idx(s<n) the indices of all prior tokens. For each layer & E (1....,,) we construct an index set
_ Jn U Topk (Pen)
where TopK (P<n) contains the prior positions with the largest attention mass with the current step. Writing (0,h) E TRtxt for the backbone's attention matrix at layer E, head h, over the prefix 8gn, the mass assigned by the tokens of 8n to prior index i E Pen is
( a
Σ jJJn
(2.)
(e) Topk (Pen) arg topk af iEPc
J
where H denotes the number of attention heads. Only the entries at I are modified by the Processor; all other cache positions remain unchanged. This realises (i) consolidation of recently written context, and (ii) reconsolidation of the few most recalled KVs during the recently written step.
C CACHE PROCESSOR SIZE ABLATION
In this experiment, we ablate the Cache Processor size and training duration using a Llama 3.2 1B backhone, by varying the Processor feedforward intermediate width, holding depth, number of heads, selection policy (RSW + top-k), and all training hyperparameters fixed (same as in Section 6. see details Section D The backbone trained for one epoch of SFT on 2.hhstuthns We train four Processors on this backbone (of 59M, 84M, 109M, 134M parameters). Each Processor is trained for four epochs, and we evaluate the end of every epoch.
Results are shown in Figure 5 Increasing training duration generally improves performance of models on all tasks, with this effect being most apparent on the MATH, TheoremQA and GSr-Hard tasks, which typically contain the hardest problems. For other tasks, performance beyond one epoch of training is more variable across all models, often plateauing early. This suggests that additional training may not be beneficial in cases where downstream tasks simpler. Additionally, there is no clear winner in model size across all tasks. This suggests that training so as to make full use of the model capacity is challenging. We hypothesise that this is due to poor credit assignment from next-step supervision, making it challenging to escape the strong local optimum that the backbone resides in, discussed in Section
17

---

Published as a conference paper at ICLR 2026
D REPRODUCIBILITY
D. MAIN RESULTS
Here list details for reproducibility of our main experimental run in Section 6.1
Training details We train on the first 128k examples from the 1M variant of the OpenMathInstruct- 2024). For all runs, we use a batch size of 128 with learning rate of 4.
2 dataset (Toshniwal We use a constant LR with no warmup for all runs except for experiments with Llama 3.1 8B, where we use a warmup ratio of 0.05 and cosine LR scheduling. We truncate training sequences max length of 512 tokens.
Evaluation details. For all evaluation runs, we employ greedy decoding, truncating responses to maximum length of 2048. We evaluate seven tasks:
GSM8K: Grade-school math word problems requiring multi-step arithmetic reasoning and short numeric answers. (Cobbe al. 2021)
MATH: Competition-style mathematics problems (e.g.. algebra, geometry, number theory, counting) with formal, multi-step solutions. [Hendrycks et al. (202I)
SVAMP:
Simple arithmetic word problems rewritten with semantic variations to test robust- ness to superficial cues. (Patel et al. 2021)
TheoremQA: Question answering that requires recalling, understanding, applying mathe- matical theorems and their conditions. (Chen et al. 2023)
LogiQA: Multiple-choice logical reasoning and argument analysis questions modeled after civil service exam items. (Liu et al. 2020)
Gaokao-MathQA: Math question answering drawn from China's Gaokao (college entrance) exams, often involving symbolic manipulation and problem solving. (Zhong et al. 2024
GSM-Hard: A harder subset of grade-school math problems with much larger numbers. designed to stress multi-step reasoning beyond standard GSM8K difficulty. (Gao et al. 2022)
Pause token baseline configuration. During training, we instantiate 16 pause tokens in the embedding table. During training/generation. we append these 16 tokens to the end of the question prefix. We perform SFT via standard cross entropy loss on response completions.
Bottlenecked Transformer Configuration. For all backbones, fix the Processor design across backbones (one block per backbone layer; hidden size dg -512 MLP intermediate size 2240; 16 heads per block, fixing reconsolidation budget & = 32. Table D.I shows parameter counts for the Processor for each backbone, these vary because the projection layers to/from K V-space and number of Processor blocks depend on the backbone's hidden dimension, number of attention heads, and number of layers.
Backbone
Processor Params
Llama 3.2 1B
88.69M
Llama 3.2 3B
184.63M
Llama 3.1 8B
211.01M
Qwen 0.6B
184.63M
Table 4: parameter counts per backbone for mathematical reasoning performance
experiment.
D.2 ABLATIONS
For these experiments, where applicable, we use the same training/architcctural configuration for SFT/Bottleneck models as is detailed in Section D.I using Llama 3.2 1B Backbone.
18

---

Published as a conference paper at ICLR 2026
D.3 COMPUTATIONAL OVERHEAD
For a Bottlenecked Transformer consisting of a Ilama 3.2 1B backhone with an 89M parameter Cache Processor, setting & = 32, memory footprint during the Cache Processor training stage is approximately 6x that of performing full parameter SFT, owing to the chunked training process which prevents parallelism for entire training examples, induces extra padding tokens, as well as the high computational cost of processing large number of entries in high dimensional KV cache. As such, wall clock time for one-epoch training run for the Cache Processor approximately 20x longer than performing SFT on the backbone. Note that this is partially engineering issue: our method is currently incompatible with efficient attention methods such as Flash Attention.
During evaluation, memory footprint of the Bottlenecked Transformer with above configuration is approximately 25% higher than a vanilla Transformer, with a 45% increase in wall clock time, for an eval batch size of 16 in both cases. This relative reduction compared to training is due to the fact that during generation, the Cache Processor is invoked infrequently (once every reasoning step).
STATEMENT ON THE USE OF LLMS
During manuscript preparation, we used large language models for editing, phrasing suggestions, and to assist in literature search. No analyses, results, proofs, figures were produced by L.LM;; all technical content, experiments, and conclusions are our own.
19


Reviews:
[{"review_id": "pPMKAIJuJ6", "reviewer": "Reviewer_8HNH", "summary": "This paper proposes Bottlenecked Transformers, which add a lightweight non-causal Cache Processor that periodically rewrites the KV cache at newline-delimited “reasoning step” boundaries to consolidate recent tokens and reconsolidate top‑k recalled past tokens for better sequence-level reasoning. The approach is motivated via an Information Bottleneck perspective arguing vanilla decoder-only Transformers over-retain history in the KV cache and thus benefit from in‑place memory edits that preserve predictive information without compressing dimensionality. Concretely, the processor performs periodic, in-place global rewrites of recent KV segments and selected prior entries conditioned on the latest context, acting as sequence-level auxiliary latent-space computation adjacent to standard decoding. Across seven mathematical reasoning benchmarks and four backbone LLMs, the method yields consistent gains over vanilla and pause-token baselines, with improvements up to +6.6 percentage points on selected tasks and models.", "questions": "1. The IB narrative relies on reducing $I(X;Z)$ while keeping or improving $I(Z;Y)$; can the authors either add a tractable variational IB objective (e.g., CPC/InfoNCE-style bounds) or present calibrated MI proxies that show the intended movement along the efficiency frontier, rather than only accuracy deltas ?\n2. I suggest the authors include strong latent and cache-operator baselines under matched compute, add non‑math and multilingual tasks, and report multi‑seed CIs and contamination checks to substantiate generalization claims\n3. Can the authors run sensitivity analyses for $k$, $R$, and invocation frequency, and report compute-normalized outcomes to justify overheads. \n4. I am curious, if training from scratch with a joint backbone‑processor objective to test whether the “frozen backbone” constraint is the main limiter, and whether end‑to‑end training reaches better predictive efficiency under equal compute?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear problem framing of ALSC at the sequence level and principled positioning of cache rewriting as consolidation/reconsolidation rather than compression, with an architecture that is simple to integrate and keeps KV dimensionality unchanged.​\n\n2. Information-theoretic perspective highlights a plausible failure mode of vanilla decoder-only training (KV states retain unnecessary sequence detail), and motivates a targeted non-causal rewrite to increase predictive efficiency without shrinking memory footprint.​\n\n3. Consistent improvements over SFT and “SFT + pause tokens” across multiple backbones and benchmarks, including sizeable gains on in-distribution math tasks (e.g., +6.6 on SVAMP with Llama‑3.2‑1B; +4.6 on GSM8K with Llama‑3.2‑3B) and thoughtful ablations on epoch-matched budgets and the reconsolidation window k.​\n\n4. Implementation details are thorough (selection policy via attention mass, layer-aligned processor blocks, gated residual rewrites), with reproducibility notes and multiple architectural/training ablations provided.​", "weaknesses": "1. Theorem 4.2 provides a lower bound linking token cross‑entropy to a sum of mutual information terms, but the text then treats autoregressive training as “maximizing both” $$I(S_{0:n};\\hat Z)$$ and $$I(\\hat Z;S_{n+1})$$, which does not follow from a loose bound and is not shown to hold per‑term, weakening the justification for the proposed remedy.\n2. The “terminal bottleneck” claim for the KV cache plus last hidden state is used to argue that the cache retains reconstructive detail that impedes generalization, yet no empirical evidence is provided that the cache can reconstruct inputs or that rewrites reduce $$I(X;Z)$$ while preserving $$I(Z;Y)$$ as required by the IB narrative.\n3. No IB‑style loss or MI estimation is used; reliance on SGD noise and the data‑processing inequality is asserted as a path to compression, but there is no quantification of predictive efficiency $$I(Z;Y)/I(X;Z)$$ before vs. after cache rewrites to support the core thesis.\n4.  Comparisons omit strong ALSC and cache‑operator baselines beyond pause tokens, such as latent rollouts, differentiable cache augmentation, or modern cache merging/compression tuned for reasoning, leaving performance advantages over alternative designs unclear under matched budgets.\n5. The authors notes reconsolidation may require prediction‑error gating, but does not test such triggers or analyze boundary sensitivity (e.g., recent‑step window length R) in the main results. \n6.  Results appear single‑seed with no confidence intervals or significance testing, and there is no contamination analysis despite training on synthetic math mixes that may overlap benchmark styles\n7. Training the processor incurs ~6× memory and ~20× wall‑clock over SFT, while inference adds ~25% memory and ~45% latency despite infrequent invocation, which is a significant overhead for modest accuracy gains on small/medium backbones. \n8. There are no MI proxies/estimates, ablations that track retrieval fidelity vs. abstraction, or causal analyses showing which information is discarded/retained by the processor.\n9. No layer/head‑level probes, attention‑pattern shifts, or KV‑state visualizations that explain why and when reconsolidation helps or hurts (e.g., MATH vs. Gaokao outcomes).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEygMwxYDi", "reviewer": "Reviewer_cuPq", "summary": "The paper proposes Bottlenecked Transformers, augmenting a decoder-only LLM with a small Cache Processor that periodically rewrites selected KV-cache entries at newline-delimited “reasoning step” boundaries to mimic consolidation/reconsolidation and improve generalized reasoning. The authors motivate this with an Information Bottleneck (IB) analysis arguing that vanilla Transformers’ KV caches act as terminal, minimally compressive bottlenecks and thus preserve extraneous input information that can hinder generalization.", "questions": "* **Ablations on triggers**: Newline ≠ reasoning step in many datasets. How do results change with alternative triggers (e.g., surprise/prediction-error gates, punctuation, tokens-per-step), or with learnable triggering?\n\n* **What is actually rewritten?** Can you provide layer-wise and head-wise analyses of rewrite magnitudes (gate values, ΔK/ΔV norms), and their correlation with downstream accuracy, to demonstrate nontrivial, stable edits?\n\n* **Sensitivity to selection policy**: Beyond top-k by attention mass, evaluate (a) per-layer k, (b) diversity-regularized selection, (c) recency vs. salience trade-offs, and (d) dynamic k conditioned on step difficulty. Table 2 hints that MATH prefers larger k. Can adaptive k close the MATH gap?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Clear theoretical motivation**: The IB framing (Theorems 4.1–4.2) formalizes the KV cache + final hidden state as a terminal bottleneck and links autoregressive training to maximizing both $I(X;Z)$ and $I(Z;Y)$, motivating selective rewrites.\n\n* **Simple, modular mechanism**: A lightweight, layer-aligned processor rewrites (i) recent-step KVs and (ii) top-k recalled past entries by attention mass; gating stabilizes updates. The schedule is practical (trigger on newline).\n\n* **Consistent empirical gains**. Across backbones/tasks, the model usually beats SFT and pause-token baselines; Table 1 and Fig 3 show broad improvements and useful diagnostics.", "weaknesses": "* **Scope & novelty relative to cache operators**: While the reconsolidation framing is fresh, the core operation (transform selected cache entries) is close to existing cache-edit/compression lines; novelty hinges on scheduling/selection rather than a fundamentally new cache objective.\n\n* **Supervision signal may be weak**: The processor is trained only via next-step cross-entropy with truncated BPTT, which the authors note causes credit-assignment issues; no explicit IB/MI control is used.\n\n* **Mixed results on OOD and symbol-heavy tasks**: MATH/LogiQA/Gaokao occasionally favor SFT; performance is sensitive to k and language/domain shift, suggesting limited generality without careful tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEJnQiVka8", "reviewer": "Reviewer_kt4M", "summary": "This paper introduces the Bottlenecked Transformer, an architecture designed to improve the reasoning capabilities of decoder-only LLMs. The authors first provide a theoretical motivation using Information Bottleneck (IB) theory, arguing that standard autoregressive training incentivizes the model's KV cache to retain unnecessary information from the input sequence, which can hinder generalization. To address this, they propose augmenting a frozen backbone LLM with a lightweight \"Cache Processor.\" This processor is a separate, non-causal Transformer that is invoked periodically (at newline characters) to perform in-place rewrites of the backbone's KV cache. The rewrite mechanism is inspired by the neuroscience concepts of memory consolidation (acting on recent KV entries) and reconsolidation (acting on a small, attention-selected set of prior entries). The processor is trained to rewrite the cache in a way that improves the prediction of the next reasoning step. The authors demonstrate that this approach yields consistent performance gains on seven mathematical reasoning benchmarks across four different backbone models.", "questions": "1. Regarding the computational overhead: Could you elaborate on the specific engineering challenges that prevent compatibility with methods like Flash Attention? Is there a clear path forward to mitigating the significant training and inference latency, or is it a fundamental cost of the non-causal, in-place rewrite operation?\n\n2. The supervision signal for the Processor seems to be a key challenge. Have you considered or experimented with auxiliary loss functions to provide a more direct learning signal? For example, could a variational IB objective be approximated, or could a contrastive loss be used to encourage the rewritten cache state to be more predictive than the original state?\n\n3. The choice to trigger on newline tokens is a simple heuristic. How sensitive are the results to this specific choice? For instance, what would happen if the trigger was changed to a period or another punctuation mark, and have you considered more adaptive triggers based on model uncertainty or prediction error, as suggested in your discussion?\n\n4. The paper makes a compelling case for memory consolidation in the context of mathematical reasoning. How do you see this mechanism applying to domains where reasoning is less about logical abstraction and more about maintaining narrative consistency or tracking complex entity relationships, such as in long-form story generation? Could the \"bottlenecking\" process be detrimental in such cases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Strong Theoretical Foundation:** The primary strength of this work is its grounding in Information Bottleneck (IB) theory. The authors provide a formal proof that vanilla Transformers are constrained in their ability to form optimal sequence representations for generalization. \n\n**Novel and Bio-Inspired Architecture:** The concept of a separate Cache Processor that performs periodic, in-place KV cache rewrites is highly novel. The design, explicitly inspired by the neural mechanisms of memory consolidation and reconsolidation, is both elegant and intuitive.\n\n**Comprehensive Empirical Validation:** The authors validate their method across seven different mathematical and logical reasoning benchmarks and, crucially, with four different open-source backbone models of varying sizes. The consistent performance improvements over both vanilla SFT and a pause-token baseline demonstrate the robustness and general applicability of the approach.\n\n**Insightful Ablation Studies:** The paper includes well-designed ablation studies. The epoch-matched comparison effectively shows that an epoch of Processor training is a better use of compute than an additional epoch of SFT. The ablation on the reconsolidation budget (k) provides valuable insights into how task characteristics interact with the model's memory mechanism", "weaknesses": "**Significant Computational Overhead:** The practical utility of the method is currently hampered by its high computational cost. The authors report that training the Cache Processor is ~20x slower than standard SFT, and inference is ~45% slower on a 1B parameter model. While acknowledged as a potential engineering issue, this overhead is a major barrier to adoption.\n\n**Weak and Indirect Supervision:** The Cache Processor is trained only via the cross-entropy loss of the next reasoning step. As the authors note, this provides a high-variance and poorly localized signal for learning optimal cache rewrites, which may explain why performance doesn't always improve with Processor size or training duration.\n\n**Simplistic Invocation Trigger:** The Processor is invoked every time a newline token is generated. While simple to implement, this heuristic is not adaptive. It may trigger too often in contexts where newlines are frequent but semantically unimportant, or not often enough in dense reasoning chains that lack newlines.\n\n**Narrow Domain of Evaluation:** The experiments are exclusively focused on mathematical reasoning. While this is an excellent testbed for logical generalization, it's unclear how the core principle—compressing the past to better predict the future—would transfer to other domains like creative writing or summarization, where preserving rich, verbatim details from the context is often essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)