Training Large Reasoning Models Efficiently via Progressive Thought Encoding
Authors: Zeliang Zhang
Xiaodong Liu
Hao Cheng
Hao Sun
Chenliang Xu
Jianfeng Gao
Venue: ICLR 2026 Poster (2026)
Keywords: efficient reasoning LLM
KV cache
test-time learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into compact representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing training-time memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, across six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3\% improvement over LoRA and +29.9\% over the baseline on average, with up to +23.4 absolute gains on AIME2024/2025 under tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.

Published as a conference paper at ICLR 2026
TRAINING LARGE REASONING MODELS EFFICIENTLY VIA PROGRESSIVE THOUGHT ENCODING
Zeliang Zhang Xiaodong Liuft, Hao Cheng", Hao Sun?, Chenliang Xu' and Jianfeng Gao University of Rochester, *Microsoft Research
ABSTRACT
Large reasoning models (LRMs) excel complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, parameter-efficien fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding inter- mediate reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2. 3---BB-Insstuu Qwen2. ITBBInstruct. and SeepSeek-R1-Disee
8B, six widely used challenging mathe- matical benchmarks show consistent gains: our method achieves +19.3% improve- ment over LoRA-basee fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought En- coding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.
INTRODUCTION
Large reasoning models (LRMs) Plaat et al. 2024 Li et al. 2025b: Huang and Chang 2022 are
emerging as new paradigm that extends large language models (LLMs) with enhanced capacity for multi-step reasoning (Fu et al. 2023). symbolic manipulation (Dave et al. 2024), and problem solving real-world scenarios (Xu et al. 2024). Unlike conventional LLMS that rely primarily on scale and corpus size for improved performance, T.nee,,,RR... LRMs explicitly emphasize reasoning-oriente
training signals and architectural design, making them particularly well suited for domains such as mathematics (Shao) ct al. 2024), science (Schmidgall et al. 2025), and programming (Wang et al., 2024). As these models continue to achieve impressive results on increasingly complex benchmarks (Phan et al.,
2025 Wang et al. 2023). the focus of research has gradually shifted from pursuing capabilities
to improving efficiency in training and deployment (Wu et al. 2025 Feng et al. 2025).
Reinforcement learning (RI.) KKalbling al. 1996) has become the standard approach for align-
ing and improving large reasoning models (LRMs)) during post-training, with methods such as
PPO (Schulman et al.) (017). GRPO (Guo et al. 2025), and related algorithms (Zheng et al. 2025a Yu et al. 2025) Li et al. 2025a) providing aine-grained control over reasoning behavior. However, RL suffers from a fundamental efficiency bottleneck: outcome-basee rewards are sparse and only available after completing long sequences of actions (Yang et al., (025). during which autoregressive decoding dominates memory and compute resources. The length of these trajectories, or ohain-of- thought (CoT) reasoning, scales with task complexity, yielding longer rollouts for more challenging problems. Such extended CoT' sequences significantly increase post-training and inference costs.
A natural strategy to address this challenge is to bound memory through sliding-window caches (AP izadeh et al. 2024) or dynamic pruning past tokens (Fu et al. 2024). However, these approaches
Work done during the internship Microsoft Research. |Correspondence to: zeliang zhang @ rochester.ddu, xioodl @ microsofttcom.

---

Published as a conference paper at ICLR 2026
often degrade reasoning quality, as discarding intermediate thoughts weakens the model's ability to integrate long-range context (Cai et al. (024). This degradation not only impacts reasoning accuracy at inference time but also reduces sample quality during the rollout stage, thereby hindering the effectiveness of training. This tension raises a critical question: can LRMs be trained efficiently under strict memory budgets without sacrificing reasoning accuracy?
In this work, we introduce Progressive Thought Encoding, a parameter-efficien fine-tuning method designed address this bottleneck. Rather than discarding evicted tokens, our approach encodes their information into fixed-size vector representations that preserve long-context understanding under limited caches. We dynamically embed this contextual information into lightweight LoRA adapters, allowing the model to retain key reasoning signals without increasing cache size. By integrating this online adaptation into reinforcement learning, our method reduces peak memory usage during post- training. The learned adapters further enable the model to maintain strong reasoning performance under constrained computational budgets during inference.
We evaluated our method on three representative models: Qwen2. 5-54-B 1--5----Inst-i Inst Qwen2. -nntntsst---ttttte-----e Inst ruct, and ThepSeee-RR-Distee MiL--Distll--laaa amaamma-8-0 8B, across six challenging mathematical reasoning benchmarks. Our approach consistently outperforms vanilla RL training, achieving up to 23.4% improvement in reasoning accuracy on AIME while reducing GPU memory usage by nearly 50%. These results demonstrate that cache-aware reinforcement learning not only makes training large reasoning models more efficient but also improves their reasoning capabilities.
Our contributions can be summarized as follows:
We identify the fundamental inefficiency of RL training for LRMs under long rollouts and formalize it as a cache-constrained optimization problem.
We propose Progressive Thought Encoding, parameter-efficient fine-tuning technique that learns from evicted tokens to preserve reasoning capacity under bounded memory.
Through extensive experiments open-weight models and math benchmarks, show that our method substantially improves both training efficiency and inference robustness, setting new standard for scalable reasoning model training.
RELATED WORK
Test-time Learning of LLMs. Test-time learning (TTL) explores how LLMs can adapt to new tasks distributions without offline retraining (Hu et al. 2025). The most basic form is in-context learning (Dong et al. 2022). where demonstrations embedded within the prompt elicit task-specific behavior, while aatrieaal-augmente generation (RAG) extends this idea by providing task-relevant documents at inference (Gao et al.) 2023 Han et al. 2024 Cheng et al.] 2025). More advanced methods allocate additional computation for reasoning during inference, including tree-of-thought search (Yao et al. 2023), self-consistency across multiple reasoning paths (Wang et al. (2022). and iterative refinement (Madaan al. 2023). Another line of work investigates gradient-based updates at test time, such as test- time training (Zuo et al., 2025) and entropy minimization techniques (Zhang al. 2025b Agarwal et al., 2025), while recent theory establishes connections between instruction tnning based TTI. and low-rank parameter updates in LLMs (Dherin et al. 2025).
Parameter-effiiient Fine-tuning of L.Mss. Since the introduction of Low-rank Adaptation (LoRA) (Hu et al.) 2022). numerous parameter-efficient fine-tuning (PEFT) methods have been developed to improve the efficiency of adapting large language models (LLMs) to downstream tasks, including QLoRA (Dettmers al. 2023), LiSA (Pan et al.] 2024), and prefix-tuning (Li and Liang 2021). While these approaches primarily focus on offline task adaptation, recent work has extended low-rank techniques to enable dynamic test-time learning, such as generative adapters (Chen et al.) 2024) and stream adapters (Muhtar et al. (2024), which allow LLMs to adapt on-the-fly to new inputs or distributional shifts, thus enhancing robustness and flexibility.
2

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 NOTATION AND PRELIMINARIES
Attention and the KV cache as memory. In the prefilling stage, given sequence (Z1,.... ..... ; each token I, is mapped to hidden state fú, which is then projected into query, key, and value vectors, W gghh U, Wyh,, where Wq. WK K, and Wv are learnable weight matrices. Let K; = [k,,..., .......,, kj] and V42 = [uI,..., my] denote the cache of keys and values up to step t. The attention output for token ny given by
i.e. q, Wohi
softmax @t
Vr.
During the decoding stage, for the next token 24+1, first compute its query Qt+11 and then let attend over the extended KV cache:
0++1 softmax g++1 ge+i|Ke, ke-1]
[Vi, Me+1]-
Vd
Thus, the KV cache grows incrementally with each new token, serving as the memory that avoids redundant computation during autoregressive decoding and improves long-context understanding.
GRPO for Reinforcement Learning in LLMs. Grouped Reinforcement Policy Optimization (GRPO) is policy gradient method designed to fine-tune large language models. Unlike classical RLHF approaches, GRPO discards the need for a critic model and instead samples multiple candidate completions per prompt, groups them, and assigns rewards at the group level.
Given a prompt the model generates n completions {øI...., v.. } at the rollout stage. Then, each completion Yi is assigned raw score s; by reward model, which is then normalized within the group to produce variance-reducel rewards:
12 1
1
8:
Sj.
Н (0g 5)2 +
The policy is updated to maximize the expected reward while staying close to reference policy Teef
(r()) (L(((-lp) mrer(-1p))
( (y
where r(y) is the group-normalized reward and B controls the KI. regularization strength. Using relative rewards within each group, GRPO provides stable training signals without critic and aligns naturally with autoregressive generation in LLMs.
3.2 CHALLENGES FOR EFFICIENT RL TRAINING
Difficult tasks often require long reasoning trajectories (Yang et al. 2025), i.e., generating more tokens obtain high-quality solutions for reward computation. The effectiveness of passive test-time scaling (Muennighoff et al.) 2023) further underscores the importance of extended reasoning in solving difficult problems. However, this demand for longer generations directly amplifies the inefficiency of the rollout stage, which has been identified as the primary bottleneck to RL training (Zheng et al. 2025b Han et al., 2025 Zhang et al. 2025a). Despite the use of KV caching to avoid redundant computation, rollouts still dominate both time and memory costs due to continuous autoregressive decoding, making efficient training particularly challenging under outcome-based reward settings.
A natural approach to mitigating memory consumption is to adopt a dynamic sliding window strategy for the KV cache (Zhang et al.) 2023). thus keeping memory usage approximately constant even as the roll-out sequences grow longer. However, aggressive token drop can significantly impair long-sequence understanding and generation (Jin et al. 2024 Cai et al. 2024), which in turn weakens the model's reasoning ability during rollouts and ultimately reduces training effecttivnesss. As illustrated in table I applying a sliding- windww cache to RL training of Qwen models leads to a clear performance drop compared to training with the full cache of all tokens. This naturally raises critical question: can we maintain a constant-cr constant-capacity cache window while still enabling the reasoning model effectively "see" all previous tokens for efficient reasoning?
3

---

Published as a conference paper at ICLR 2026
Update parameter
Sampling
3
Answer
Problem prompt
4
Answer
3
2
3
Answer
2
Answer
Evicted token
Cache
window
model
n-th token generated
EOS token
Figure Overview our method. During the rollout process, the model continuously learns the dropped tokens to achieve balance between generation efficiency and long-term memory.
To formalize this challenge, we modify the standard GRPO formulation by redefining the rollout distribution. the original objective, a trajectory is sampled under the full-cache policy Tp(: (- p). In our setting, the trajectory is instead generated under cache policy D. which prunes the KV cache online during decoding. At each step D selects pruned context CP = CachePrune yct). and the token distribution becomes
T
8 v
p)
II
Tove CP)-
(2)
Accordingly, the cache-aware GRPO objective is
CuRoo Oret) = E By (-1) ((w)) - B 3KL T8 (. (-1p) Teneh C (1p))
(3)
where Bg denotes the parameters of the generating model under partial-cace reference model that operate with the full cache. Given a task prompt after the model training, we expect moz (y p) zs (v p), where 0y and 0" are optimized from eq. 3 and eq. (1) respectively.
rollouts, and fzet is
3.3 OUR APPROACH: LEARNING THINK TOKENS PRIOR TO EVICTION
Motivated by prior work on dynamically adapting models to novel inputs test time (Chen et 2024 Muhtar et al. 2024). we take different approach from simply discarding the evicted thinking tokens. Instead, first learn from these tokens up- date small set of parameters fg. enabling the test-time policy x x p) to approximate the full-cache policy T0(y p) under given eviction strategy D.
Cross- attention
Sya
FaR
Evicted key/value
Specifically, for a given question æ, during the rollout stage, we continuously decode next think- ing tokens (yI.....34 3 based on the policy Tg (y p) until the KV cache is full. Based on the token eviction strategy D, earlier tokens {yer..... ..........e.............. will be evicted from the cache. Rather than discarding these tokens, we use them to update compact latent representation
Norm
Self- attention
query
B
Low-rank update
Global key//aluu
Figure 2: The computation of context state S. with the help global query vector Jy. which serves as a learnable summary all evicted context encountered far. The update the LoRA weights is computed
A W A (((Wa,)) (W3 (WRV.) B,
(4)
where we denote 9g as global latent query that aggregates information from evicted tokens, W W and We the weight matrices map the global query tokens gg. evicted key K, and value tokens
4

---

Published as a conference paper at ICLR 2026
Ve into the compressed latent space, A and B the weight matrices map the evicted context state Se computed by the evicted tokens to the model weights.
The model then continues decoding {31+1..... under the updated policy Th (y) p), where O' = 0+ W. Each time the cache fills, we compute a new evicted context state S and update Se Normalize(S. S²), and recompute & W accordingly.
To bootstrap adaptation, before processing any evicted tokens we initialize the context state with learnable global tokens as S2 (Wa9, (WRk,) kgg) Wip where we define hg the global tokens and q3 W hh: Ag Wx hg. and 2 Wyhg This initialization makes qy explicit carrier of evicted-context information, enabling streaming adaptation while keeping memory usage constant. The full computation is illustrated in fig.
The selection of D during training. In our training setup, all question tokens are permanently retained in the cache, while a simple sliding winddww eviction strategy is applied only to the thinking tokens. This straightforward design supports efficient batch operations across samples, whereas more sophisticated importance- papddt eviction would incur additional computational overhead. The decision to always keep question tokens is directly motivated by the sink-token mechanism in (Zhang) al. ) as both serve to anchor and preserve the prompt context, ensuring that the model maintains stable grounding even when the th-in-of- becomes very long.
4 EVALUATIONS
Table 1: Comparison of methods across different models on benchmark datasets. The best average performance per model is highlighted bold. Note: Benchmark improvements are reported relative to Baseline, while FLOPs/Memory reductions are reported relative to LoRA.
Studied Models
Methods
Maximum TFLOPs of Attention
Peak GPU
Mcm. (%)
Mean GPU
Mem. (%)
Math500 pass
Olympiad 0
Minerva Math pass
p
AMC
AIME2024
AIME2025
16
Buseline
50.8
27.2
16.1
34.3
20.0
13.3
26.9
Quen2.553B-Inssiee
LoRA,
LoRA
2.6.1
4.2
38.0.4u
82.8
31.0.
63.5
S3.2.2
50.0.ax
27.8.cn
27.7-oa
15.9a3
16.
35,9.16
33.1-1.2
16.7.3
20.0
16.7.a4
10.0
28.2.1.
25.6..3
Ours
2.7.15
45.
32 32.6.309
54.0.32
29.0.18
16.2.u.
45.0,m0
16.7.34
Baseline
56.8
'4,7
18.5
48.4
23.3
16.6
35.1
Qwen2.5-7B Doon2.5-7B-Instruu
LoRA,
LoRA
3..12
5.7
63.1
85,8
45 45.4.33
59.3
59, 59.4126
38,7
23.7.52
23.4.49
52.5..1
50.6.12
30.0.61
20.0.3
26.7.101
26.7.00
38.1
36.7 Jas
Ours
3.6
67.2.
48.6.p07
61.2,24
38.7 4m
25.3
42.5.41
30.0,.
30.0
39.6
Baseline
53.6
28.7
18.6
42.5
10.0
20.0
30.1
DeepSeek: RI Distil-Lllama-88
LoRA
7.4
88.7
53.5
57.4,18
35.3.66
18.3
55.0.125
23.3,33
20.0.c
34.9.a3
LoRA.
4.6.a
19.1.
47.1
54.2-06
31.9-13
16.0-4
45.0.15
3677-16
26.7-2
35.1.50
Ours
4.6.38
59.8.1.9
46.8.67
$7.6..10
39.7.110
16.5404
60.0.175
56.7.367
43.0.m
45.6.155
4.1 EXPERIMENTAL. SETUP
Models. We evaluate our method on three representative ween-weight instruction-tume models of varying scales and architectures: (1) Qwen2
(Team 2024), 4. Baaaaaeeter transformer with 32 decoder layers, a hidden dimension of 4,096, 32 attention heads (128 dimensions per head), and rotary positional encodings; (2) Qwen2. 7B Inst [Team] 2024), a mid- scale 7.2B-parameter model with 32 decoder layers, hidden size of 5,120, and 40 attention heads. Its architecture follows the same design principles as the 4B variant but with larger hidden width and attention capacity; (3) DeepSeek wtw-----iiiiii-------i------ilie
(Guo et al. 2025 Vavekanand and Sam 2024). an 8.0B-parameter model distilled from DeepSeek-RI into LLaMA-3.1-88 It comprises 32 transformer layers with hidden dimension 4,096, 32 attention heads, SwiGLU activation, and rotary embeddings. Compared with the original LLaMA-3 3. 1-8B model, it has better capacity on long-sequence generation.
Benchmarks and Metries. We conduct evaluations on six math reasoning benchmarks covering diverse difficulty levels and reasoning depth: (1) Math500 (Hendrycks et al. 2021). curated set of 500 challenging word problems requiring symbolic and multi-step reasoning; (2) Olympiad- Bench (He et al. 2024). 674 olympiad-styy problems designed test deep mathematical reasoning; (3) Minerva Math (Lewkowycz et al.) 2022), 672 problems sampled from arXiv and textbooks, emphasizing symbolic manipulation; (4) AMC (merican Mathematics Competitions 2023). 40
5

---

Published as a conference paper at ICLR 2026
mahh500
shimstdd
350
322
335
100
16.0
200
250
325
235
369
270
720
15:55
26
76
26
30
76
1536
26
30
16
2535
28
150
1035
(a) Evaluation of Qwen2 - models.
LakA
ma/5000
15-
12-
10-
1535 Mindzw angh
Windbw hirgt
Window lmggo
Window largth
Windie kerth
Figure 3: Evaluation of Owen-7B-Instruce and -aw.-l--aaaaa--a models.
(b) Evaluation DeepSeek-RR--iisee
mod- els trained by different methods on four benchmarks. We set the same maximum number of tokens for generation as 3072, and vary the KV cache window length from 768 3072. Each value corresponds to the mean pass @ score over five independent runs.
middle- to high-school competition problems focused on combinatorics, number theory, and algebra; (5) AIME2024 and AIME2025 (Codeforces), recent American Invitational Mathematics Examina- tion sets, each containing 30 highly challenging problems. Due to their extreme difficulty, AIME datasets are evaluated using the pass@ 16 metric. For all other datasets, we report pass@) averaged over 3 independent runs, to ensure fair and robust comparisons.
Compared Methods. We compare four approaches: (1) Baseline, the original model prior to RL training: (2) LoRA, RL-trained models with low-rank adaptation applied: (3) LoRA,, RL-trained models with LoRA and a sliding-winden cache for token eviction; (4) Ours, RL-trained models using proposed method, where evicted tokens explicitly learned before being discarded.
Implementation Details. Unless otherwise specified, the maximum sequence length during rollout set to 3072, with global batch size of 512. We use the DAPO-Math-1777 17K dataset (Yu al. 2025) as our training dataset. We use the Adam optimizer with a learning rate of x 10-5 and a maximum gradient norm of 1.0. The rank of LoRA and our method is fixed 32, For LoRA, and our method, the sliding-window cache size is set to the maximum question length in the current micro-batch, with 25% of tokens evicted upon cache saturation to improve efficiency during training and inference. Our method additionally employs 32 global tokens. All models are trained until convergence, and experiments are conducted on NVIDIA A100 GPUs (40 GB each).
4.2 EVALUATION ON MATH REASONING TASKS
We first evaluate the training efficiency and task performance of the trained models using different methods. Training efficiency quantified using three metrics: (i) maximum TFLOPs required by attention, (ii) peak GPU memory utilization, and (iï) mean GPU memory utilization across training. These jointly reflect the computational and memory efficiency of the different cache strategies. Table reports the results.
Qwen2. 1-33BInnsttuee TFLOPs and nearly 83% peak memory usage. LoRA, reduces peak memory to 38% but accuracy drops to 25.6%. contrast, the proposed method achieves 30.1% %. the highest across all methods, while requiring only 2.7 TFLOPs and 45% peak memory. This demonstrates that naive eviction
Full-cache LoRA attains 28.2% average accuracy but requires 4.2

---

Published as a conference paper at ICLR 2026
Table 2: AIME2024 and AIME2025 pass 16 results (%). Maximum generation length is 6,144 tokens. KV cache window sizes range from 768 to 1,536. Note: Improvements are reported relative to Baseline.
DeepSeck-Rl-
Models
Qwen2 5.1.5544-BBiii Instruct
Qwen2.55778 In--srrrcttt
Distili Liaill-llmaBBB SB
Dataset
Method
768
1024
1536
Arg.
768
1024
1536
Avg.
768
1024
1536
Avg.
Baseline
10.0
16.6
20.0
15.53
23.3
19.97
3.3
3.3
8.87
AIME2012
LoRA
10.0a
13.3
20.0ua
14
100
23.3
30.0,4
21,10,11
3.300
3.300
23.3,33
9,97.1
LoRA,
13.3a3
13
167
14
16.6
20.0.67
20.0
18.87
677.1
16.7.14
367.167
10.03,15
Ours
16.7
20.0,sa
20.0u
18,90
26.6.z
26.6.11
30.0.a>
27.73
26.7aza
30.0,a
56.7.w
37,80.
Baseline
6,7
13.3
13.3
11.10
10.0
16.7
16.6
14.43
6.7
10.0
20.0
12.25
AIME2025
LoRA,
LoRA
6.700
6.7wr
10.0.))
6.7as
16.7.s
10.0.s
10.03.u
8.90
20.0.
6.7,3
26.7.ico
23.3.a>
26.7.mu
26.7.m
24.47
18,90.43
6.7cm
6.76m
20.0,00
10.0w
26.7.a1
20.0kn
17,79,,6
12:23
10.0.ss
16.7,54
16.7.s
1444,
23.3.113
26.7.m
26.7.m
25.60.
26.7.mm
30.0,am
43,3,25
33.
severely harms reasoning performance, but eviction- aware training not only recovers but improves accuracy relative to full-cache LoRA.
Qwen2. 5-7B-Inst ruct. The trade-off between accuracy and efficiency becomes more evident at larger scale. LoRA achieves 38.1% accuracy but incurs high memory cost (85.8% peak). LoRA, lowers memory to 63. 1% but reduces aceuracy to 36.7%. The proposed method achieves the best average accuracy (39.6%) %) while cutting FLOPs almost in half compared to LoRA (3.6 vs. 5.7). This suggests that eviction-aware training particularly beneficial as model size increases.
D iiittillll am. For the largest model, efficiency constraints dominate. Full-cache LoRA requires 7.4 TFLOPs and 89% peak memory. LoRA, reduces resource usage but sacrifices accuracy. By contrast, our method yields marked performance gain, achieving 45.6% average accuracy, a +10.7 improvement over LoRA, while consuming only 4.6 TFLOPs and 59.8% peak memory. The improvements are especially notable on challenging benchmarks such as AIME2024 (433.4) and AIME2025 (423.3).
4.3 EVALUATION UNDER DIFFERENT COMPUTATIONAL BUDGETS
To assess the robustness of different methods under constrained memory, we evaluate performance across progressively reduced KV cache sizes. In practice. such reductions correspond tighter computational budgets during inference, where only a fraction of the activations can be stored.
Figure 3 summarizes results across multiple reasoning benchmarks, including Olympiad, Miner- vaMath, AMC, and Math500, where we set the maximum response length as 3,072. Each curve reports accuracy as the available cache decreases from full capacity to highly constrained settings. As expected, the Baseline and LoRA methods degrade rapidly with shrinking cache size, reflecting their reliance on complete historical context. LoRA, alleviates this issue to some extent by incorporating adaptation learning from the training process, but its effectiveness remains limited when the window becomes narrow. In contrast, our method consistently sustains higher accuracy across all computational budgets, demonstrating resilience to cache truncation. Quantitatively, aver- aged across all datasets and cache settings, our approach achieves an accuracy of 39.37, compared to 32.99 for LoRA and 30.31 for the Baseline. This corresponds to relative improvements of +99.3% over LoRA and +29.9% over the Baseline. Importantly, these gains are achieved without requiring additional inference-time memory, our method maintains constant cache usage regardless of the budget.
sliding- window sliding-window
We further validate these findings on harder benchmarks, AIME2024 and AIME2025, which re- quire longer chains of reasoning. Here, we allow up to 6,072 tokens for generation (exceed- ing the training setting) and set the maximum cache size to 1,536 tokens to accelerate decod- ing. We then report @ 16 scores across cache sizes {768, 1024, 1536] in table 2 Across both years and all backbones, our method achieves the highest average performance. Rela- tive to LoRA, the average gains are +6.63 +6.70 on Qwen2. 5 15--7-150 Instruct, and +27.83 / +21.11 on i.e., LoRAc, are likewise substantial (e.g., +8.86 on Qwen2. 5- AIME2024 and +27.77 on DeepSeek-RL.
ama--aamm-888 8B. Improvements over the sliding-window Cache,
AIME2024), underscoring the limitations of navee con- text truncation. More results Qwen-2.5-4BB Tnat are provided in appendix B
7

---

Published as a conference paper at ICLR 2026
Table 3: Training efficiency com-
parison across different maximum generation lengths during rollout.
20
Mathod
Vanita
With
Deveudton Lorgh
3K
0X
3K
4k
5K
Peak caru mom (%)
86.7
95.5
59.8
50.2
604
(c) Global tokens.
(d) Token dropping.
(%)
MADH--00
Figure 4:
Ablation study on (a) global token usage and (b) token dropping strategies.
In summary, while our approach reduces training cost, particularly by lowering peak memory usage, without sacrificing task performance, these results further show that also reduces inference cost by sustaining accuracy under tight cache budgets.
4.4 ABLATION STUDY AND DISCUSSION
Progressive Thought Encoding Enables Scalable CoT RL Training. We employ the proposed progressive encoding method to efficiently reduce memory consumption, particularly peak usage during training. By lowering memory requirements, we enable longer and more complex reasoning processes during the rollout stage. In this section, present experiments demonstrating how the saved memory allows us to train DeepSeek- R1 R1-Distiiiiiii -Llama with larger maximum generation lengths, ranging from 4K to 6K tokens per rollout sample.
As shown table 3 increasing the maximum generation length during rollout con- MATH-500. Mcanwhile, progressive en-
sistently improves reasoning performance coding keeps both peak and mean memory usage stable and significantly lower than vanilla RI. training. Encouraging the model generate longer outputs not only supports more extended reasoning but also leads to consistent gains on MATH-500. These results demonstrate that can achieve longer reasoning with limited memory over- head, yielding better overall performance.
Generation of long sequences for reasoning. To assess the scalability our method under extended reasoning trajectories, we evaluate the RI-trrineed DeepSeek- MaeoSeek-R--Diie R1-Distill-L 8B model on MATH-500 using substantially longer generation lengths. During inference, we fix the context window to IK tokens to impose a strict memory constraint, while vary- ing the maximum generation length from 3K up to 64K tokens. This setup allows us to examine how well different approaches lever- age increasingly long reasoning chains when the available KV cache remains limited The results are provided in fig. 5 showing that all methods benefit from longer reasoning sequences.
Score EN on MArn-500
3K
6K
12K
24k
45K
64X
Mak generation longth
Figure 5: Performance on MATH-500 under fixed IK context window the maximum generation length increases from 3K to 64K.
Across the entire length range, method demonstrates the strongest scaling behavior. The original model, LoRA, and LoRA, show moderate improvements that gradually plateau as the sequence grows longer, whereas our approach continues to yield steady gains even 64K tokens. This indicates that progressive thought encoding not only preserves reasoning information under tight cache budgets, also scales favorably as reasoning trajectories extend far beyond the training rollout length.
The use of global context tokens. In our proposed method, we introduce global tokens to improve training efficiency. To evaluate their impact on model performance, we compare against several base- lines: (1) Baseline, the original Qwen 2. S-Innstuuct model; (2) Global-Only.
method with the update of context state S. from evicted tokens disabled: (3) #Global-O, initializing s, with zero, effectively removing global token initialization: and (4) #Global-16/32/488
method with the number of global tokens varied from to 64. We conduct experiments on the MATH-500 dataset under different cache sizes (756, IK, 1536, 2K, 3K). The results presented in fig. 3c
8

---

Published as a conference paper at ICLR 2026
It can be observed that disabling global tokens (#Global-O) yields only marginal improvements the baseline. In contrast, integrating global tokens with the evicted token update of S. consistently enhances performance across different KV cache lengths. outperforming the Global-Only variant by a clear margin of 1.2% at just 768 cached tokens. However, adding more global tokens does not always lead to better results: for example, #Global-64 underperforms compared with #Global-32 and #Global-16 at the most constrained cache length of 768 tokens.
Integration with inference-time token dropping strategy. In our work, we adopt the sliding window strategy for token eviction, which does not account for token impor- tance. To address this limitation, we integrate several advanced token dropping strate- gies during generation and evaluate their performance on the MATH-500 dataset, including H2O (Zhang et al. 2023), P (Cai et al., 2024). and HeadKV (Fu et al. 2024).
As shown in fig. 3d compared to the sliding window eviction strategy, these advanced to- ken dropping methods consistently improve rea- soning performance, particularly under limited cache capacity. For example, with a cache win- dow length of 768, the baseline model achieves success rate of 34.4%. Using a sliding window cache increases performance to 48. 48.4%, while VeadKV achieves the accuracy at 50.7%, nar- rowing the gap to full cache accuracy by 3.3%. These results demonstrate that token selection matters for reasoning efficiency.
However, these advanced strategies incur non- trivial cost. Integrating HeadKV during the roll- out stage (batch size 512) increases iteration time from 19 to 26 minutes (+37% runtime) for +2.3% accuracy gain.
Figure 6: The statistics on the generation length.
Consequently, we retain the dlding-windoww approach for training and leave efficient integration of advanced token-dropping methods into RL rollouts as future work.
On the length of generated response. We also analyze the distribution of generated response lengths across different methods on the MATH-500 dataset. We set the maximum number of generation tokens to 3096, the cache window size to 768, and the number of sink tokens to 512, i... 256 tokens stored within the sliding window.
As shown in fig. 6 although LoRA, outperforms vanilla LoRA under limited cache size (approxi- mately 10% 1. see fig. 3 most of these gains come from short responses, and only a few problems solved with long responses. In contrast, our proposed method not only achieves the best overall reasoning performance under this setting but also maintains strong capability on long-form reasoning. These results support our claim that dynamically encoding evicted tokens into model weights enables the model to consistently "remember" them throughout the generation process.
Why Progressive encoding can achieve better results? Our method achieves higher accuracy because the progressive encoding of evicted tokens provides a continuous mechanism for preserving long- range reasoning information that would otherwise be lost under sliding-windor truncation. Instead of discarding early thought tokens, their compressed contextual representations are folded into the LoRA weights, enabling the model to retain global reasoning signals even when only a small portion of the KV cache is visible. This acts as form of denoising and incremental distillation, strengthening the model's 'ss ability to maintain coherent multi-step reasoning trajectories. Empirically, this leads to longer and more stable chains of thought during problem solving (see fig. (). and substantially improves performance across constrained-cache settings. Together, these effects allow the model to approximate full-context reasoner while operating under tight memory budgets, explaining the consistent gains over LoRA and sliding-windt window baselines.
9

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We introduced Progressive Thought Encoding, a parameter-effiiiet fine-tuning approach that allows large reasoning models to train and infer effectively under limited computing resources, Rather than discarding evicted tokens, our method encodes their information into model weights, preserving long-context reasoning ability while substantially reducing memory and compute costs. Through experiments on three open-weight models and six challenging math reasoning benchmarks, we demonstrate consistent gains over LoRA and sliding-window cache baselines, achieving up to +23.4 absolute accuracy improvements AIME2024/2025 while cutting peak memory nearly in half. Beyond boosting efficiency, our results show that cache-aware training enhances reasoning robustness under constrained computational budgets, enabling longer and more effective rollouts during RL training. We believe this work is a step toward scalable RL training for reasoning models and opens promising directions for adaptive eviction strategies, multimodal reasoning tasks, and integration with inference-time optimization techniques to further advance the efficiency-accuracy frontier.
REFERENCES
Shivam Agarwal, Zimin Zhang. Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effective- ness of entropy minimization in Ilm reasoning. arXiv preprint arXiv:2505.1511 2025.
Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho. Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 12562-1255 2024.
MAA American Mathematics Competitions. American mathematics competitions, 2023.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based pyramidal information funneling. arXiv preprint arXiv:2406.02065 2024.
Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, and Hao Cheng- Generative adapter: Contextulizing language models parameters with single forward pass. arXiv preprint arXiv:241 1.05877, 2024.
Mingyue Cheng, Yucong Luo, Jie Ouyang. Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, et al. A survey on knowledge-oriented retrieval-augmented generation. arXiv preprint arXiv:2503. 10677, 2025.
MAA Codeforces. American invitational mathematics examination-aii aime 2024, 2024.
Neisarg Dave, Daniel Kifer, C Lee Giles, and Ankur Mali. Investigating symbolic capabilities of large language models. arXiv preprint arXiv:2405.13209 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-10000 10115, 2023.
Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, and Javier Gonzalvo. Learning without training: The implicit dynamics of in-context learning. arXiv preprint arXiv:2507. 16003, 2025.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00223 2022.
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. arXiv preprint arXiv:2504. 10903, 2025.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pages 10421-10430. PMLR, 2023.
10

---

Published as a conference paper at ICLR 2026
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter. A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint 240.::2400.1925,, 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmmnned generation for large language models: A survey. XrXiv preprini urXiv:2312.10997, 2(1), 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprins arXiv:2501.12945 2025.
Haoyu Han, Yu Wang. Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. Rerrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00 2024.
Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang. Wenqi Shi, Menglong Chen,
Sicheng Zhang, Zeshun Lan, Chunshi Deng, et al. Asyncflow: An asynchronous streaming rl framework for efficient Ilm post-training. arXiv preprint arXiv:2507.01663, 2025.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038 003874 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Jinwu Hu, Zhitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models. arXiv preprint arXiv:2505.20633, 2025.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, arXiv preprint vrXiv:2212.104000 2022.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend Ilm context window without tuning. arXiv preprint 24X0:22001.01225, 2024.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/99 2021.
Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian. Optimizing safe and aligned language generation: A multi-objective grpo approach. arXiv preprint arXiv:2503.21889, 2025a.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint 2ix::2022.174111 2025b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feff-eee Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
11

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
Dilxat Muhtar, Yelong Shen, Yaming Yang. Xiaodong Liu, Yadong Lu, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Weiwei Deng. Feng Sun, al. Streamadapter: Efficient test time adaptation from contextual streams. arXiv preprint arXiv:2411.0928) 2024.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. Advances Neural Information Processing Systems, 37:57018-57049, 2024.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang. Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprini arXiv:2501.14249, 2025.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511 2024.
Samuel Schmidgall, Yusheng Su, Ze Wang. Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using 1lm agents as research assistants. arXiv preprint arXiv:2501 0422270 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06323 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024.
Raja Vavekanand and Kira Sam. Llama 3.1: An in-depth analysis of the next-generation large language model. Preprint, July, 2024.
Pciyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. arXiv preprint arXiv:23/2.08933. 2023.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1:XXi::2233.1177. 2022.
Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, and Mingxuan Yuan. Unlocking efficient long-to-short I1m reasoning with model merging. arXiv preprint arXiv:2503.2066 2025.
Jiaming Xu, Kaibin Guo, Wuxuan Gong, and Runyu Shi. Osagent: Copiloting operating system with llm-based agent. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2024.
Wang Yang, Zirui Liu, Hongye Jin, Qingyu Yin, Vipin Chaudhary, and Xiaotian Han. Longer context, deeper thinking: Uncovering the role of long-context ability in reasoning. arXiv preprint vrXiv:2505.17315. 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:1 1809-11822 2023.
12

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXtv:2503.14476. 2025.
Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Rlep: Reinforcement learning with experience replay for Ilm reasoning. arXiv preprini arXiv:2507.07451, 2025a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint Xiv:2254.005812, 2025b.
Yiqi Zhang, Huiqiang Jiang, Xufang Luo, Zhihe Yang, Chengruidong Zhang, Yifei Shen, Dongsheng Li, Yuqing Yang. Lili Qiu, and Yang You. Sortedrl: Accelerating rl training for ilms through online length-aware scheduling. In ES-FoMo IlI: 3rd Workshop on Efficient Systems for Foundation Models.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34611-34716 2023.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025a.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Ilm reasoning via selective rollouts. arXiv preprint XrXiv:2566.02177 2025b.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuckai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long. Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint a5Xiv:2504. 16084, 2025.
13

---

Published as a conference paper at ICLR 2026
Tilh-500
Ag:
D.
350
1536
30
700
26
1505
704
1035
38
1335
26
100
35.4
26
30
Figure A1: Evaluation of Qwee2.5-4B-Instieee models.
A THE USE OF LARGE LANGUAGE MODELS
In accordance with the ICLR 2026 policies the of Large Language Models (LLMs), disclose that we used an LLM (OpenAI's ChatGPT) solely for writing assistance. Specifically, the model employed polish the language of the manuscript, including improving grammar, clarity, and readability.
No part of the model's 's output used to generate research ideas, derive results, conduct experiments, or analyze data. All scientific contributions, including the design experiments, implementation of methods, data analysis, and interpretation of results, entirely the work of the listed authors, who take full responsibility for the content of this paper.
B RESULTS ON
Following the settings in Section +3 evaluate Bren-2.5-4B-Instzue under different KV-cache budgets, with results shown in Figure AT Across all four benchmarks (math500, olympiad, minervanth, and amc), our method (red curve) consistently outperforms the Baseline, LoRA, and LoRA variants. The gains are most pronounced at shorter window lengths (e.g., 768 and 1 K), where baseline models experience substantial accuracy degradation. For instance, math500, improves by more than 12 points over the baseline at 768 tokens, and it maintains its advantage even the window length grows 3 K. Similar trends appear olympiad and amc, where our curve remains flat and robust while the baselines fluctuate or decline.
approach
The rightmost panel shows the averaged results across all tasks, where our method consistently achieves the highest performance across the entire range of window lengths. Notably, our curve peaks around 1.5 K and remains stable thereafter, suggesting that our approach not only more resilient to cache constraints but also scales gracefully with longer contexts. This demonstrates that training with cache-awar aware eviction leads to robust generalization and mitigates the performance drop observed in other fine-tuning strategies.
C IMPACT OF CACHE-EVICTION STRATEGY ON THE UPDATE OF 4g
To analyze how the cache-eviction strategy influences the update of the global latent vector 9g. we evaluate under different eviction ra- tios. The training setup matches that used in the main experiments, where 25% ratio applied during training. At inference time, however, vary the ratio from 25% to 5% while keeping the context window fixed at 1024 tokens. The results reported in AT
Table A1: Effect of eviction ratios on MATH-500.
5--en-2.5-3B-11
Ratio
25%
20%
1.5
10%
5%
Score
50.8
51.4
51.9
50.7
49.6
We observe that decreasing the eviction ratio initially improves performance: reducing the ratio to 15% yields the highest accuracy, suggesting that more frequent but smaller update steps enable q, to capture more fine- -raiind information from evicted tokens. However, when the ratio becomes too small 5%). performance degrades noticeably. This indicates that overly fine-grained eviction
14

---

Published as a conference paper at ICLR 2026
leads to noisier update signals with insufficient contextual content per step, resulting in unstable LoRA adaptation.
Overall, these results show that the eviction strategy plays critical role in shaping the quality of the update signal for 9g- Moderate eviction ratios provide a more reliable balance between update frequency and information richness.
15


Reviews:
[{"review_id": "TGyZJO5po9", "reviewer": "Reviewer_J7E7", "summary": "The authors  introduced  a new memory efficient  RL method to post-train LLMs under under strict memory budgets. The key idea is to keep a  constant memory capacity cache window (similar to sliding window attention) . The authors also introduce global tokens . These global tokens are used with evicted from cache tokens  for weights update. The authors compared proposed method with 2 prior methods: (1) RL + LoRA (2) RL with LoRA and SWA.", "questions": "The comparison with prior work seems not complete.  The proposed method uses global tokens. Should not these tokens be also used in LoRA and LoRA_c for fair comparison? \n\nHow do you explain that Mean GPU memory for \"ours\" is less than LoRA_c in the last row of Table 1?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Good practical method to reduce memory footprint and compute during RL stage for reasoning models", "weaknesses": "The comparison with prior work seems not complete.  The proposed method uses global tokens. Should not these tokens be also used in LoRA and LoRA_c for fair comparison? \n\nI would recommend to remove sections 2.1,  2.2 and 2.3 from  the \"related work\" . This is common knowledge very weakly related to proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjsq8FvIcF", "reviewer": "Reviewer_arKo", "summary": "The paper proposes a parameter-efficient fine-tuning method under RL to address the compute resource challenges posed by long-context in difficult problems. The sliding-window caches or dynamic pruning of past tokens approaches, which affects inference accuracy. In contrast, the progressive thought encoding proposed in this paper, by updating the context state to preserve the complete context information as much as possible, ultimately improves the training effect while simultaneously reducing the computational cost.", "questions": "1. Experimental results for DeepSeek-R1-Distill-Qwen-7B are required;\n2. Consider supplementing with evaluation experiments at longer inference lengths;\n3. The analysis lacks an explanation for why this approach achieves higher accuracy compared to the LoRA.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method outperforms both LoRA and LoRA_c in terms of accuracy across various models and evaluation datasets, while requiring only a marginal increase in computational resources compared to LoRA_c.\n2. The method's reliability is demonstrated through comprehensive evaluation across multiple benchmark datasets.", "weaknesses": "1. Among the three models evaluated in this paper, Qwen2.5-4B-Instruct and Qwen2.5-7B-Instruct are not LRM. Their output lengths are shorter compared to DeepSeek-R1-Distill-Llama-8B, making their persuasiveness less compelling.\n2. The maximum sequence length is only 3072, lacking evaluation experiments at longer reasoning lengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4t8rlMA2y1", "reviewer": "Reviewer_MbcZ", "summary": "Large scale chain of though RL runs with full cache make it memory and compute intensive. Simple solution is drop earlier context once cache limit hit or sliding window cache but authors show this comes at cost reasoning peformance. Further they propose to learn a global query vector $q_g$ from these evicted tokens KVs which can function like a memory. This $q_g$ is updated with a simple self-attention like update rule and then injected into a lightweight Lora Adapter via a small weight update $ \\Delta W$. The model parameters now through these remember the evicted tokens. This adds minimal computational cost but allows scaling up RL by fixing cache size. They experiment with DAPO recipe on top of qwen-2.4 4/7B models and DeepSeek-R1-Distill-Llama-8B and show reasonable improvements over slidding window cache baseline.", "questions": "- What do the authors think is the impact of the cache eviction strategy on the q_g update would be. Eg bulk eviction or one at a time.\n- Contribution [1] : Is that a novel contribution or already established in the community. Authors should clarify\n- What do authors think about mamba hybrid models like nano-v2-12b https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2 as they are optimized for inference. Its a hybrid model so cache still grows up but not to the same level. Also they will still able to model the long reasoning context much better.\n- Comments on writing\n    - In line 196 Table 1 is mentioned but we don’t know what lora_c is and its not explained so confusing to read\n    - line 220 is not coherent\n    - global query is not well introduced as being the latent that learns the evicted context.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of encoding the “to be evicted token” into latent vectors which are them embedded into LoRA adapter is very novel and clever. This makes the Lora adapter like a learned dynamic memory without blowing up KV cache linearly\n- Well formulated and simple idea with clear experiments\n- Good ablations and analysis", "weaknesses": "- Results and experimental setup is not convincing enough.\n- Unclear if it will apply to full finetuned models compared to the LoRA only setting in this paper. Also unclear if this will translate over to larger models but I understand that will be outside the scope of this study.\n- max sequence length of 3072 is not enough for reasoning RL runs and doesn’t give me confidence in the results especially given the authors tout this as benefitial for long reasoning RL. Only at longer context lengths the benefits or downside of this appraoch can be demonstrated.\n- AIME25 still has variance at pass@16 so the slight improvment in those benchmarks is not conclusive\n- Its also unclear why this approach should be benefitial over full-cache setup especially given the short max 6k context experiments. At larger contexts I can see how longer context can hurt sometimes. More thorough investigation is needed.\n- $q_g$  is just one fixed vector for the whole evicted sequence. How much capacity can it even have? As we scale up the reasoning RL which can go upto 64k sequence length in frontier RL runs, I am not convinced this $q_g$  can hold that long context well. Ofc it can be better than no context of evicted tokens in sliding window cache but unclear how much we will lose compared to full cache.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dG4Hu6TzJT", "reviewer": "Reviewer_Dfk2", "summary": "This paper introduces the Progressive Thought Encoding method, a novel reasoning architecture for reasoning models. By encoding expelled intermediate reasoning tokens into fixed-size vectors and integrating them as a LoRA adapters, it avoids the high memory consumption of full-cache backtracking while preserving key reasoning signals. Experiments conducted on 3 open-source models and 6 mathematical reasoning benchmarks demonstrate that the proposed method achieves consistent improvements over LoRA and LoRA_c, while effectively reducing GPU peak memory usage and FLOPS.", "questions": "Please consider discussing the issues raised in the weaknesses section.\n\n4. I still have some confusion regarding the end-to-end procedure of RL training. For traditional RL, storing only the model's text responses is sufficient to compute the policy gradient. For the method proposed in this paper that adds LoRA heads to the model during test time, are any additional operations required?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method achieves better training results than PEFT methods on mainstream mathematical datasets.\n2. Monitoring of FLOPs and GPU memory confirms that the approach of compressing tokens and reducing activations indeed contributes to efficient training.", "weaknesses": "1. Baselines. Although the ultimate goal is to reduce computational overhead, I am uncertain whether LoRA is an appropriate baseline. This work appears to be more aligned with studies on token compression and similar model architecture methods rather than PEFT methods. Have the authors considered comparing with relevant approaches? Please correct me if there is a misunderstanding.\n2. Compatibility with existing infrastructure. The compression during reasoning seems to require per-sample maintenance of a series of related states, such as global query vectors. Can these modifications to the model architecture be easily implemented on transformers or vllm libraries to adapt to the existing infrastructure?\n3. Context window. A 3K context window is generally insufficient for complex mathematical problems, especially for models trained with reasoning capabilities. The Distill-R1 model also has a longer output length. How does the performance change when the output length increases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)