SIRI: Scaling Iterative Reinforcement Learning with Interleaved
  Compression
Authors: name: Haoming Wen; name: Yushi Bai; name: Juanzi Li; name: Jie Tang
Published: 2025-09-29 17:59:08

Abstract: We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved
Compression, a simple yet effective RL approach for Large Reasoning Models
(LRMs) that enables more efficient and accurate reasoning. Existing studies
have observed repetitive thinking patterns in LRMs, and attempts to reduce them
often come at the cost of performance. In this paper, we show that this
trade-off can be overcome through a training regime that iteratively alternates
between compressing and expanding the reasoning budget, by dynamically
adjusting the maximum rollout length during training. The compression phase
cuts the rollout length, forcing the model to make precise and valuable
decisions within a limited context, which effectively reduces redundant tokens
and increases reasoning density. The expansion phase then relaxes the length
limit, providing space for the model to explore and plan in long-horizon
settings. Remarkably, we find that after each compression-expansion cycle, the
model's performance improves even as its output length decreases, steadily
pushing it closer to the Pareto frontier in the performance-efficiency
trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves
performance on AIME24 by 43.2% while reducing token usage by 46.9% after three
iterations, and SIRI-high achieves the highest accuracy compared to all other
methods (Figure 1). Our findings shed light on the potential of periodically
oscillating the LRM's output truncation length during training to dynamically
balance exploration and efficiency in reasoning, converging towards an optimal
"sweet spot" between the two. Our models are publicly available.