Title: Decomposing Policy Optimization into Proxy Objective and Knowledge Distillation for Long-context Reasoning
Primary area: foundation or frontier models, including LLMs
Keywords: Long-context Reasoning

Abstract: Reinforcement learning (RL) paradigms have demonstrated remarkable success in enhancing the complex reasoning capabilities of Large Language Models (LLMs). However, models trained exclusively on short-context tasks often struggle to generalize their capabilities to longer documents. Extending RL to long-context settings remains a fundamental challenge, primarily due to the high computational cost and deteriorating quality of the sampling reasoning trajectories. To bridge this gap, we propose Decomposed Policy Optimization (DePO), a novel framework that decouples policy optimization for long-context reasoning into two parallel parts: (1) leveraging short-context environments as a tractable proxy objective to acquire high-quality and efficient reasoning patterns via RL; and (2) transferring these capabilities to long-context settings through knowledge distillation, enabling the model to replicate its refined short-context reasoning strategies over extended contexts. Empirical evaluations across multiple long-context document question-answering benchmarks show that DePO consistently outperforms all baseline approaches. Notably, compared to naive long-context reinforcement learning, DePO achieves 2.8\% improvement in performance while also reducing training time overhead by 49.5\%. Furthermore, DePO demonstrates strong generalization capabilities, compatibility with various RL algorithms, and consistent improvements across multiple base models, offering a scalable and efficient solution for advancing long-context reasoning.

TL;DR: Decomposed Policy Optimization (DePO) reformulates long-context reasoning via short proxy optimization and distillation, outperforming baselines while significantly cutting training cost.