LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM
  Reasoning
Authors: name: Weizhe Chen; name: Sven Koenig; name: Bistra Dilkina
Published: 2025-10-01 20:57:22

Abstract: Since the release of Deepseek-R1, reinforcement learning with verifiable
rewards (RLVR) has become a central approach for training large language models
(LLMs) on reasoning tasks. Recent work has largely focused on modifying loss
functions to make RLVR more efficient and effective. In this paper, motivated
by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy
Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects
training data at each step based on the average response length. We evaluate
LSPO across multiple base models and datasets, demonstrating that it
consistently improves learning effectiveness. In addition, we conduct a
detailed ablation study to examine alternative ways of incorporating length
signals into dynamic sampling, offering further insights and highlighting
promising directions for future research.