DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per
  Token via Reinforcement Learning
Authors: name: Shih-Yang Liu; name: Xin Dong; name: Ximing Lu; name: Shizhe Diao; name: Mingjie Liu; name: Min-Hung Chen; name: Hongxu Yin; name: Yu-Chiang Frank Wang; name: Kwang-Ting Cheng; name: Yejin Choi; name: Jan Kautz; name: Pavlo Molchanov
Published: 2025-10-16 20:05:57

Abstract: Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve
strong performance via extended chains of thought but often generate
unnecessarily long outputs. Maximizing intelligence per token--accuracy
relative to response length--remains an open problem. We revisit reinforcement
learning (RL) with the simplest length penalty--truncation--and show that
accuracy degradation arises not from the lack of sophisticated penalties but
from inadequate RL optimization. We identify three key challenges: (i) large
bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward
signal. We address them with Doing Length pEnalty Right (DLER), a training
recipe combining batch-wise reward normalization, higher clipping, dynamic
sampling, and a simple truncation length penalty. DLER achieves
state-of-the-art accuracy--efficiency trade-offs, cutting output length by over
70 percent while surpassing all previous baseline accuracy. It also improves
test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple
concise responses in parallel with 28 percent higher accuracy and lower
latency. We further introduce Difficulty-Aware DLER, which adaptively tightens
truncation on easier questions for additional efficiency gains. We also propose
an update-selective merging method that preserves baseline accuracy while
retaining the concise reasoning ability of the DLER model, which is useful for
scenarios where RL training data is scarce.