DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per
  Token via Reinforcement Learning
Authors: name: Shih-Yang Liu, name: Xin Dong, name: Ximing Lu, name: Shizhe Diao, name: Mingjie Liu, name: Min-Hung Chen, name: Hongxu Yin, name: Yu-Chiang Frank Wang, name: Kwang-Ting Cheng, name: Yejin Choi, name: Jan Kautz, name: Pavlo Molchanov
Organization: NVIDIA

Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve
strong performance via extended chains of thought but often generate
unnecessarily long outputs. Maximizing intelligence per token--accuracy
relative to response length--remains an open problem. We revisit reinforcement
learning (RL) with the simplest length penalty--truncation--and show that
accuracy degradation arises not from the lack of sophisticated penalties but
from inadequate RL optimization. We identify three key challenges: (i) large
bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward
signal. We address them with Doing Length pEnalty Right (DLER), a training
recipe combining batch-wise reward normalization, higher clipping, dynamic
sampling, and a simple truncation length penalty. DLER achieves
state-of-the-art accuracy--efficiency trade-offs, cutting output length by over
70 percent while surpassing all previous baseline accuracy. It also improves
test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple
concise responses in parallel with 28 percent higher accuracy and lower
latency. We further introduce Difficulty-Aware DLER, which adaptively tightens
truncation on easier questions for additional efficiency gains. We also propose
an update-selective merging method that preserves baseline accuracy while
retaining the concise reasoning ability of the DLER model, which is useful for
scenarios where RL training data is scarce.