Title: Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Large Reasoning Models, Test-time scaling

Abstract: Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions—supervised fine-tuning on shorter traces, user-controlled budgets, or RL with uniform penalties—either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement‑learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt’s online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Post‑training DeepScaleR-1.5B with ALP cuts average token usage by 50\% without significantly dropping performance.  Relative to fixed‑budget and uniform‑penalty baselines, ALP redistributes its reduced budget more intelligently—cutting compute on easy prompts and reallocating saved tokens to difficult ones—delivering higher accuracy on the hardest problems with higher cost.

TL;DR: Training a model to allocate test-time compute budget based on difficulty of prompts.