Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement
  Learning
Authors: name: Chenghao Yang; name: Lin Gui; name: Chenxiao Yang; name: Victor Veitch; name: Lizhu Zhang; name: Zhuokai Zhao
Published: 2025-10-06 18:15:43

Abstract: Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm
for enhancing the reasoning capabilities of large language models (LLMs), yet
its success hinges on effective exploration. An ideal exploration strategy must
navigate two fundamental challenges: it must preserve sample quality while also
ensuring training stability. While standard fixed-temperature sampling is
simple, it struggles to balance these competing demands, as high temperatures
degrade sample quality and low temperatures limit discovery. In this work, we
propose a simpler and more effective strategy, Exploratory Annealed Decoding
(EAD), grounded in the insight that exploration is most impactful on early
tokens which define a sequence's semantic direction. EAD implements an
intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by
annealing the sampling temperature from high to low during generation. This
dynamic schedule encourages meaningful, high-level diversity at the start, then
gradually lowers the temperature to preserve sample quality and keep the
sampling distribution close to the target policy, which is essential for stable
training. We demonstrate that EAD is a lightweight, plug-and-play method that
significantly improves sample efficiency, consistently outperforming
fixed-temperature sampling across various RLVR algorithms and model sizes. Our
work suggests that aligning exploration with the natural dynamics of sequential
generation offers a robust path to improving LLM reasoning.