Model-Task Alignment Drives Distinct RL Outcomes
Authors: name: Haoze Wu; name: Cheng Wang; name: Wenshuo Zhao; name: Junxian He
Published: 2025-08-28 20:02:10

Abstract: Recent advances in applying reinforcement learning (RL) to large language
models (LLMs) have led to substantial progress. In particular, a series of
remarkable yet often counterintuitive phenomena have been reported in LLMs,
exhibiting patterns not typically observed in traditional RL settings. For
example, notable claims include that a single training example can match the
performance achieved with an entire dataset, that the reward signal does not
need to be very accurate, and that training solely with negative samples can
match or even surpass sophisticated reward-based methods. However, the precise
conditions under which these observations hold - and, critically, when they
fail - remain unclear. In this work, we identify a key factor that
differentiates RL observations: whether the pretrained model already exhibits
strong Model-Task Alignment, as measured by pass@k accuracy on the evaluated
task. Through a systematic and comprehensive examination of a series of
counterintuitive claims, supported by rigorous experimental validation across
different model architectures and task domains, our findings show that while
standard RL training remains consistently robust across settings, many of these
counterintuitive results arise only when the model and task already exhibit
strong model-task alignment. In contrast, these techniques fail to drive
substantial learning in more challenging regimes, where standard RL methods
remain effective.