Title: Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
Primary area: foundation or frontier models, including LLMs
Keywords: Reinforcement learning with verifiable rewards, Large Reasoning Models, Exploration and Exploitation

Abstract: Reinforcement learning with verifiable rewards (RLVR) typically adopts *Pass@1 as the reward*, causing policies to prefer conservative and similar actions. The RLVR algorithm has faced issues in balancing exploration and exploitation. 
Thus, identifying an appropriate reward metric is crucial. To investigate this, inspired by the Pass@k metric that is used to evaluate a model's ability boundary, we first adopt Pass@k as the reward to train a policy (i.e., **Pass@k Training**), and observe the improvement in its exploration ability. Next, we derive an analytical solution for the advantage estimation process of Pass@k Training, reducing the resource overhead and estimation variance introduced by the sampling process. Our analysis reveals that exploration and exploitation are not inherently conflicting objectives, while they can mutually enhance each other. 
Moreover, Pass@k Training with analytical derivation essentially involves directly designing the advantage function. Inspired by this, we explore the *implicit reward design* in RLVR, showing promising results and highlighting a potential future direction.