Pass@k Training for Adaptively Balancing Exploration and Exploitation of
  Large Reasoning Models
Authors: name: Zhipeng Chen, name: Xiaobo Qin, name: Youbin Wu, name: Yue Ling, name: Qinghao Ye, name: Wayne Xin Zhao, name: Guang Shi

Reinforcement learning with verifiable rewards (RLVR), which typically adopts
Pass@1 as the reward, has faced the issues in balancing exploration and
exploitation, causing policies to prefer conservative actions, converging to a
local optimum. Identifying an appropriate reward metric is therefore crucial.
Regarding the prior work, although Pass@k has been used in evaluation, its
connection to LLM exploration ability in RLVR remains largely overlooked. To
investigate this, we first use Pass@k as the reward to train the policy model
(i.e., Pass@k Training), and observe the improvement on its
exploration ability. Next, we derive an analytical solution for the advantage
of Pass@k Training, leading to an efficient and effective process. Building on
this, our analysis reveals that exploration and exploitation are not inherently
conflicting objectives, while they can mutually enhance each other. Moreover,
Pass@k Training with analytical derivation essentially involves directly
designing the advantage function. Inspired by this, we preliminarily explore
the advantage design for RLVR, showing promising results and highlighting a
potential future direction.