LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
Authors: name: Wenkai Yang, name: Weijie Liu, name: Ruobing Xie, name: Yiju Guo, name: Lulu Wu, name: Saiyong Yang, name: Yankai Lin
Organization: Tencent Hunyuan

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a core paradigm for enhancing the reasoning capabilities of Large Language
Models (LLMs). To address the lack of verification signals at test time, prior
studies incorporate the training of model's self-verification capability into
the standard RLVR process, thereby unifying reasoning and verification
capabilities within a single LLM. However, previous practice requires the LLM
to sequentially generate solutions and self-verifications using two separate
prompt templates, which significantly reduces efficiency. In this work, we
theoretically reveal that the closed-form solution to the RL objective of
self-verification can be reduced to a remarkably simple form: the true
reasoning reward of a solution is equal to its last-token self-rewarding score,
which is computed as the difference between the policy model's next-token
log-probability assigned to any pre-specified token at the solution's last
token and a pre-calculated constant, scaled by the KL coefficient. Based on
this insight, we propose LaSeR (Reinforcement Learning with Last-Token
Self-Rewarding), an algorithm that simply augments the original RLVR loss with
a MSE loss that aligns the last-token self-rewarding scores with verifier-based
reasoning rewards, jointly optimizing the reasoning and self-rewarding
capabilities of LLMs. The optimized self-rewarding scores can be utilized in
both training and testing to enhance model performance. Notably, our algorithm
derives these scores from the predicted next-token probability distribution of
the last token immediately after generation, incurring only the minimal extra
cost of one additional token inference. Experiments show that our method not
only improves the model's reasoning performance but also equips it with
remarkable self-rewarding capability, thereby boosting its inference-time
scaling performance.