LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
Authors: Wenkai Yang
Weijie Liu
Ruobing Xie
Yiju Guo
Lulu Wu
Saiyong Yang
Yankai Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
LLM Reasoning
Self-Rewarding
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose LaSeR, an highly efficient and effective algorithm for jointly optimizing the reasoning and self-rewarding capabilities of LLMs.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). To address the lack of verification signals at test time after RLVR, prior studies incorporate the training of model's self-verification capabilities into the standard RLVR process, thereby unifying reasoning and verification capabilities within a single LLM. However, previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which doubles the inference cost per sample and significantly reduces efficiency. In this work, we theoretically reveal that the closed-form solution to the RL objective of self-verification training can be approximately reduced to a remarkably simple form: the true reasoning reward of a solution is equal to its last-token self-rewarding score, which is computed as the difference between the policy model's next-token log-probability assigned to any pre-specified token at the solution's last token and a pre-calculated constant, scaled by the KL coefficient. Based on this insight, we propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a Mean Squared Error (MSE) loss that aligns the last-token self-rewarding scores with the verifier-based reasoning rewards, and jointly optimizes the reasoning and self-rewarding capabilities of LLMs. The optimized self-rewarding scores serve as auxiliary reward signals in both training and testing to enhance model performance. Notably, our algorithm derives these scores from the predicted next-token probability distribution of the last solution token immediately after solution generation, thereby incurring only the minimal extra cost of at most one additional token inference. Experimental results show that our method not only improves the reasoning performance of the model also equips it with remarkable self-rewarding capability, thereby further boosting its inference-time scaling performance.

Published as a conference paper at ICLR 2026
LASER: REINFORCEMENT LEARNING WITH LAST-TOKEY SELF-REWARDING
Wenkai Yang' Weijie Liu", Ruobing Xie?, Yiju Guo', Lulu Wu", Saiyong Yang', Yankai Lin 'Gaoling School of Artificial Intelligence, Renmin University of China PLLM Department, Tencent (wenkaiyang, yankailin eerelllnneerruu. @ruc. edw...edd.co
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). To address the lack of verification signals at test time after RLVR, prior studies incorporate the training of model's self-verification capabilities into the standard RLVR process, thereby unifying reasoning and verification capabilities within single LLM. However, previous practice requires the LLM sequentially generate solutions and selffverifications using two separate prompt templates, which doubles the inference cost per sample and significantly reduces efficiency. this work, we theoretically reveal that the closed-form solution to the RL objective of self-verification training can be approximately reduced a remarkably simple form: the true reasoning reward of solution is equal to its last-token self- rewarding score, which computed as the difference between the policy model's s next-token log pogobabbbiyityy assigned to any pre-specified token at the solution's 's last token and a pre-calculated constant, scaled by the KL coefficient, Based on this insight, we propose LaSeR (Reinforcement Learning with Last-Token Self- Rewarding), an algorithm that simply augments the original RLVR loss with a Mean Squared Error (MSE) loss that aligns the last-token self-rewarding scores with the verifier-based reasoning rewards, and jointly optimizes the reasoning and self-rewarding capabilities of LLMs. The optimized self-rewarding scores serve as auxiliary reward signals in both training and testing to enhance model performance. Notably, our algorithm derives these scores from the predicted next- token probability distribution of the last solution token immediately after solution generation, thereby incurring only the minimal extra cost of at most one additional token inference. Experimental results show that our method not only improves the reasoning performance the model also equips it with remarkable self-rewarding capability, thereby further boosting its inference-time scaling performance
INTRODUCTION
In the past few years, Large Language Models (LLMs) (Achiam et al. [003 MetaAl 2024a Qwen
Team 2024 Liu et al. 2024a) have advanced significantly, excelling varios domains (Li et al.
2023 Wang et al. 2024b). However, they still face limitations in complex reasoning tasks (A-MOO)
2024a| OpenCompass 2025 Rein al. 2024 Jain al. 2025). Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has shown great promise in enhancing the complex reasoning abilities of LLMs, demonstrated by OpenAl ol (Jacch et al., 2024) and DeepSeek-RI (Guo
et al. 2025). By rewarding reasoning paths based on the consistency between final outcomes and ground answers through a deterministic verifier, RLVR incentivizes LLMMs to produce more deliberate reasoning chains while effectively mitigating the risk of reward hacking (Gao et al. 2023).
'Work done during internship at Tencent.
Conresponding author.
Code and models available

---

Published as a conference paper at ICLR 2026
Training Stage
Testing Stage
Responses
D
Prompt
D00
O -.,
Rollout
RYB
Prompt
Graf
xBp
A
Aa
CR -1)4 + =A
Verifier-based Advantages
Self-Rewarding Score
Self- Laff-Rewarding Advantages
Log-Prob. Prob. of Special Token
Vertfie-based Reward
EOS Taken
celf-Rewarding-baae Correctness
Figure I: The full illustration our method LaSeR. During training, our approach augments the standard RLVR process with an additional MSE loss between the verifier-based rewards (rw) and the last-token self-rewarding scores (r.), where Ts is the difference between the policy model's next-token bog-probabilities of pre-specified special token at the final response token and a pre-calculated constant Cee, scaled by the KL coefficient 3g. The optimized self-rewarding scores can serve as auxiliary reward signals in both training and testing to enhance model performance.
Despite its effectiveness, limitation standard RLVR is its inability to continue providing verifica- tion signals for model outputs in scenarios where ground truth answers unavailable, such as during test-time inference (Zuo et al. 2025). To address this, the standard approach has been to train an external verifier (Lightman et al. 2023 Snell et al. 2024 Zhang et al.) [022 Gao et al., 2024 Yang iet al. 2025b) to evaluate candidate solutions. More recently, in pursuit of endowing models with the ability for autonomous and continual self-improvement (Zuo al. 2025), several works (Sareen et al.) 2025 Liu et al., [225a Zha et al.) 2025 Jiang et al. 2025) have instead explored jointly optimizing the reasoning and verficccitinttion capabilities
single policy model within the RLVR framework. However, we argue that these methods have a major issue of inefficiency: the external verifier requires additional training on separate I.LM during or after reinforcement learning (RL); while joint optimization involves generating both solutions and self-verifications sequentially under two separate prompt templates, which doubles the per-sample inference cost and reduces generation efficiency.
In this work, we propose LaSeR (Reinforcement Learning with Lase-Token Self-Rewarding), lightweight and highly effective algorithm that achieves this goal, jointly optimizing reasoning and self- verification capabilities at nearly zero additional cost. Our core insight is that model's assessment in its own solution can be captured in the last token's 's predicted probability distribution. We first show theoretically that the RI. objective of self-verification has closed-form solution, where the true reasoning reward from the verifier is equal to the next-token log- tog-robbbbility ratio between the policy and reference models for pre-specified special token (an unused token like "<lvision crciisio_sstrreee start |>" that serves as the pre-defined ground truth for verifications on correct candidate solutions) the last response token, scaled by the KL coefficient. We refer to this scaled log- probability ratio as the last-token self-rewarding score. Furthermore, we observe that for randomly chosen special token, its predicted log probability under the reference model is practically a constant, small value across all problems and solutions (see Figure/Zand Figure B). This enables to simplify the self-rewarding score into remarkably simple form that depends only on the policy model's outputs and pre-calculated constant, making it exceptionally efficient compute.
Building on above analysis, we replace the explicit RL optimization for self-verification with simple Mean Squared Error (MSE) loss. As illustrated in Figure we train the model to align its last-token self-rewarding score with the true reasoning reward from the verifier. In specific, after the policy model generates the solution for each problem, calculate the last-token self-rewarding score based on its last token's next-token log-probabilit, for the pre-specified special token, and construct the corresponding MSE loss. This MSE objective is added directly to the standard RLVR loss, allowing
2

---

Published as a conference paper at ICLR 2026
for seamless joint optimization for both the reasoning and self-rewarding capabilities of the policy model. At both training and testing time, our method generates each candidate solution and computes the self-rewarding score in single forward pass, incurring the cost of at most one additional token inference with no extra generation required. This is significantly more efficient than prior approaches, which require a separate inference step. The optimized self- rewardingg scores can not only complement the original reasoning rewards during RLVR to further enhance training performance, but also be used at test time rank and weight solutions for more accurate answer aggregation.
We conduct experiments on both LLaMA3.2 | 2024b) and Qwen2.5 (Qwen Team 2024) architectures, including pre-trained, miidtrrineed and reiforredd variants, to demonstrate the effective- ness of our method in broader math reasoning tasks. Experimental results show that our methods not only effectively improve the reasoning performance of the policy model, but also allows its self-rewarding accuracy to reach a high level, thereby equipping the model with better confidence calibration of its own outputs and improving its inference-time scaling performance.
2 RELATED WORK
RLVR for LLM Reasoning Reinforcement Learning with Verifiable Rewards (RLVR), which sorely calculates binary rewards based on the final answers, has been shown be highly effective in enhanc- ing the reasoning capabilities of LLMs (Jaech et al., [024 Guo et al. 2025 Team et al. 2025b). Cur-
rent studies can be categorized into several directions, including but not limited (1) designing more
efficient and effective RLVR algorithms (Schulman et al. 2017 Shao et al. 2024 Yu et al. 2025a
Yue et al. 2025b Liu et al., 20256 Zheng et al. [025 Dai al. 2026), (2) extending RLVR to gen-
eral reasoning domain (Ma et al.
2025 Zhou et al. [025 Yu et al. 2025b) and agent scenarios (Wang)
al.
2025b Team et al. 2025a Dong et al. (225)) (3) collecting diverse verifiable datasets (Hu et al.
2025
He et al.) 2025 Liu & Zhang [025 Ma et al. 2025 Fan et al. (2025). ) and (4) analyzing the
mechanisms of RLVR (Mukherjee et al. 2025 Yue et al. 2025a Wen et al. 2025 Huan et al. 2025).
External Verifiers for LLM Reasoning Training external verifiers to identify the correctness of the LLM-ggnnrated solutions is an effective way to enhance the reasoning performance of LLMs in the in- ference time. External verifiers usually fall into two categories: (1) Scalar Reward Models: Outcome- supervised Reward Models (ORMs) (Cobbe et al. [021]] Yang et al. 2024) and Process- supervisd Reward Models (PRMs) (Lightman et al.] 2023 Wang al. [022a Skywork-ol 2024 Yuan et al., 2024) are two representative approaches. ORMs provide supervision by evaluating the final answer, while PRMs offer more fine-grained feedback by assessing the intermediate reasoning steps. (2) Gen- erative Verifiers: Recent studies have explored the potential of training LLMs to perform natural lan- guage critiques of reasoning solutions generated by the LLM generators, and then to judge their final outcomes (Zhang et al.) 2024 Gao et al.) [2024] Yang et al.) 2025b) Zhao et al. (025). This paradigm has demonstrated stronger verification performance than scalar reward models, as it enables the LLM verifier to conduct deliberate chain-of-thought reasoning before arriving at the final judgment.
Self- Verification for LLM Reasoning Several recent studies (Sareen et al., 2025 Liu et al. 2025a Zha et al. 2025 Jiang et al. 2025 Lu et al. 2025) aim to unify the roles of generator and verifier by equipping single policy model with self-verification capability. The trained self-verification capability can be used in both the RL training and inference-time scaling stages to enhance the model performance. However, these approaches require generating solutions and verificatiis sequentially during training and inference. In contrast, method derives the self-rewarding signal directly from the next-token probability distribution of the final token of the generated sequence, achieving a more efficient and effective unification of generation and seff-verification. We note that a recent study (Lee et al. 2025) also aims to obtain the self-verification result after producing the solution. As discussed in Section 3.4 and Appendix C our framework is more general and accommodates it as a special case.
3 METHODOLOGY
3.1 PRELIMINARIES
RL Objective We denote xg as the target policy model to be optimized, and मee as the reference model from which no is initialized. D is the query set, as is input and y is the generated response
3

---

Published as a conference paper at ICLR 2026
to æ. The standard optimization objective of RL is formalized
O Fe max E- D..-D.ye (- [r(x, y) (5 (Toll
(1)
where r(z, y) is reward function, DKL is the Kullback-Leibler (KL) divergence loss.
RLVR Recently, RLVR (Guo et al. 2025 Hu et al. 2025) has emerged as an effective paradigm for enhancing the reasoning capabilities of LLMs. In RLVR, the reward function typically chosen as a deterministic verifier re, such as a rule-based verifier, to evaluate whether the final extracted answer = y matches the ground-truth answer a", and produce binary feedback (e.g.. (0.1)). That is,
if a is semantically equivalent to a", 0 otherwise.
[
To (a,, y) = 1 (azaa)
Policy Gradient Method Policy Gradient (Sutton et al. 1998) is a widely adopted algorithm to optimize the objective of Eq. ① which updates the policy model via the estimated gradient
V VoO ne = (o Σ A. log (u y+t))))))))))))))
where A₂ is the advantage function measuring the relative value of the action a, (i.e., token ye) compared to the baseline value under state St (i.e., sequence (æ. yea)). In practice, A, can be estimated in various ways (Schulman et al.] 2017, Ahmadian et al.) (024)). For example, Group Relative Policy Optimization (GRPO) (Shao et al.] 2024) estimates the baseline value the average reward within a sampled group (y², "".""""" for the same problem, and computes the relative advantage for each token yf in sequence y' as
A (r' mean(r"...... ))/ts "or ry = (x.
Implicit Reward Previous studies (Rafailov et al. 2023
Peters & Schaal 2007 have identified that
the optimal solution to the objective Eq. [ satisfies that
( 3 logl mh (
+ 8 log (xx,
where Z(x)= Ey ( exp( 8 r. (r(((z (Ix y)) is partition function. Glog (y(()
is termed as the
(y/z)
implicit reward, which has been used in prior works (Mitchell et al. 2024 Liu et al. 2024b to analyze the behavioral shift induced by the alignment process.
3.2
LASER: REINFORCEMENT LEARNING WITH TOKEN
3.2.1
FORMAL FORMULATION
In training, ground-truth answers can be reliably used to determine the correctness of solutions. At test time, however, when ground-truth answers are unavailable, the use of verifiers becomes crucial for evaluating solution quality and providing feedback signals. To address this problem, in this work, we explore the promising paradigm of jointly optimizing the self-verification and reasoning capabilities of LLMs within the RLVR framework, thereby enabling them not only to produce high-quality reasoning paths but also to evaluate their outputs at test time.
According to Eq. 5). as Zxx) remains the same for all y, a straight-forward forward idea is to utilize the ze (o(w)
implicit reward ,3 Blg Tn [n)()))) as the indicator to rank different generations at test time. However, this approach has critical drawback: the absolute value of the implicit reward is length-biased, since the ma(y)) and (/() = B E, log Sta( im m.))))) increases proportionally with the response length. In reasoning tasks, the incorrect solutions are usually longer than the correct solutions (Hassid) al. 2025), making the implicit reward unreliable in evaluating solution correctness (see Ap- pendix D). Furthermore, disregarding (xx) and directly aligning the implicit reward with the true rea- soning reward during training degrades the policy model's generation ability (Cui al. 2025). since a fundamental gap (i.e., 3 log 2(z)) exists between the solution to RLVR and that to reward modeling.
absolute value of ,9 log
TT(y, (maa).)
In this work, we begin by formulating our approach from the RL objective of verification. Given a problem æ, and candidate solution y, the model is required to produce a verification z to identify the correctness of the solution: z ~ To y). Thus, the RL objective of verification can be written as
V.T max Ep- D. N R (-)))===o (p..)) [f(x, y, 2) B.L ( [To]|
(6)
To
4

---

Published as a conference paper at ICLR 2026
where Ng is the generator to solve the problem (can also be the target model Te itself in the verffiicriinatiion setting), f(z, y, x) is the verification reward that measures the consistency between the true correctness of y and the verification result of z. In practice. 2 be either single token-for instance, "Yes" or "No" to directly indicate whether the solution is verified as correct incorrect- sequence that includes both chain of thought and the final judgment. In this work, we focus on the former setting and simplify the ground label space to two single tokens Ze (e.g., "Yes") and 2; (e.g., "No"). That is, the verification reward can be formulated a
f(x, y. = {
(z = 2e and ro (a,, y) = 1) or (z = 2; and r. r (z. y) 0)
otherwise.
Similarly, following from Eq. (5) the close-form solution to Eq. (6 can be written as
f(x, v)) 3. log (=( [ze[= + B. log 2(,, y), Z( y) = Σ Tea(= y) exp( Be
f(x, v))). (8)
Now, let's take closer look at ((x 3). First, for 2 E fzer 2i), Teee( (2/2, y) is extremely small positive value for any problem-solution pair (æ, y), i.e., Trel 2 a/, y) O, for (Zc, 2. The reason is that the model is not specifically optimized for predicting the next token once it completes the generation and produces the final token (typically the "<EOS>" token). We present numerical analysis to validate this claim in Figure 7 and Figure 8 and we can see the value of Trer(2fe y) is less than 9 for common tokens and even less than for unused special tokens. Then, under an appropriate choice of Bo we can get
E Ter[=/2 v) exp( 1 3.
Z(x, y) = Σ Keel (2| y) exp( ((x,8,z))=
f(x, v(,)
8.
1 (1 11--I,,.ee I..(
+ nee (= 3) exp( (r.,))))) + )er( [2 3) exp( 3.
B
= (1 - 0 ex) 0= = log ( y) = 0.
The above analysis reveals that, under our formulation, the partition function can be naturally discarded. Consequently, the optimal solution to Eq. 6 can be approximately reduced to:
ar y, =) 3, log[ z| y) /ee (2| g)]
(10)
In partieular, the true verification reward when the model verifies solution as correct is:
f(x, y.z.y.20 2e) = (z y) B, logl To (zc/x, y) //e 20/xx, y)]
(11)
The first equation is derived from the definition Eq. 77). The second equation reveals that the true reasoning reward is equal to log-probability ratio of the policy model to the reference model at zes scaled by the KL coefficient. Thus, to optimize the model's verification capability, we do not need explicitly perform RLVR procedure. Instead,
directly optimize the following MSE loss:
L = Eg- D (+1x) (3, logl me (zela y)/ Tree( ze x. y) r. (x, y))
(12)
Thus, in the self- verification setting where mg me. we can directly adds the above loss into the origi- nal RLVR loss to jointly optimize the reasoning and self-verification capabilities of the policy model:
To (2(2)(=)))) (1.
( v) )]
S.g max Be- ro y) 3DKL ( Be log 49
(13)
TTr( ar,
where a is a loss balancing coefficient. We refer the term r, log m.) to the last-token self-rewarding score, since it depends on the log-probability distributions of the last token in y.
( a.))
3.3 OTHER TECHNIQUES
Here, we discuss several practical techniques to further simplify and improve the efficiency and effectiveness of the self-rewarding MSE loss introduced above.
In Appendix C
we further provide the derivations demonstrate the general form of our framework.
5

---

Published as a conference paper at ICLR 2026
Simplification of the Log-Probability in the Reference Model As illustrated in Figure and Figure [) the quantity log Tett (x, 3) remains almost constant and stable during policy model's training. Therefore, self-rewarding score during both training and inference. This eliminates the need for forwarding y through the reference model and thus further enhances efficiency. In specific, Gref is the mean value of log Ter &e x,, y) on small set of pre-generated set of (æ, y). Furthermore, based on the findings in Figure 7 we select an unused special token as ze make Tier( =e/er, y) closer to 0 and further minimize its impact on the approximation of Z y) = 1 and the stability of training.
regard it as a pre-calculated constant Get in calculating the last-token
Refa-Re Loss Re-Weighting During training, the numbers of correct and incorrect solutions are imbalanced, and their ratio dynamically changes. To prevent the last-token self- self-rewarding score from being biased toward the class with more samples, we apply a class-level loss re- re-weighting strategy within each optimization step. In each step, we calculate the total numbers of correct and incorrect solutions (identified by the verifier) for all problems in the current batch N₂ and N₁- Then, we apply the loss re-weighting
1
1 No + N;
ΣΣ We (r-( + wi (v)((.)))o)) [3. log to(=c/ Bo - To (1 y)), (14)
where We 2x and w; 2xN, re-weithting factors. This practice achieves more balanced self-verification capability. We provide empirical validations on this in Appendix []]
N.+NN
N.+N.
Integration of serifier-based and Self-Rewarding-aa based Advantages The last-token self-rewarding scores not only be used at test time, but also facilitate the training process through the integration advantages. We believe such practice can help mitigate of verifier-based and sefff-reardinn-basee
the issue of misjudgments by rule-based verifiers, which often occur when the format of ground-truth answer is overly complex, and produce more fine-grained rewards. For example, in GRPO, the final advantage can be calculated as:
A = (1 -T) mean(ri "r., std(rb, K))
r. mean(r......r R. std(r!, "0,,,,r,, rf)
(15)
where ry = (... y²) and r = B, log To (zc/orr))) y¹) Borae-
To stabilize training, we adopt a filtering strategy that sets T = 0 for any group whenever the standard deviation (1,,5, { within this group falls below threshold T, which is set to 0.1.
Separate Warm-Up of Reasoning and Relf-Rewarding Capabilities During the initial phase of training, we optimize only the last-token self-rewarding score, without integrating self-rewarding- based advantages into the learning process. After certain steps when the last-token self- self-rewarding loss is sufficiently small, we proceed to integrate verifier-based and self-rewarding -aase advantages. Moreover, when training from base (i.e., pre-trained) models, we first perform standard RLVR without incorporating the last-token self-rewarding loss in order to warm up the model's reasoning capability, followed by warm-up phase for the self-rewarding capability before the advantage integration.
By combining all the aforementioned techniques, our full algorithm Reinforcement Learning with Last-Toker Relfrdig (LaSeR), is summarized in Algorithm I and illustrated in Figure] T During the testing phase, once the model generates a solution, we compute the last-token self- rewarding score based r, B. log To [=elarr, y) Gy. We then clip it into the interval [0. 1]. Notably, we point out that this clipping is optional: after optimization, the self-rewarding scores naturally fall within [0, 1] (refer Table 7. further validating the boundedness of our self-scoring mechanism. The comparison between this score and 0.5 determines the self-verification outcome of the solution, the score itself can be further used to perform weighted majority voting.
3.4 BRIEF DISCUSSION
Comparison Between LaSeR and Prior Approaches Compared with previous methods (Sareen) ctt al., 20255) Liu et al. 2025a Zha et al., 2025) that requires the policy model to perform separate generations for solutions and verifications, our method directly derives the self-rewarding result from the next-token log-probatde probbabbiylyy of the final solution token. In the RL process, the computation of token pogaaabbabllitis is typically carried out after all the generations Therefore, we can directly replace the token id of the first padding token with the token id of the pre-specified token before computing the propabbbasllsiies of the sequences, thereby incurring
completed (Sheng et al.) 2024).

---

Published as a conference paper at ICLR 2026
additional computation cost during training. During inference, method requires only one more token inference after the solution is completed, which substantially reduces the computational cost compared to previous methods.
Difference Between Last- Token Self- Sell-Rewa Loss and Supervised Fine- Sine-Tuning Loss An alternative train the ceff-verification capability is to optimize the following SFT/BCE loss by maximizing the next-token probability of the token 2e or 2; based on the context (æ, y):
LSPT = - -. (, [re (2) y) logg To (= y) (1 (z. y)) log To [z y)
(16)
The major difference between SFT loss and our last-token self self-rewarding loss in Eq. (12) is that the SFT loss drives TT TT( y) to fit 1 when r. (z. y) 1, which may lead to strong interference with the optimization reasoning capability. In contrast, our loss drives To y) toward exp(1/3..) (e. y) for r,,( 1.0. When 3 is relatively large, no Relmr y) remains still very small, thereby exerting only negligible influence on the original RLVR optimization (e-g. To To (2.. = e 13 when Tret( (ze(< yy e- and B, 0.1). We provide further discussions in Appendix CanixxCannd and put
the empirical comparison in Appendix K
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Base Models and Baselines We conduct empirical validations on both LLaMA3.2 MetaAl (024b)
and Qwen2.5 (Qwen Team 2024 architectures, including three base models: OctoThinker-3B B--hor
Base
Wang et al. 2022a) (mid-trained version of LLaMA3.22 2-3.-----B 3B-Base), Qwen2.5-7B-Base (Qwen
Team 2024) (pre-trained model) and Open-Reaasnner-Zero-TT (Hu al. 2025) (reinforced version of In principle, our method can be seamlessly integrated into any RLVR framework, as it only introduces an additional MSE loss term. In the main experiments, we adopt the widely used GRPO (He et al. 2025) as the base algorithm and primarily investigate the effectiveness of applying our method within GRPO. We also conduct experiments with PPO (Schulman al. 2017 to demonstrate the generalizability of our method, and present the corresponding results in Appendix
Owen2.5-7B-Base
Training and Evaluation Datasets We adopt DeepMath- 103K (He et al.) 2025), large-scale and high-quality mathematical reasoning dataset, for our RL training data. In testing, we evaluate both the reasoning and self-verification performance of each model on five typical math reasoning benchmarks: MATH500 (Hendrycks al.) 2021), AMC23 (AI-MO) 2024b). AIME24 (AI-MO 2024a), AIME25 (OpenCompass] (2055) and OlympiadBench (He et al. 2024). Additionally, we also perform experiments in the general reasoning domain to validate the effectiveness of our method
in general reasoning tasks beyond math reasoning. The results and analysis are in Appendix R
Training Settings
The detailed training hyper-parametery of GRPO put in Appendix G The
prompt template for each model is in Appendix [T] When applying our method, we set the hyper- parameters (B,, 7) 10.1, 0.1, (refer Appendix I). ze selected "<vision.stee start>>" for
for DDwen..5-BB-aat and OpennRReasoner-Zero--1
and "<reserved_spede
The simplified constant of the reference log- bog-robabiliiy, Gut is -23.0 for Qwen2.55-B-BBoc and Seen--Ressnne--eee
OctoThinker-3B. Ber-Tinneee---hor---BBBe
and -2500 for More details are Appendix G
3B-Short-Baai as
estimated from the results in Figure 7
Evaluation Settings During generation, we set both the temperature and top -p to 1.0 for all models. The max www.geeeeennneeeeee is 8192. On MATH500 and OlympiadBench, we sample solutions for each problem; whereas on AMC23, AIME24, and AIME25, we sample 32 solutions per problem. We then report the average Pass @ accuracy of each model on each benchmark. We also evaluate the self-verification performance of each model by computing the self-verification Fl score, defined as the harmonic mean of self- verfficciiinttoon accuracy on self self-generated correct and incorrect solutions. Baselines perform self-verification based on the prompt in Appendix[[ T Any solution without final answer is automatically treated as incorrect and excluded from the verification accuracy calculation. Detailed self seff-verification accuracy results are provided in Appendix N
4.2 MAIN RESULTS AND ANALYSIS
We put the main results in Table | The key conclusion is that, across different model variants, our method not only yields better reasoning performance for the policy model compared with
7

---

Published as a conference paper at ICLR 2026
Table Reasoning and self-verification performance of each model on five mathematical reasoning benchmarks. We do not report the results of DctoThinkee-basee models on AIME24-25, as the number of correct solutions is quite insufficient for reliable evaluation.
Reasoning Accuracy
Self- Verification Fl Score
Method
MATH- AMC-
AIME
AIME-
Olym.- Bench
MATH- 500
AMC-
AIME-
AIME-
Olym.- Avg.
Avg.
500
23
24
25
23
24
25
Bench
OctoThinker-33 Baikker--BShoortt Short-Base
Base
3.7
1.3
1.0
2.0
22.3
11.2
13.7
15.7
GRPO
49.8
25.3
17.3
30.8
56.9
47.3
51.0
LaSeR
53.1
27.0
18.2
32.8
73.6
70.2
73.6
72.5
SWA
52.9
26.1
18.2
32.4
80.4
70.9
66.0
72.4
Qwen2.5-7B-BBal
Base
35.8
20.6
3.5
1.6
12.3
14.8
36.4
30.8
27.6
32,9
36.9
32.9
GRPO
79.9
55.9
16.2
13.8
43.3
41.8
54.6
59.7
36.6
41.5
53.5
49.2
LaSeR
80.2
58.1
15.4
15.7
42.7
83.2
82.5
79.6
-SWA
78.0
58.3
15.4
12.3
41.7
41.1
79.7
80.2
81.3
74.9
79.9
Open-Reasoner-Zeee Zero-7B
Base
81.9
60.3
15.6
15.1
46.9
44.0
26.7
51.3
45.9
55.2
37.5
43.3
GRPO
83.1
61.9
18.1
15.0
47.1
45.0
57.1
44.8
14,6
28.1
49.5
38.8
LaSeR
82.8
62.7
19.1
15.1
47.8
45.5
87.2
79.7
64.6
77.7
78.7
77.6
SWA
83.2
62.6
19.0
14.5
47.6
45.4
87.5
77.7
63.3
77.3
77.9
76.7
Table Comparison of verification Fl scores between LaSeR (self-rewarding) and external reward models responses generated by the same policy model Decmn-ReeacoeeeZen--BBBeaeeee Bold indicates the best result, and underline indicates the second-best
Method
MATH500
AMC23
AIME24
AIME25
Olym.
Avg.
Swen2.5-Math- -B-ath-BBBBB 78-PRM800K (7B RM)
56.3
42.5
51.4
50.8
38.5
47.9
Qwen2.5 1----MMa-M------ (7B RM)
86.0
79.6
70.8
67.3
76.0
75.9
1ree2..5Math-RMM-. RM-72B (72B RM)
86,8
79.4
71.0
71.4
75.5
76.8
SyennReeaomer-7cro07B.. (7B RM)
85.9
78.1
73.8
79.2
77.3
78.9
LaSeR (7B Self-Rewarding)
87.2
79.7
64.6
77.7
78.7
77.6
the baseline, but also substantially enhances its self-verification capability by enabling the self-rewarding scores to achieve remarkably high F1 scores.
Regarding reasoning performance, applying our algorithm leads to higher accuracy in most settings and consistently yields higher average accuracy on the three base models. We think there are two main reasons this improvement: (1) First, our method encourages the model encode assessment of the overall solution in the final response token, which leads to better confidence calibration. Improved calibration itself can have a positive impact on the model's learning. (2) Second, by integrating self- rewarding-bol based advantages into verifier-base advantages, advantage estimation, which in turn facilitates more effective learning. For comparison, we report the results without relf-rewa based advantages (-SWA) in Table Regarding self-verification performance, applying a simple last-token self-rewarding MSE loss substantially enhances the self- rewarding capability of the models, achieving around 80% Fl scores. This demonstrates strong self-verification accuracy on both correct and incorrect solutions. To further highlight the self- rewarding capabilities, we display the comparison results of verification Fl scores between LaSeR and several ad advanced external reward models Dowen2.5-Mann MaMaath-B-PPM---- PB-M-00O (Zhang et al. 2025). 1ren2.5-Math-PRMM--1
approach enables more fine-grained
(Zhang et al., 2025). and Qwen2.5-Mann Math-RM-722 (Yang et al. (0244) ))
evaluating the solutions generated by the different reinforced models by LaSeR. The full results are in Table 6. while here we only display the results on Reeen-Reasner-Zeet 7 in Table 2 Moreover, to ensure fairer comparison, we additionally train ORM using the same backbone 7B) and on the same data distribution (a total of 500K responses generated (Open- (Open-Reasoner- 7B RB-LASeR DeepMath- by Reasoner-Zeron
resulting in Open- Serasoner-Zero-7B The comparison results demonstrate the great effectiveness of self-rewarding. Moreover, it is worth noting that training and employing external reward models introduces additional computational
8

---

Published as a conference paper at ICLR 2026
88
86
Accumey
Ascumoy
81
Accumcy
56
85
35
-2
84-
Qoa HANg
83-
CRGTTB-L (RSGER)
21
2
2
2
2
2-
2
2
Numba of Sampled Solutions
Number Sampled Solutions
Number of Sampled Solutions
(a) Results OT-3B-hased models (b) Results of Qwen2.5--BBbban (c) Results ORZ-7B- mod- cls on MATH500
on MATH500
models on MATH500
Acarracy N6
49
48
22-
20-
%
QreR DAAAAAA
032-1--320 MAR3O
18
44
S0 SAREA)
16
21
23
2
27
2
22
2
2
25
Number Sampled Solutions
Number Smmlle Solutions
Number of Sampled Solutions
(d) Results of OT-3B-based based models on OlynpiadBench
(e) Results of Qwen2.5-7B B-sedd models on OlympiadBench
(I) Results ORZ- 7B-baee mod- els on OlympiadBench
Figure 2: The majority voting (Maj@ K) and weighted majority voting (RM@K K) results.
overhead during both training and inference. In contrast, method jointly optimizes the reasoning and self-rewarding capabilities within the RLVR framework and obtains the verification outcomes directly after solution generation, incurring extra computational cost at either training Furthermore, in Appendix (P] we show that the model optimized through LaSeR can even score the outputs of other models, demonstrating strong generalization in cross-model verification.
time.
4.3 INFERENCE-TIME SCALING RESULTT
Here, we explore the effectiveness of self-rewarding in the inference-time scaling via weighted majority voting. We compare majority voting results with (RM@) K) and without (Maj@ K) weighting by the last-token self-rewarding scores, on MATH500 and OlympiadBench. The results are shown in Figure 2 We denote the three base models by "OT-3B", "Qwen2.5-7B". and "ORZ-7B". The suffixes "-GRPO" and "-LaSeR" indicate the variants trained with GRPO and our method LaSeR, respectively. The results show that the optimized self-rewarding capability of the model is highly effective improving its own inference-time scaling performance.
5 ANALYSIS
5.1 MORE CALIBRATION ANALYSIS SELF-REWARDING SCORES
In this section, we conduct additional calibration analysis to better understand the properties of our self-rewarding scores. We take Open-Reasone Deasoner--Zero-lB-LLaes as the experimental model. To ensure reliability, we sample 32 times for each query in the following analysis.
We first visualize the distribution of optimized self-rewarding scores on OlympiadBench in Figure Furthermore, analyze the average AUROC score of the self-rewarding scores with respect to correctness. The AUROC defined as the probability that, within the same query group (solutions generated for the same query), correct solution receives higher self-rewarding score than an incorrect one. The full results are in Table 9 These results re-validate the high discriminative power of the self-rewarding score between correct and incorrect solutions.
Then, we calculate and report the Expected Calibration Error (ECE) all self-rewarding scores each test set. ECE quantifies the discrepancy between self-rewarding score and its empirical

---

Published as a conference paper at ICLR 2026
Propartio Soluttons
0.08
Orred Si boome Setraee
Ciltsihe Supple Coaa.
7000
Carre Siivkes
03-
Respunse Lengtt
6000
0.00
0.6
6200
0.4
5000
02-
2000
0.00
50
0.2
0.4
0.6
3.8
01
0.2
0.4
65
0.5
0.
02
0.4
0/6 0.5
1.0
Sell-Rewarding Seore
Self-Rowarding Sovre
Self-Rowanding Scure
Figure 3: Self-rewarding score
Figure 4: Reliability diagram of self-rewarding scores on tween response lengths and self- OlympiadBench.
Figure 5:
Relationship be-
distribution on OlympiadBench.
rewarding scores.
Table 3: Comparison of reasoning and self-verification performance with and without reference
pog-prbbability simplification in our method. Base model is Open- Reasoner-Zeroo
Reasoning Accuracy
Self Verification Score
Method
MATH-
AMC-
AIME-
AIME-
Olym-- Bench
MATH-
AMC-
AIME-
AIME-
Olym.- Bench
Avg.
Avg.
500
24
25
500
24
25
w/ Simpl.
82.5
61.6
18.8
16.2
46.5
45.1
82.3
79.3
77.9
79.2
78.4
79.4
w/o Simpl.
81.0
61.2
17.3
17.3
48.3
45.0
81.8
79.2
79.0
78.9
77.5
79.3
accuracy. The results on OlympiadBench are Figure while we put the full results in Table 10 As shown, our method demonstrates strong confidence calibration results.
Finally, we analyze the effect of response lengths. We visualize the relationship between response lengths and self-rewarding scores within a single chosen query group from OlympiadBench in Figure 5 as a preliminary analysis. Then, we calculate and report the average Spearman Rank Correlation between self-rewarding scores and response lengths over all query groups on each test set in Table These results reveal that shorter responses generally tend to receive higher self-rewarding scores, which is consistent with recent findings (Hassid et al. 2025 Yang al. 2025c that shorter CoTs on the same query are generally more preferable.
5.2 THE IMPACT OF SIMPLIFYING THE REFERENCE LOG-PROBABILITIES TO A CONSTANT
As discussed in Section 3.3 we approximate the log-probability of the pre-specified token under the reference model, log -et (2 x y). by its mean computed over small sample set when calculating the last-token self- self-rewarding scores. Here, we empirically validate this practice by conducting comparison experiments on Oppn-Reassoner-Zero-
with and without reference log-proot probbabbbity simplification in our method. We evaluate the checkpoint after 200 optimization steps in each setting (corresponding to the last checkpoint before advantage integration). The results are reported in Table 3 As shown, the simplification does not affect the optimization of reasoning and self- rewarding capabilities, since the performance under the two settings remains comparable. However, it effectively reduces the computational cost of calculating the last-token self-rewarding value by half.
CONCLUSION
In this work, we propose LaSeR, a lightweight and effective algorithm that jointly optimizes the reasoning and self-rewarding capabilities of LLMs. By deriving the closed-form solution to the RL objective of verification, we uncover a concise yet intriguing formula: the true reasoning reward provided by the verifier is equal to the last-token self-rewarding score produced by the model. This self-rewarding score depends on the model's next-token pogporbbbylity for a pre-specified token at the final response token, a pre-calculated constant, and the KL coefficient. Based on this insight, our method simply adds a MSE loss between the verifier-based reasoning rewards and the corresponding last-token self-rewarding scores into the standard RLVR process. The optimized relf-rewarding scores can not only be incorporated back into the RL process to further enhance training, but also achieve high verification accuracy test time, thereby improving solution ranking and selection.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work aims to jointly optimize the reasoning and self-rewarding capabilities of LLMs. Our method enables the LLMs self-verify the correctness of its solutions with minimal extra computational cost and high verification accuracy, improving both performance and efficiency. Our method has the positive impact the field of LLM reasoning by enabling LLMs to produce not only higher-quality and more interpretable solutions, but also accurate self- verifiicfiis....ss. All datasets and models used in this study are publicly available.
REPRODUCIBILITY STATEMENT
We open-source the code and release the model checkpoints. We provide the full experimental settings Section 4.1 and Appendix to facilitate reproducibility. We discuss all the techniques used our method in Section 5.2 Appendix B and Appendix ! We provide the ablation studies on hyper-parameters Appendix
parameters in Appendix All prompt templates used in our experiments are presented in
ACKNOWLEDGMENTS
We sincerely thank all the anonymous reviewers and (S)ACs for their careful reviews and constructive comments. This work was supported by The National Natural Science Foundation of China (No. 62376273 and No. U2436209), Beijing Nova Program (No. 20240484568) and Beijing Natural Science Foundation (L253001).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wwXiv:2303.08774 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 44704 2024.
AI-MO. Aime 2024.
2024a.
AI-MO. Ame 2023.
2024b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internlm2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAAFEDBBACK: Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint 2xxi:25002.01466. 2025.
Yanqi Dai, Yuxiang Ji. Xiao Zhang, Yong Wang, Xiangxiang Chu, and Zhiwu Lu. Harder is better: Boosting mathematical reasoning via difficult-awaree grpo and multi-aspect question reformulation. arXiv preprint 260:::2601.26111 2026.
11

---

Published as a conference paper at ICLR 2026
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint ar arXiv:2507719849 2025.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint WiX::2405.07860 2024.
Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812 2025.
Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, et Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. arXiv preprint arXiv:2406. 14024, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. pp. 10835-10866 PMLR, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12943 2025.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, al. Controllable preference optimization: Toward controllable multi-objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1437-1454, 2024.
Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. Don't overthink it. preferring shorter thinking chains for improved Ilm reasoning, arXiv preprint arXiv:2505.17813, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1828-3850, 2024.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerro: An open source approach to scaling up reinforcement learning on the base model. arXiv preprini 2rXi::2503.24290 2025.
Maggie Huan, Yuetai Li, Tuney Zheng. Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general I1m capabilities? understanding transferability of Il reasoning. arXiv preprint wrXivv2507..0443. 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprins 1rX1v:2412.16720. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http:////ooeeeeeeeeeeeeeeeeee/er
12

---

Published as a conference paper at ICLR 2026
Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan. Pag: Multi-turn reinforced Ilm self-correction with policy as generative verifier. arXiv preprint arXiv:2506. 2025.
Hyunseok Lee, Seunghyuk Oh, Jachyung Kim, Jinwoo Shin, and Jihoon Tack. Revise: Learning to refine at test-time via intrinsic self-verification. In Forty-second International Conference on Machine Learning, 2025.
Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. Pradient-adaptie policy optimization: Towards multi-objective alignment of large language models. arXiv preprint arXiv:2507.01915, 2025.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of foolloii models. hhtpp:://ii thub. wwt.:ccoocaaaaaaeeeeo httpr/ttttu-laaaapaaeeeeee 5 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 19X4::24/2.19437, 2024a.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov. Yejin Choi, and Noah A. Smith. Tuning language models by proxy. In First Conference on Language Modeling, 2024b. URL
https://openrevieet net:/forum?id-ddribbhnnt
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. 2025.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505. 13445, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Songshuo Lu, Hua Wang, Zhi Chen, and Yaohua Tang. Urpo: A unified reward & policy optimization framework for large language models. arXiv preprint arX1v:2507.17515, 2025.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing IIm reasoning across all domains. arXiv preprint 4rX., 2025.
MetaAl. Introducing llama 3.1: Our most capable models to date. https://ai.metar com/blog// meta-1lama-22 3-1/ 2024a.
MetaAl. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https
//al. //ai.meta. 3o-- 0ttp44//iiiiieeeeeeeooooooeeeeee 2024b.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttss://peerrvvie..rr foorum2id=E77vvol111
Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711, 2025.
OpenCompass. Aime 2025. http::/:/hnggiiitteeoooooossssts
2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
13

---

Published as a conference paper at ICLR 2026
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th interaational conference on Machine learning, pP- 745-750, 2007.
Qwen Team. Qwen2. 5: A party of foundation models. Qwen (Sept. 2024). url: https://wwenm.. github. io/blog/qwen2, 5, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-53741 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petry, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman, Gpqa: graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
Kusha Sareen, Morgane M Moss, Alessandro Sordoni, Rishabh Agarwal, and Arian Hosseini. Putting the value back in rl: Better test-fime scaling by unifying IIm reasoners with verifiers. arXiv preprint arXiv:2505.04844 04842, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning open language models. arXiv preprint arXiv:2402.033000 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 2409.19256, 2024.
Skywork-ol, Skywork-ol open series. https://uugiinfface.co/kkwwo November 2024. URL
https://huggingffee http://a.aaaa....coor.r.rrr co/Skyr
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033 04 2024.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: introduction, volume MIT press Cambridge, 1998.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with llms. arXiv preprint
2025b.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426-99 9426-9439, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 2024b.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXlv:2506.20512. 2025a.
Zihan Wang. Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding svol-tvo in Ilm agents via multi-turn reinforcement learning, arXiv preprint arXiv:2504.20773 2025b.
14

---

Published as a conference paper at ICLR 2026
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXiv:2506.14245, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu. Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mate technical report: Toward mathematical expert model via self- seff-improvement. arXiv preprint arXiv:2409. 12122, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint AXXx:2505.00388 2025a.
Wenkai Yang. Jingwen Chen, Yankai Lin, and Ji-Rong Wen. Deepcritic: Deliberate critique with large language models. arXiv preprint SKKv:2505500666 2025b.
Wenkai Yang. Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time compute for LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, /ttpa//////////////o//oo/ooo//oa
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025a.
Tianyu Yu. Bo Ji, Shouli Wang. Shu Yao, Zefan Wang. Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Rlpr: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506. 18254, 2025b.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint urXiv:2412.01981, 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. In The Thirteenth International Conference on Learning Representations, 2025. URL
net/
fforum?id-2ea5TNNVE
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025b.
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang- Wei Hong, Duane S Boning, and Dina Katabi. RI tango: Reinforcing generator and verifier together for language reasoning. arXiv preprint mwXiv:2505.15034 2025.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15244 2024.
Zhenru Zhang, Chujie Zheng. Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0730) 2025.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint 2xXv:2554.00891 2025.
15

---

Published as a conference paper at ICLR 2026
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.8807, 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang. Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493, 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang. Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
16

---

Published as a conference paper at ICLR 2026
A THE STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
In this work, we only use LLMs to correct grammatical issues and polish the writing. We do not use LLMs for research ideation or full paper writing.
B ADDITIONAL RELATED WORK
RLHF for LLMs Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al. 2022 has been a key technique for aligning a model's capabilities and behaviors with human preferences. The standard pipeline typically consists of three stages: supervised fine-tuning high-quality instructional data, learning reward model from pairwise human preference data, and optimizing the policy model via reinforcement learning to maximize the learned reward. Prior work in RLHF spans broad range of directions, including but not limited to: (1) Developing more effective or efficient RLHF algorithms and frameworks (Schulman al. 2017 AAhmadian et al. 2024 Rafailov et al. 2023); (2) Constructing diverse, large-scale, and high-quality preference datasets (Bai et al.
2022 Cui et al. 2024): (3) Training stronger or domain-specific reward models to improve RLHF
performance (Dong et al., 2024 Cai et al. 2024 Yuan et al. 2025); (4) Exploring multi-objective
alignment (Guo et al. 2024 Li et al.
2025)
C GENERAL FORM OF SELF-REWARDING MSE LOSS
In the main content, we derive the self-rewarding MSE loss in Eq. (22 by instantiating the verification reward in Eq. (7) under the binary 0/1 setting (though this is common choice). Here, make further derivations to obtain the general form of our self-rewarding MSE loss by allowing the rewards for correct and incorrect verifications take arbitrary real values, denoted Å. and Å. respectively. Then, we clarify why our method adopts the MSE loss instead of the BCE loss, and highlight that the solution the BCE loss Eq. (I6) is in fact special case of all available solutions in our framework.
Under the general definition, following the same derivation in Eq. we obtain Z(a,, y) exp( (,,,) which indicates log 2(x, y) f.. (still constant but not necessarily be 0). Therefore, log (xx, y) 0 in Eq. ( is special case when we define A; = 0. Then, the optimal solution to the original RI. target be reduced general form
f( 2) Po log[ (ol==l22,, v)/ Tee (z (z y)] A.
That is,
F(xr, y: 2) F; Bo [og[mo(z)ar g) (2/2, y-
Now, we can see that the left-hand side of the equation is no longer 0/1 binary reward but rather two arbitrary scalar values. Therefore, natural to model this objective using an MSE loss rather than BCE loss
Following above, the true verification reward when the model verifies solution as correct is To (zelz,
f(x, y. 2e) i fel.r..
= B, logl Ne
y)] + f..
Note that the reasoning rewards ro (2, y) defined in Eq. 2 can also be two arbitrary values, but we simplified them to {0. 1} and this will not affect our derivation. This formulation indicates that the probability T <x, y) alone is sufficient to model the reward score. In this general framework, our self-rewarding MSE loss is
L = E. D.y (1r (B log[[TT (2e v)/TTerr////teoeee (zelzz y) + P. (fotr, (mmg)- + (m))))
Now, we show that the solution to the SFT/BCE loss Eq. (16) is in fact special case of all available solutions in our framework. After optimization (i.e., the optimal solution to the MSE loss), we can see that when r. y)= (i.e., the solution is incorrect based on the deterministic verifier), To T=(2(22.)) y) T < B) which means the Te(2glxt y) remains unchanged (almost near zero, consistent with the solution to SFT loss when r, y) = 0). When r, (x. y) 1 (i.e., the solution is correct based on the deterministic verifier), our method drives 20(2. pr y) to fit
17

---

Published as a conference paper at ICLR 2026
Owez.
25
Ocwrteer
23.11 4G
Cumulative implici: Reward
600
20
400
Value log
11.99 1:22
10
9.13-9.13
Correct Solution
5-
1000
2000
3000
4000
5000
Token Index
Reference Rnd Specified Token ze
Figure 6:
Cumulative implicit reward values
The mean and standard deviation of log Tzer(Ze/z,, y) under different combinations of reference model Nee and pre-specified token & over 300 input-output pairs.
Figure 7:
across 32 reasoning trajectories sampled from
Open-Reasoner-Zero-- on an AIME2024 prob-
lem. Red lines correspond to wrong solutions and green lines correspond correct solutions.
Tee(( v) exp( (. As we can see, when 3, continues to decrease, the optimized probability (2(e2,,, 3) approaches I, which is exactly the solution to the BCE loss. However, in our framework. we can explicitly control the value of 3,, allowing <e(=<< y) to converge to any desired target. As shown in Appendix K this controllability provides greater flexibility compared with directly forcing To (r, y) toward 1, while also exerting much less interference on the optimization of the reasoning ability. Thus, our framework fundamentally different but more general method.
D THE LENGTH BIAS IN IMPLICIT REWARD
Here, we present the trend of the cumulative implicit reward values (log (.<(v.))) where Tee Qwwn2.5-7B-B.il across 32 reasoning trajectories sampled from Open- SpeenReassonr-Zert
AIME2024 problem, showing how they vary with the increasing trajectory lengths. As illustrated in Figure 6 the curves of all samples exhibit positive correlation between the implicit reward and the number of tokens, and longer trajectories tend to yield higher final implicit reward scores, indicating strong length bias in implicit reward. Since incorrect solutions are generally longer than correct ones in reasoning tasks (Hassid et al. (025),, implicit reward is therefore not a reliable indicator of the relative quality of reasoning paths at test time.
E STATISTICS OF log TRER (eur((2 [ y)
We present the mean and standard deviation of log ( xx y) computed over 300 input-output pairs. The reference model Wee chosen either Qwen2.5-7B-Base or Booohhherr-BB-horr--
and the evaluation is performed under two different choices of z, for each reference model (one common token and one unused special token): "Yes" and "<vision_se |>" for OctoThinker-331 3---hBrrt-Baaa Base. The results in
for Qwen2.5-7B-
Base, "Yes" and "<I pesserved.special..e Figure TigureFindices iindicates that log Terr( 2g 2x 3) remains nearly constant and extremely small, with only low standard deviation across different x and y. Thus, we can consider log )eer( = ar y) constant when calculating the last-token self-rewarding scores, which effectively reduces the computational cost by half.
Furthermore, the term log [ze y). the sequence y is generated by the policy model, whose parameters evolve during training. To ensure that our simplification continues to hold throughout the optimization process, we report the dynamics of both the mean and the standard deviation of -log ( y) over the course of training. Specifically, deviation of log (2, x, y) based on the 300 generated samples from the policy model at training steps 0, 100, 200. and 1000. We conduct evaluations by taking Dwen2.5-7BBBB BaBe the
calculate the mean and standard
18

---

Published as a conference paper at ICLR 2026
23.4
Value of -log Tee x,y)
23.3
23.2
23.1
23.0
22.9
22.8
0
100
200
300
400
500
600
700
800
900
1000
Training Steps
Figure 8: The dynamics of mean and standard deviation of log (zela y) during policy model's training.
Algorithm 1: LaSeR: Reinforcement Learning with
Self- Self-Rewarding
Input: Initial policy model TT. prompts D, verifier ru, warm-up hyper-parameters w, and War coefficient B., pre-specified token &e, pre-calculated Get E(z.y) (2e( y)]
for Step S., do
Set aold + :e:
2. Sample batch prompts D. from D;
Generate solutions {y | for each x D.:
4. Calculate verifier-basee based rewards and advantages (e.g., Eq. 44). calculate RL loss;
If 2 wr, calculate last-token self-rewarding loss based on Eq. (14 and add it to RL loss;
6. If 2 War. calculate self-rewarding-bor -baed advantages and perform advantage integration based on Eq. (15);
7. Update the policy model Tee using any RL algorithm with integrated loss and advantages;
end
Output: T0
base model and "wwvisio_sstareeeeeeee as the specified special token 2., The dynamics are presented (re((Zelae y) remains stable throughout the policy
in Figure As shown, the mean of - llg Tref( (e| model updates, which supports our motivation and confirms feasibility of approximating it as pre-computed constant in our method to improve efficiency.
F FULL ALGORITHM
We display the full procedure our method in Algorithm
G DETAILED TRAINING SETTINGS
use verl (Sheng et al. 2024) as our RL training framework. The basic training in both GRPO training and LaSeR training for each model are put in Table and the newly introduced training hyper- prrer-parameters for LaSeR are put in Table The number of optimization steps is 1000 for Qwen2.5-7B-B.o and Octo Thinker-BB-BBBB-- Shor-h-r-Ba Base, and 500 for Open- Rren-RResoner-Zero-- reasoning reward of 1.0 is given the final answer and the answer format are both correct; otherwise. is 0.0. In our method, the reasoning warm-up is performed for 1ree2.5-7B-Banl and OctoThinker
In RL,
19

---

Published as a conference paper at ICLR 2026
Table 4: Basic training hyper-parameters of both
Table 5: Unique training hyper-parameters of LaSeR.
GRPO and LaSeR.
Sypper-praamterr
Value
Hyper-parameter
Value
Train Batch Size
128
Coefficient 8r
0.1
Micro Batch Size
128
Loss Weight
0.1
Rollout
8
Self-Rewarding Adv. Weight
0.1
Maximum Prompt Length
2048
Reasoning Warm-Up Steps
200
Maximum Response Length
8192
Self Seff-Reearding Warm-Up Steps
200
Temperature
1.0
Top
1.0
LR
1 x 10
KL Coefficient
0.0
Table Comparison of verification Fl scores between LaSeR (self-rewarding) and external reward models (Qwen2.5-Math- 7B-Ph880KK, Qwen2.5-Math-PPRe and Qwen2.5-Math-RMM-25 on responses generated by different policy models.
Method
MATHS00
AMC23
AIME24
AIME25
Olym.
Avg.
Generator: Octo
-aan2.5-Mtt-7B-PPPPP (7B RM)
77.0
68.9
68.5
71.5
Qwen2.5- (7B RM)
64.1
69.5
Qwen2.5 -01.5-5-MM---M--222 (72B RM)
89.2
71.7
72.9
77.9
LaSeR (3B Self-Rewarding)
73.6
70.2
73.6
72.5
Generator: Qwen2.5-7B-Lae
-een2.5-Mtth-TB-PPPM
(7B RM)
59.4
52.7
58.8
53.8
52.0
55.3
Swen2.5-Math-PRM.. (7B RM)
82.5
79.2
75.1
72.3
77.8
77.4
Qwen2.5-Mard RM-72B (72B RM)
87.8
80.7
81.3
74.8
75.4
80.0
LaSeR (7B Self- Rewarding)
83.2
82.5
79.6
74.3
78.3
79.6
Generator:
Swen2.5-Math--i M--tth-----a PRMMOOK (7B RM)
56.3
42.5
51.4
50.8
38.5
47.9
Qwen2. 11---Matt-------- (7B RM)
86.0
79.6
70.8
67.3
76.0
75.9
Qwen2.5 1---5---t-MMM- RM-72B (72B RM)
86.8
79.4
71.0
71.4
75.5
76.8
LaSeR (7B Self-Ree Relf-Rewarding)
87.2
79.7
64.6
77.7
78.7
77.6
3B-Short-Base only, and the self-rewarding warm-up is perform for all models. The number of reasoning warm-up steps 200 for both Qwen2.5-7B-Bas and -hinker--n-n--hoee and the number of self-rewarding warm-up steps is 200 across all models.
H COMPARISON OF VERIFICATION PERFORMANCE BETWEEN LASER AND ADVANCED EXTERNAL VERIFIERS
Here, we display the comparison results of verification F1 scores between LaSeR and three advanced
external reward models Diven2.5-Math-7B-PPR... (Yang et al. 2024)) on evaluating the solutions 7B (Zhang et al. 2025), and Qwen2..5Matt-RRM
(Zhang et al. 2025), Qwen2.5-Math-
generated by the different reinforced models, i.e., Octo Sctokhinker-333 3B-Short-LaSed and O Reasoner- 7B-R-eRR,, The full results in Table 6 show that LaSeR outperforms equally sized -fute-of-the-ar external verifiers in assessing the model's 's solutions, and matches the verification performance of 72B reward model, demonstrating its non-tre trivial effectiveness in enhancing self-rewarding capability.
S0we2.5--BBLLSSee
20

---

Published as a conference paper at ICLR 2026
Baseline
Cucf. 0.1,
0.2, 0.5
Caef. 0.1, Loss Weight
Coef. 0.2, Loss a
Coef. = Loss Weight 0.1
0.75
0.70
Training Rewards
0.65
0.60
0.55
25
50
75
100
125
150
Optimization Step
(a) Training rewards with EMA smoothing
Training Self- Verificationon F1 Scores
0.8
0.6-
0.4
0.2
0
25
50
75
100
125
150
Optimization Step
(b) Training self verificaitnn scores with EMA smoothing
Figure 9: The curves of training rewards and training self-verification Fl scores under different combinations of
with EMA smoothing (EMA coef.=0.9).
I ABLATION STUDIES ON SELF-REWARDING HYPER-PARAMETER
Here, we display the curves (with Exponential Moving Average (EMA) smoothing) of training rewards and training self-vent self-verification Fl scores of our method under different choices of coefficient 3, and self-rewarre relf-rew MSE loss weight a. The experiments are conducted on Open Heam--Rasone--eeee 7B, which help to skip the reasoning warm-up phase compared with using Qwen2.5-7B. Soo-hhnnee--B-hott--ee
and while the results are similar in other two base models ater reasoning warm-up. The dynamics of training rewards and training self-verification Fl scores are displayed in Figure 9 As we can see, assigning larger weight to the last-token self-rewarding loss has more detrimental impact on the model's reasoning capabilities. On the other hand, the coefficient B, has little impact optimizing the self-rewarding scores, long as it remains within reasonable range (0.1 0.5). However, much smaller values of 3,, can impair the model's reasoning capability, For example, when B, 0.05, should
indicated by the analysis in the end of Section 3.4
21

---

Published as a conference paper at ICLR 2026
Coef. Bu 0.1, Loss Weight a' 0.1, wi Loss Reweighting
Coal' B, - 0.1, Loss Weight a 0.1, w/o Loss Reweighting
0.75
0.70
Training Rewards
0.65
0.60
0.55
25
75
100
125
200
Optimization Step
(a) Training rewards with EMA smoothing
Training Self- Selt-Veiifiaaiinn Scores
0.6
0.4-
0.2
0.0
0
25
50
75
100
125
150
175
200
Optimization Step
(b) Training self velffieriiination Fl scores with EMA smoothing
Figure 10: The curves of training rewards and training self-verification scores of our method with and without class-level loss re-weighting practice (EMA coef.=0.9).
have To (2cl2, y) -3 0.05 under [ze(< y) c and r. (x. y) = 1, then the large value of To (2,2 4) causes large interference with the optimization of reasoning capability. In main experiments, we choose (3,, (0.1.0.1).
J THE EFFECT OF CLASS-LEVEL RE- WEIGHTING ON THE BALANCED SELF- VERIFICATION CAPABILITY
We present the training dynamics of our method on Open-Reasoner-Zerut 7B, with and without class-level loss reightih,, in Figure [or comparison. As shown, applying loss re-whtttting leads to a more balanced self-verification performance by mitigating the bias toward the majority class with larger sample size, while still maintaining high reasoning accuracy.
22

---

Published as a conference paper at ICLR 2026
Last-Token Self Seff-Rewurding Loss, Coef. B. 0.1, Loss Weight == 0.1
Supervised Fine-Tuning Loss, Loss Weight a 0.1
0.75
0.70
Training Rewards
0.65
0.60
0.55
25
50
75
100
125
150
175
200
Optimization Step
(a) Training rewards with EMA smoothing
Seff-Rewarding /SrdinnggrTTTT Loss (log scale)
2
1
SFF
-1
-2
25
50
75
100
125
150
175
200
Optimization Step
(b) Self-rewarding/SFT loss curve on a log scale
Figure 11: The comparison of the training dynamics between the last-token self-rewarding loss and the SFT loss.
K COMPARISON BETWEEN LOKE-TOKEN SELF-REWARDING LOSS AND SUPERVISED FINE-TUNING Loss
Following the discussion in Section 3.4 we compare the training performance of our introduced last-token self-rewarding loss with the supervised fine-tuning (SFT) loss Open-Reasonne-Zeene
The training dynamics are shown Figure IT] As observed, applying the SFT loss to optimize the
self-rewarding capability causes substantial interference with the optimization of reasoning capability, leading to a marked degradation in training rewards. Moreover, the SFT loss degrades extremely slowly, indicating that directly driving (2, y) from 0 to for (. y) is inherently difficult. However, our method only requires fitting To 2 x. y) to exp(1/3.)) Tean 2, [x, y) for r, (xx y) 1, which is considerably easier and introduces much less interference.
23

---

Published as a conference paper at ICLR 2026
Mixed Advantages
Self-Rewarding-basee Advantages Only
self-rewarding warm-up
0.75
Training Rewards
0.65
0.60
0.55
0.50-
0.45
0
25
50
75
100
125
150
175
200
225
250
275
300
Optimization Step
Figure 12: The ablation of only using self-rewarding scores for RL.
Table 7: The the range between the minimum and maximum self-rewarding scores over all solutions each test set.
Model
MATH500
AMC23
AIME24
AIME25
OlympiadBench
[0.000, 1.010]
[-0.031, 1.001]
[-0.023, 0.858)
[-0.061, 0.722)
[-0.036, 1.013]
Owen2.5-7B-LaSoR
[-0.021, 1.031]
[0.013, 1.020]
[-0.052, 1.010)
[-0.041, 0.986]
[-0.056, 1.029]
ORZ-7B-LaSeR
[-0.005, 1.013]
[-0.085, 1.005]
[-0.035, 0.998]
[-0.031, 0.963)
[-0.036, 1.016]
L ABLATION OF ONLY USING SELF-REWARDING SCORES FOR RL
In our method, after self-rewarding warm-up. we we incorporate self-rewarding- based advantages into the verifier-based advantages provide fine-grained learning signals. Here, explore the effect of only using the self-rewarding scores for RL. Specifically, taking as the base model, we take the checkpoint after 200 training steps with our method the starting point. We then continue RL training using only the self self-rewarding score as the optimization signal reasoning ability (while still using the rule-based rewards to optimize the self-rewarding scores). The dynamics of training rewards are shown in Figure 12] We observe that after an additional 60 steps, the training collapses. This indicates that using only self-rewarding scores for RL easily leads to training instability, and should be complemented with the rule-based rewards.
M THE VALUE RANGE OF THE OPTIMIZED SELF-REWARDING SCORE
In Table show the the range between the minimum and maximum self reffrenwad scores over all solutions in each test set of each model. The results validate that after optimization, the self-rewarding score naturally falls within the target interval [0, 1].
N DETAILED SELF-VERIFICATION RESULTS
We report the detailed self-verification results of each model on self-generated solutions across all benchmarks in Table 8 including both overall accuracy and Fl score. Our method consistently yields significant improvements model's self-rewarding and velf-verification capabilities, while incurring only minimal additional computational cost.
24

---

Published as a conference paper at ICLR 2026
Table 8: Detailed self-verification results.
MATH500
AMC23
AIME24
AIME25
Olym.
Method
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Base
60.2
22.3
52.3
11.2
62.0
13.7
GRPO
58.2
56.9
66.7
47.3
66.4
48.8
LaSeR
77.0
73.6
77.3
70.2
80.3
73.6
-SWA
81.0
80.4
84.1
70.9
83.5
66.0
Qwen2.5-7B-Baat
Base
45.0
36.4
30.7
30.8
24.5
27.6
28.2
32.9
33.8
36.9
GRPO
76.5
54.6
61.1
59.7
60.4
36.6
72.5
41.5
54.6
53.5
LaSeR
88.0
83.2
81.5
82.5
92.2
79.6
90.5
74.3
79.5
78.3
87.8
79.7
79.6
80.2
94.3
81.3
92.2
74.9
83.9
83.3
Reen-RRasoner-Zero-
Base
79.6
26.7
66.6
51.3
39.6
45.9
47.6
55.2
55.2
37.5
GRPO
52.9
57.1
50.9
44.8
66.9
14.6
78.9
28.
54.7
49.5
LaSeR
90.1
87.2
77.7
79.7
87.2
64.6
92.8
77.7
80.1
78.7
.SWA
89.0
87.5
76.2
77.7
87.7
63.3
93.6
77.3
80.2
77.9
Table 9; The average AUROC of reward scores each benchmark. In calculation, discard query groups which all solutions are either correct or incorrect, as such cases provide no meaningful signal for assessing the effectiveness of the self-rewarding scores. To ensure reliability, we sample 32 times for each query. The generator is
Method
MATH500
AMC23
AIME24
AIME25
Olym.
Avg.
Qwen2.5-Math RM-72B (72B RM)
0.77
0.73
0.71
0.60
0.65
0.69
LaSeR (7B Self Rewarding)
0.72
0.68
0.77
0.74
0.67
0.72
Table 10: ECE of reward scores on each benchmark (lower better). To ensure reliability, we sample
32 times for each query. The generator is
Method
MATHS00
AMC23
AIME24
AIME25
Olym.
Avg.
Qwen2.5-Math 1.--5-MM------222 RM-72B (72B RM)
0.178
0.152
0,346
0.384
0.194
0.251
LaSeR (7B Self- Rewardirddd))
0.077
0.162
0.072
0.071
0.066
0.090
Table 11: The Spearman Rank Correlation between self-rewarding scores and response lengths on Been-RRasonee--eeo-7B----- To ensure reliability, we sample 32 times for each query.
MATH500
AMC23
AIME24
AIME25
Olym.
Avg.
Spearman Rank Corr.
-0.26
-0.44
-0.58
-0.52
-0.40
-0.42
O FULL RESULTS OF THE CALIBRATION ANALYSIS OF SELF-REWARDING SCORES
Here, we display the full results of the calibration analysis of redardiga scores made in Section We first display the results of average AUROC score (the probability that, within the same query group. a correct solution receives higher self-rewarding score than incorrect one) of the self-rewarding scores with respect to correctness in Table The results demonstrate the high discriminative power of the self-rewarding score between correct and incorrect solutions. The comparison results Expected Calibration Error (ECE, the discrepancy between self-rewarding score and its empirical accuracy) Table 10 demonstrate that our method achieves better confidence calibration. Finally, we show the average Spearman Rank Correlation (the direction and strength of the monotonic relationship between two variables by computing the correlation between their ranked values) between self-rewt rewarding scores and response lengths in Table The results reveal that shorter responses generally tend to
25

---

Published as a conference paper at ICLR 2026
Table 12: Cross-model verification results. Each row corresponds the average verification Fl score of given verifier, evaluated on solution sets generated by different generators, as indexed by the columns.
Generator
Verifier
OT-3B-Shorte Short-GRPO
Qwen2.5-7B-GRP. ORZ-7B-
UT-3B-Short-GRPe 51.0
42.7
43.6
Qwen2.5-7B-GRPO
56.4
49.2
46.4
ORZ-7B-LaSeR
69.2
77.5
77.6
Table 13: Results of applying our method within PPO framework on Open- Dpenn-Reasoner-Zeno-7Z.
Reasoning Accuracy
Self- Verification F1 Score
Method
MATH-
AMC-
AIME-
AIME-
Olym.- Bench
MATH-
AMC-
AIME-
AIME-
Olym.- Bench Avg.
Avg.
500
23
24
25
500
23
24
25
Base
81.9
60.3
15.6
15.1
46,9
44.0
26.7
51.3
45.9
55.2
37.5
43.3
PPO
82.1
58.6
16.7
14.8
47.0
43.8
51.8
43.7
14.7
15.4
40.5
33.2
LaSeR
82.9
61.2
15.4
15.7
47.9
44.6
85.6
80.0
76.1
81.6
78.3
80.3
receive higher self-rewarding scores, which consistent with recent findings (Hassid al. Yang et al. 2025c) that shorter CoTs on the same query are generally more preferable.
2025
P CROSS-MODEL VERIFICATION RESULTS
Here, we explore the cross-model verification performance. In specific, we compare the verification F1 scores three models Open-Reasoner-Zeere BB-LaSBR- evaluated -evaluuua on each other's responses. The results are in Table Each value the table represents the average F1 score across all five benchmarks. Surprisingly, we find that our method not only enables effective self-rewarding, but also achieves high accuracy and F1 when evaluating the CoTs generated by other models, demonstrating strong generalization ability.
Open-Reasoner-Zerro BRRPOO, and
Q RESULTS APPLYING LASER WITHIN PPO
Here, conduct additional experiments using PPO the RL algorithm and explore the potential of applying our method within the PPO framework to validate the generality of our method. We conduct experiments
Dieen-Reasone--Zeo-755 The overall reasoning and self-verification results presented in Table and the inference-time scaling results
Figure 13 We use "LaSeRPPO. PPO.. to denote our method based on the PPO framework. As shown, our method also achieves strong ef- fectiveness in enhancing the reasoning, self-verification, and inference- -imee scaling performance of the policy model within the PPO framework.
R THE GENERALIZABILITY OF LASER TO GENERAL REASONING DOMAIN
We conduct additional experiments to explore the generalizability of our method to general rea- soning domain. We use a filtered version (Yu al. 2025b) of Weblnstruut-verrffed dataset (Mal et al., 2025), and conduct RI experiments on 3wen3-4B Base (Yang et al.) 2025a). We use the general-verifinn 1.r- 5B model from Ma et al. (2055) as the model-based verifier and adopt GRPO as the RL algorithm. The basic training and testing hyper-parameter for experiments on WebInstruct- verified are the same as those in Table 4 and Table 5 while the number of optimization steps here is 800. The simplified constant of the reference log pogoarbbabylity Get is 23.0. For method, do not perform the ad advantage integration strategy here. The reason is that we observe the self-verification Fl score of our method during training is relatively low in the general reasoning setting (only between
26

---

Published as a conference paper at ICLR 2026
ORZ-7B-PPO (Maj@K)
ORZ-7B-LnSeR (Maj@K]
ORZ-7B-LnSeR (RM@K)
88
54
53
87
Accuracy (%)
ccuracy (%)
52
86
51
85
50
84
49
83
48
47
2
2
2
2
2
2
2
Number of Sampled Solutions
Number of Sampled Solutions
(a) Results MATH500
(b) Results OlympiadBench
Figure 13:
The majority voting (Maj@K and weighted majority voting (RM@K) results of (ORZ-7B-PPO) and Tpen--RRsssneeeeeeen--B--Lerr (ORZ-7B- LaSeR PPO, on MATH500 and OlympiadBench.
70
(pm0340-17711NN Qye2--p- DUPO Magg
6.08
Proportios of Solittoes
0.05
Propertion of Solutions
6.00
Accuracy (%)
60.
Oreac L.kk pouae
8:04
8.
55
0.03
6.04
8:22
6.0
6.00
45-
40
MMLU-Pou
Self-Rewarding
Rewa-RRnn Score
(a) Evaluation accuracy on MMLU- (b) Self-rewarding score distribution (c) Self rewardina score distribution on MMLU-Pro
Pro and PoQA--Damood
on
Figure 14: The generalizability of LaSeR on general reasoning tasks.
65% and 70%, and the relf-rewarding score distributions in the test sets shown in Figure 14(b) and and
Figure [4(c] also reveal this phenomenon). This leads to large noise in the seff-rewarding-base advantage estimation, and consequently, the integration of self relf-rewarding-baae performance degradation. We conduct evaluations two general reasoning benchmarks: MMLU- Pro (Wang et al., (0244) and each dataset for each model, and calculate both the average accuracy and the (weighted) majority voting accuracy.
advantages results in
(Rein et al.] (022).. We sample solutions per problem
observe that jointly optimizing the self- rewarding capability does not impact the model's general reasoning ability, allowing the policy model achieve comparable average reasoning accuracy the baseline. (2) However, as mentioned above, the optimized self-rewarding score on general reasoning tasks does not achieve the high accuracy seen math reasoning tasks. We can see that the self-rewarding score distributions for correct and incorrect solutions on MMLU-Pro exhibit certain overlap, and the distinction further diminishes on the more challenging benchmark GPQA-Diamons this: (a) The model's general reasoning ability is inherently weaker than its math reasoning ability, which limits the upper bound its self-rewarding capabilities in the general reasoning domain. (b) The model-based verifier used in the experiment seeneral-veriierrr 1. 5B) has limited verification ability, resulting in high noise in the reasoning rewards, which in turn affects the optimization of the self-rewarding capability. (3) Though not perfect, the optimized self-rewarding scores still provide useful signals during inference time, leading to better weighted majority voting results. To examine this effect, analyze the distribution of self-rewarding scores across solutions generated
We have several findings: (1) First, from Figure 14(a)
We speculate that two factors may contribute to
27

---

Published as a conference paper at ICLR 2026
Table 14: The average self-rewarding AUROC score of Owen3-4B-LaSeR assess how well the self-rewarding score separates correct from incorrect solutions. In calculation, we discard query groups which all solutions are either correct or incorrect, as such cases provide no meaningful signal for assessing the effectiveness of the self-rewarding scores. To ensure reliability, here, we sample 32 times for each query. For computational efficiency, we evaluate on 1000 samples (Yu et al. 2025b) from MMLU-Pro.
MMLU-Pro-1000
GPQA-D
AUROC
0.61
0.58
for the same query. We calculate and report the average AUROC score, which is the probability that correct solution sampled from the same query group (solutions generated for the same query) receives a higher self-rewarding score than an incorrect one. The results are reported in Table [1] We find that the AUROC scores around 0.6, indicating that within the same query group, the self-rewarding scores assigned to correct solutions are generally higher than those assigned to incorrect ones. This reveals that, though not perfect, the self-rewarding signal can still effectively support solution re-ranking, thereby improving the performance of weighted majority voting.
S FURTHER REDUCTION OR INCREASE OF SELF-REWARDING COST
In this section, we discuss two additional variants of LaSeR for future work. In the current method, we calculate the last-token rewarden score based on the next-token peg-rbbabilityy distribution of the "<EOS>" token, requiring one additional token inference compared with standard inference. One potential way to further reduce the inference cost of LaSeR is to derive the last-token self- rewarding score directly from the predicted log-probability of pre-specified token &, at the "<EOS>" token position. Specifically, let yr denote the "<EOS>" token in y. Then, the reduced last-token self-rewarding
can be defined T. 3, log o (Zelz,, MrT) Bo we have observed that
Fer(zz|an jer) remains nearly constant across (x, y) (e.g., approximately -28 for Qwen2.5-7BB Base). In this case, can achieve ideally zero additional inference cost for self-rewarding compared with standard generation by directly calculating the self-rewarding score from the log-probability distribution the "<EOS>" token position, without requiring any extra token inference. In theory, this works because setting a relatively large 3g still yields a small value of Te = yer) (e.g., To (2<l22r yer) -18 when B. 0.1 and Get -28), thereby allowing Te Te (<E0S>>az xx F<T) to still dominate the probability mass. However, although the probability is very low, we observe that the generator may still select 2e at the end the sequence few cases during training, which can adversely affect training stability as the generator continues to generate after 2ę- One srraight-forwant solution may be to set the sampling hyper-paun parameter top-p to value less than 1.0. Future work can further investigate advanced strategies to make the above adjustment more principled and robust.
While reducing the self-rewarding cost improves efficiency, an alternative is to increase the in- ference cost in exchange for stronger self-rewarding capability. That is, instead of com- puting the self-rewarding score based on the log-pro bog-probability distribution of a single token only, increase the number of additional inference tokens by calculating it over M tokens as rs Bo IM log To @ (Zelxx, 1x,00 y, 2e) M3.c.. It is promising direction for future research to
mh-1 times
explore whether inereasing the number of additional inference tokens can yield positive inference-time scaling effect for latent self-rewarding capability.
T PROMPT TEMPLATES
We show the training, evaluation and self-verification prompt templates used in our experiments in the end.
28

---

Published as a conference paper at ICLR 2026
Training and Evaluation Prompt Template for OctoThinkerr---hhorttBss
<hos token> conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
User: You must put your answer inside \boxed{] and Your final answer will be extracted automatically by the ] tag.
{question)
Assistant:
Training Prompt Template for Qwen2.5-7B-Base
tooe_token> A conversation between User and Assistant. The User asks question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.
User: You must put your answer inside <answer> </answer> tags, i.c., <answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed tag. This is the problem:
{question}
Assistant: <think>
Zero-Shot Evaluation Prompt Template for Qwen2.5-7B-Bass
jim start] >system
You are helpful assistant.< jim end]
jim start] >user {question)
Please reason step by step, and put your final answer within \boxed lim end] >
lim start/ >assistant
Training and Evaluation Prompt Template for Open--ReasonerzZerro-77
A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.
User: You must put your answer inside <answer> </answer> tags, i.e., <answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed(} tag. {question}
Assistant: <think>
Training and Evaluation Prompt Template for Qwen3-4B-BB
|imr_tart| >user
(question]
Please reason step by step, and put your final answer within Xboxed (.s lim end] lim_start] >assistant
29

---

Published as a conference paper at ICLR 2026
Prompt Template for Self-Verificeen Verificciin (Modified from Liu et (2025a)
Below you are presented with question and tentative response. Your task is evaluate the response and assign rating to the response based on the following clear criteria:
Rating Criteria:
1. Missing final answer, incorrect response with the wrong final answer: assign \boxed(0).
Correct response with the correct final answer: assign \boxed (1).
### Question Begin ###
{question]
### Question End ####
### Response Begin ### [response]
### Response End ####
First provide your evaluation process, then clearly state your final rating value enclosed in \boxed } at the end.
30


Reviews:
[{"review_id": "ZHauyzuurw", "reviewer": "Reviewer_CW7C", "summary": "LaSeR trains a policy to encode a self‑evaluation signal in the final step of its answer: a last‑token score that tracks whether the just‑generated solution is correct. Instead of running a separate verifier sequence, the method adds a tiny auxiliary loss so that this one‑token score matches a standard verifier’s binary reward during RLVR. Inference then needs only the policy’s next‑token distribution at the last position, which also supports inexpensive weighted voting across samples. Experiments on several math benchmarks show small but consistent accuracy gains over GRPO baselines and strong self‑verification F1.", "questions": "1. Reference‑free test time: How stable is the pre‑computed constant across model updates or decoding settings? Would a quick online calibration (or an ensemble of special tokens) reduce drift?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Elegant and practical idea. This method compresses self‑verification into a single last‑token readout, avoiding a second generation pass and adding almost no training or inference cost.\n\n2. Near‑zero overhead: this method requires no second pass, just a next‑token query; training-time cost is essentially free since token log‑probs are already computed.\n\n3. Empirical Justification: The method improves Pass@1 across multiple backbones and yields high self‑verification F1; the score boosts inference‑time scaling via simple weighted voting.\n\n4. The argument about length bias in cumulative implicit rewards is valid and demonstrated (Appendix B), aligning with new empirical results suggesting shorter chains often outperform longer ones.", "weaknesses": "1. Limited breadth: The evaluation centers on math; results on a general reasoning set show only modest separation. More domains (code, QA, open‑ended tasks) would strengthen the case.\n\n2. Calibration analysis of self-rewarding: the reward model here is essentially a binary classification model. F1 is a good metric but not comprehensive enough. It would strengthen the paper if the authors could include reliability diagrams / ECE / AUROC for the last‑token score vs. correctness, and analyze length effects (short vs. long CoT) in line with recent findings that shorter chains can be preferable.\n\n3. This work misses discussions of prior RLHF (not RLVR) papers that incorporates reward signals explicitly in RL training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ARR43PfWQ", "reviewer": "Reviewer_mU64", "summary": "The paper proposes LaSeR, a novel policy optimization method that enables a model to self-verify its own reasoning outputs efficiently by leveraging the next-token probability of a pre-specified special token as a self-rewarding score, thereby improving test-time scaling. The authors report improved reasoning accuracy, self-verification F1, and test-time scaling efficiency across multiple reasoning benchmarks.", "questions": "1. How does LaSeR perform under Best-of-N [5] evaluation without weighting?\n\n2. What is the actual inference-time gain (in token count or latency) compared to prior self-verification approaches?\n---\n[5] Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, with clear theoretical motivation and solid empirical validation.\n\n2. Extensive experiments across multiple reasoning benchmarks and ablation studies demonstrate robustness\n\n3. The proposed idea of last-token self-rewarding is conceptually simple but empirically effective.", "weaknesses": "1. The proposed idea overlaps with prior work such as ReVISE [1], which also learns to self-verify using a special-token confidence and applies it for weighted voting at test time. \n\n2. The comparison with external verifiers (in Appendix F) may not be fair without training the verifier on the same dataset and backbone.\n\n3. Although several components are introduced (re-weighting, warm-ups, advantage integration), ablation studies are insufficient to isolate their effects.\n\n4. Evaluation is limited to math reasoning; generalization to other domains (code [2,3], general-reasoning [4]) is unclear.\n---\n[1] Lee et al., Revise: Learning to refine at test-time via intrinsic self-verification, ICML 2025\\\n[2] Austin et al., MBPP: Program Synthesis with Large Language Models\\\n[3] HumanEval:Evaluating Large Language Models Trained on Code\\\n[4] AlpacaEval: An Automatic Evaluator of Instruction-following Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FHyEs9pHjX", "reviewer": "Reviewer_xmH7", "summary": "The paper presents **LaSeR**, a reinforcement learning algorithm that augments the standard RLVR objective with an additional Mean Squared Error (MSE) loss aligning last-token self-reward scores with verifier-based rewards, enabling joint optimization of reasoning and self-verification. By approximating the reference log-probability as a constant, applying class-balanced re-weighting, and integrating the self-reward into advantage estimation, LaSeR provides an efficient improvement over vanilla GRPO with minimal computational overhead. Experiments on multiple mathematical reasoning benchmarks and three base models show consistent gains in both reasoning accuracy and self-verification performance, as well as the effectiveness of self-verification on test-time scaling and its generalization to other domains.", "questions": "- How is the weighted majority voting implemented?\n- How are the special tokens chosen for representing correctness (e.g., `<vision_start>`)? Are they selected randomly?\n- Figure 6 shows that many correct solutions have self-reward scores below 0.5, even though the mean score for correct cases is higher than for incorrect ones. Does this suggest that the self-reward signal works better for re-ranking rather than binary verification? It would be helpful if the authors could clarify this behavior or provide more results regarding this point.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Theoretical grounding and originality:** The derivation of the self-reward term is well-grounded in established RL literature, with a clear and rigorous justification for approximating the partition term as zero using both theoretical analysis and empirical evidence. While related works exist, LaSeR presents a coherent and novel formulation of self-verification that links last-token modeling with verifier-based rewards.\n- **Efficiency and practicality:** LaSeR introduces minimal training and inference overhead compared to prior self-verification or RLVR methods. The approximation of the reference log-probability as a constant further simplifies implementation without sacrificing performance, making the approach highly practical for large-scale use. Additional training techniques grounded in empirical observation further enhance the method’s engineering value.\n- **Comprehensive experimentation:** The study conducts extensive experiments across three base models and five mathematical reasoning benchmarks, with an additional evaluation on a science dataset. Detailed ablations on hyperparameters and the constant reference log-probability assumption validate the design choices. Together, these experiments strengthen the reliability of the conclusions.", "weaknesses": "- **[Motivation]** The motivation could be more clearly articulated. The paper frames LaSeR mainly as an efficiency improvement over existing joint reasoning–verification training paradigms but does not sufficiently justify *why* integrating these two abilities within a single model is necessary, instead of using a separate generator–verifier system. A clearer discussion of the conceptual or practical advantages of joint training would strengthen the motivation and emphasize the broader significance of the approach.\n- **[Significance]** The reasoning accuracy gains over the GRPO baseline are modest, and the performance difference between LaSeR and LaSeR w/ SWA is small. This limits the overall perceived significance and makes it difficult to assess the impact of the SWA module.\n- **[Experimentation]** The experimental analysis is somewhat limited. For instance, the generalization evaluation on GPQA reports only inference-time scaling results, without presenting zero-shot reasoning or verification performance, making it difficult to fully assess out-of-domain robustness. The comparison between SFT and MSE reward curves also omits the final performance results, leaving the claimed advantage of MSE insufficiently supported. Furthermore, the paper does not investigate why reasoning ability improves under LaSeR or whether the auxiliary objective influences the model’s reasoning dynamics or solution style. A deeper analysis in these aspects would offer clearer insight into the mechanisms driving the observed improvements.\n- **[Presentation]** Section 3 could be more condensed, as some of the detailed derivations and techniques may be moved to the appendix to improve readability and maintain focus on the key contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m5xlaxw1Lh", "reviewer": "Reviewer_9vmN", "summary": "LaSeR proposes a way to make LLMs “self-reward” by showing that, under its RL objective, the verifier reward can be approximated by a last-token self-rewarding score, the KL-scaled difference between the policy’s next-token log-probability on a pre-specified token at the final solution token and a precomputed constant. It augments RLVR with a simple MSE loss aligning this score to verifier rewards and then uses the learned score as an auxiliary signal at train and test time. The experiment shows this method can improve model reasoning ability and self-verification ability for test-time scaling.", "questions": "- As the primary focus is to integrate with GRPO using verifiable reward, why MSE loss is used for Eq. 12 instead of BCE loss?\n- How is the weighted majority voting for RM@K?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Detailed analysis of the proposed approach, combining both theoretical and empirical analysis.\n- LaSeR is lightweight & efficient. Adding an MSE term aligning “last-token self-reward” to verifier rewards; computing the score costs at most one extra token. \n- Weighting majority-vote by the learned self-reward scores improves solution selection at test time.", "weaknesses": "- Potential issue in the partition function expansion (Eq. 9)\n  - The paper defines the verification reward as **one-hot** over the *ground-truth* label: reward = 1 **only** when the predicted token equals the correct label (e.g., ($z_+\\in{z_c,z_i}$) depending on the task’s truth), and 0 otherwise (Eq. 7). \n  - However, when expanding the partition function (Z(x,y)) in Eq. (8), Eq. (9), it adds both $(z_c)$ and $(z_i)$ terms with the factor $(\\exp(1/\\beta_v))$, treating *both* as rewarded. I think only the *true* label ($z_+$) should carry reward 1 while the other label $(z_-)$ gets reward 0. Then\n $  Z(x,y)=\\sum_{z\\notin{z_c,z_i}}\\pi_{\\text{ref}}(z|x,y) e^{0}+\\pi_{\\text{ref}}(z_+|x,y)e^{1/\\beta_v}+\\pi_{\\text{ref}}(z_-|x,y) e^{0}\n  =1+\\pi_{\\text{ref}}(z_+|x,y)\\big(e^{1/\\beta_v}-1\\big)$\n- 0.5 threshold on a non-probability score (L318). The score ($r_s=\\beta_v\\log\\pi_\\theta(z_c\\mid x,y)-\\beta_v c_{\\text{ref}}$) is a scaled/shifted log-probability (unbounded, not a probability). Thresholding it at **0.5** is arbitrary—changing ($\\beta_v$) or ($c_{\\text{ref}}$) shifts the decision boundary. It also ignores the competing label ($z_i$) even though verification is binary.\n- The experiment is limited. Only base model and GRPO compared, and some setup (Open-Reasoner-Zero-7B x AIME25) shows limited advantage or is just not better than baseline at all. As the margin is so small, the statistical interval should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vuskcyUSup", "reviewer": "Reviewer_Z5vT", "summary": "This paper introduces a self-verification calculation method. It expands on the implicit reward and provides an expression through approximation to eliminate the expensive partition function evaluation, enabling the model to estimate the reward through a simple single token probability. The derived expression simplifies reward estimation and allows the model to self-evaluate its reasoning confidence. Empirical results show improvements on mathematical reasoning benchmarks, particularly in F1 scores for self-verification tasks.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Approximating implicit rewards through single-token probability is elegant and efficient. Theoretically it is novel. The derivation can be extended to future works for self-verification since it is simple enough to apply.\n\n2. There are empirical performance improvements in self-verification F1 scores on math reasoning tasks. Table 1 F1 scores show clear improvements over the self-verification.", "weaknesses": "1. The key observation of negligible $log Z$ does not have a strict guarantee that it will be approximately 0 when the model weights are changed. It would be beneficial to add additional analysis like Figure 4 for the post training model.\n\n2. It is unclear why the self-reward is added on top of the true reward instead of being used directly for optimization. This design choice should be justified. The claim could be stronger to have an ablation study with only self-verification as the reward signal, separating it out from GRPO advantage.\n\n3. Equation 9 would benefit to have more explanations. \n\n4. Appendix F appears to evaluate only on the model’s own outputs. It would be informative to test cross-model generalization, for example, using Qwen2.5-7B-Base to verify outputs from Open-Reasoner-Zero-7B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)