Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models
Authors: Zizhuo Zhang
Jianing Zhu
Xinmu Ge
Zihua Zhao
Zhanke Zhou
Xuan Li
Xiao Feng
Jiangchao Yao
Bo Han
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model reasoning
self-supervised RL
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Co-rewarding, a novel self-supervised RL framework that improves training stability for large language model reasoning.

Abstract: While reinforcement learning with verifiable rewards (RLVR) is effective to improve the reasoning ability of large language models (LLMs), its reliance on human-annotated labels leads to the scaling up dilemma, especially for complex tasks. Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs, yet they frequently encounter the non-negligible training collapse issue, as the single-view supervision signal easily forms the self-consistent illusion, yielding the reward hacking. Inspired by the success of self-supervised learning, we propose \textit{Co-rewarding}, a novel self-supervised RL framework that improves training stability by seeking complementary supervision from another views. Specifically, we instantiate Co-rewarding in two ways: (1) \textit{Co-rewarding-I} is a data-side instantiation that derives reward signals from contrastive agreement across semantically analogous questions; and (2) \textit{Co-rewarding-II} is a model-side instantiation that maintains a slowly-updated reference teacher with pseudo labels to realize self-distillation. Intuitively, such instantiations introduce different levels of discrepancy to increase the difficulty of training collapse on trivial reasoning solutions. We also explore their orthogonally combined version to further boost the performance. Empirically, Co-rewarding exhibits stable training across various setups, and outperforms other self-rewarding baselines by $+3.31\%$ improvements on average on multiple mathematical reasoning benchmarks, especially by $+7.49\%$ on Llama-3.2-3B-Instruct. Notably, Co-rewarding reaches or even surpasses RLVR with ground-truth (GT) label in several cases, such as a Pass@$1$ of $94.01\%$ on GSM8K with Qwen3-8B-Base remarkably higher than GT. Our code is released at~\url{https://github.com/tmlr-group/Co-rewarding}.

Published as a conference paper at ICLR 2026
CO-REWARDING: STABLE SELF-SUPERVISED RL FOR ELICITING REASONING IN LARGE LANGUAGE MODELS
Zizhuo Zhang Jianing Zhu
Xinmu Ge Zihua Zhao
Zhanke Zhou
Xuan Li"
Xiao Feng'
Jiangchao Yao Bo Han
'MMRR Group, Department of Computer Science, Hong Kong Baptist University
Shanghai Innovation Institute *CMIC, Shanghai Jiao Tong University
fcszzzhang, cs jnzhu, cszkzhou, csxuanli, xiaofeng [ @comp hkbu. edu. hk bhanml @comp. hkbu. edu. hk, {g3rald, jtuszzh, Sunarker jtu. edu. cn
ABSTRACT
While reinforcement learning with verifiable rewards (RLVR) is effective to im- prove the reasoning ability of large language models (LLMs), its reliance on human- annotated labels leads the scaling up dilemma, especially for complex tasks. Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs, yet they frequently encounter the negliggeee training collapse issue, as the single-view supervision signal easily forms the self-consistent illusion, yielding the reward hacking. Inspired by the success of self-super learning, we propose Co-rewarding, novel self--uuprvvied RL framework that improves training stability by seeking complementary supervision from another views. Specifically, we instantiate Co-rewarding in two ways: (I) Co- rewarding-I is data-side instantiation that derives reward signals from contrastive agreement across semantically analogous questions; and (2) Co-rewarding- // is model-side instantiation that maintains a slowly-updated reference teacher with pseudo labels to realize self-distillation. Intuitively, such instantiations introduce different levels of discrepancy to increase the difficulty of training collapse on trivial reasoning solutions. We also explore their orthogonally combined version to further boost the performance. Empirically, Co-rewarding exhibits stable training across various setups, and outperforms other self-rewarding baselines by 4-3.31% % im- provements on average on multiple mathematical reasoning benchmarks, especially by +7.49% on Llama-3.2-3B Instruct. Notably, Co-rewarding reaches or even surpasses RLVR with ground-truth (GT) label in several cases, such as a Pass @ of 94.01% on GSM8K with Qwen3-8B- Base remarkably higher than GT. Our code is released at https: // //ghtttthuuuu com/ htm//mllrl. r- gro.g group/Co-rewardinn..
Qwen3-1.7B
Qwent-8i
ITT Brleed
Saping Seag
60
80
hougg Pho
XATT hudg no
60
40
60
Unand frees
Geaznd
Coreaarding
40
20
Self Cerántyy
Costecading .1
Co-areading
SelCCenairy
Eanopy
Eacopy
Mayany- Vertng
Majoniy Voning
0
20
50
150
100
200
300
Ske
Sup
Figure Performance overview. Reasoning comparison of Pass@1 value and validation curves. Our Co-rewarding achieves better and more stable (without collapse) training than other baselines.
INTRODUCTION
Large language models (LLMs) (Achiam et al., 2023; Dubey et al., 2024) have demonstrated remarkable general-purpose purpose capabilities in wide range of linguistic tasks (Li et al., 2026; Chen
"Equal contribution.
1 "ooresponndecee Bo Han (bhanml@comp.hiee wwtp://ccoopp...cc.co..oo and Jiangehao Yao (Sunarker Saamarke@@sjtu.cdd.oo edw..n...c....

---

Published as a conference paper at ICLR 2026
et al., 2026). To further elicit their reasoning ability in complex scenarios, reinforcement learning with verifiable rewards (RLVR) (Shao et al., 2024; Yu et al., 2025) is developed for post-training with externally verifiable signals like program execution results (Luo et al., 2025) mathematical equivalence (Shao et al., 2024). Despite the impressive improvement, the reliance on high-quality ground-truth (GT) labels of RLVR remains
major bottleneck (Ouyang et al., 2022; Bai et al., 2022) in the spirit of the scaling law, which subsequently motivates the emerging exploration of self-rewarding methods with unlabeled data (Zhao al., 2025b; Zuo al., 2025; Zhang et al., 2025c).
One prominent line of such label-free methods leverages the internal signals (e.g., entropy (Zhang al., 2025d; Prabhudesai al., 2025) and self-certainty (Zhao al., 2025b)) to strengthen the confidence of the model in reasoning. Another critical line seeks the answer-level consensus (Zuo et al., 2025; Shafayat et al., 2025) to construct pseudo labels as reward basis. while effective initially, these self-rewarding approaches frequently exhibit non-negligible training collapse (Zhang et al., 2025e) (indicated as right of Figure 1), which limits the scalability of such label-free training manners.
The collapse phenomenon stems from reward hacking (Laidlaw et al., 2025) under self-consistent illusion: the reward signal internally produced by the policy model from a single-view data perspective, which is easily trapped by trivial solutions along with training (see Figure 7). Specifically, for entropy- or certainty- based rewards, the policy model may concentrate probability mass on a small set of tokens and produce repetitive strings that minimize entropy or maximize self-certainty (Zhang et al., 2025e). And for consensus- based rewards, the policy model can converge to a consistent yet incorrect answer that attains high consensus across rollouts (Shafayat al., 2025). Overall, the policy model continually reduces uncertainty without sustained gains in correctness, inflating the reward but eroding exploration and diversity. It ultimately collapses when persistent hacking strategy emerges.
To this end, we introduce Co-rewarding, a self-supervised RL framework that seeks complementary supervision from another views, inspired by self supervised learning (Chen et al., 2020; Grill et al., 2020; Caron et al., 2021). Conceptually, one fundamental characteristic of self-rewarding methods lies on that supervision intertwined with current policy on single-view outputs, for which we propose to seek reasoning invariance across different views (see Figure 2). Specifically, we investigate two initiations of Co-rewarding: (1) Co-rewarding-I: a data-side initiation that constructs rewards via contrastive agreement across semantically analogous questions, each providing pseudo labels for the other; and (2) Co-rewarding-lf a model-side initiation that introduces an extra teacher with dynamically updated policy and provides stable -seudo-labels insulated from current online policy. Additionally, we also explore the combined instantiation, Co-rewarding-lo -III, which integrates data-side cross-supervision with model-side self-distillation to further boost the performance.
By introducing cross-view view supervision on data and decoupling the reward signal from the current policy, Co-rewarding effectively mitigates training collapse and yields stable self-supervised RL training. Extensive experiments across multiple datasets validate the stability and superiority of Co-rewarding, compared several recent baselines across several LLM families including Qwen3/2.5 and Llama. Notably, both Co-rewarding-I and -II reach or exceed training with ground-truth labels in several settings, such as achieving up 94.01% Pass@ on GSM8K. Our main contributions are
We introduce new perspective, from self-supervised learning, to elicit reasoning capability via another views of supervision, which prevents the model from training collapse (Section 3.1).
We propose Co-rewarding, a novel self-supervised RL framework that is initiated by the data and model sides to construct self-generate rewards promote stably reasoning elicitation (Section 3.2).
We empirically demonstrate the general effectiveness of Co-rewarding to achieve superior reasoning performance on L.Mss, and also present various ablation studies and further analyses (Section 4).
2 PRELIMINARY
Problem Setups. Given a LLM To parameterized by and a dataset D of question-answer answer pairs (z. a), the model generates a response y TT(- æ) autoregressively. Let y = (yI...., Ya). where each token is sampled as y ~ Tt( x, yer) given the generated prefix yer. We consider the LLM outputs stepbystep reasoning trace and final answer. A verifiable reward function r(a, y) compares
2

---

Published as a conference paper at ICLR 2026
the extracted answer ans(y) with the ground truth follows: If ans(y) is correct with answer a,
{
r(a, v)
If ans(s)) is incorrect with answer
Then, the general objective of training LLM for reasoning via RLVR (Shao et al., 2024; Yu et al., 2025) can be formulated with the policy model Hy as follows:
max ((z,))D, y~<o(=) (r( y) - 3 KL| (r) me)
where "ee is an initial reference policy, and / is a coefficient controlling the KL divergence to prevent excessive deviation from the reference model. Intuitively, the training target to maximize the reward in passing specific reasoning questions while maintaining the general capability of LLM.
Group Relative Policy Optimization (GRPO). practice, we adopt GRPO (Shao et al., 2024), widely used and representative optimization method for objective Eq. (2) that estimates the advantage by normalizing the reward across multiple sampled outputs for the same question. Specifically, for given question GRPO samples G outputs from the old policy Sold {y: (-1x() It then computes a reward for each output ys via deterministic reward function, forming group of rewards {r(a, |= to estimate the advantage A, as follows:
A, r(a, v/)) mean (r(a, 1))
())
Then, the target policy optimized by maximizing the advantage while ensuring the policy model remains close to the reference policy:
JGRPO (0)
(-11))
Ivcl
min (,,(())A (l.(())A,.ciii((cce(()) cipp(c..ccc(())1111-0 c) - -DK ( (Tol(
(4)
G
Ra(A)
where
(VViic (i.<e)
(Te) To(yi, Ir, vi)
log Teer ( 1,, Bi, 1. (5)
(0)
DxI. DxI (To
Tan (yr. xr, Mi, ct)
Trer(y Mi,ct)
[ Vi,<e)
Note that the clip(:; 1 - e, +) in Eq. (4) is used to ensure that updates do not deviate excessively
from the old policy by bounding the policy ratio between 1 and + in risk funetion R(Å). We also provide a comprehensive discussion on additional training variants for RLVR, such as DAPO (Yu al., 2025) and Dr. GRPO (Liu et al., 2025a), which we leave in Appendix A due space limits.
3 CO-REWARDING
In the following, we present Co-rewarding in detail, a novel self-supervised RL framework for LLM elicit the latent reasoning capability through the intuition of seeking complementary supervision.
3.1 CONCEPTUAL PHILOSOPHY: INVARIANCE BEYOND THE SINGLE- VIEW
At the core of self-rewarding methods lies a fundamental tension: signals from its own outputs, inevitably intertwining supervision with policy and risks collapse. True reasoning competence, however, cannot be reduced to the mere correctness of isolated answers. It should instead reflect invariance that extends beyond the single-view output for consistency. This calls for training signals that remain valid across different data views persist throughout the temporal evolution the model, providing a more reliable basis on which self-supervised RI. can rely. this aspect, stability arises from invariance that prevents reasoning against superficial variations in data and guides the model towards increasingly valid reasoning trajectories throughout training.
the model derives supervisory
This philosophy yields our Co-rewarding framework, whose core idea is ground self-supervised RL in invariance rather than the suspicious single-view feedback. We instantiate it in two orthogonal ways and one combined version: by enforcing amalogy-ivvariate by disentangling supervision through temporal invariance on the model side (Co-rewarding II), and by integrating both mechanisms in unified instantiation (Co-rewarding-III).
the data side (Co-rewarding-J
3

---

Published as a conference paper at ICLR 2026
Co Co-rewarding-fi Side
Question: 1+1=?
Question
Verieer
Label
Pollcy
Cross-effrence
Poceale
Quaedion y'
Veeifer
GT Answer: 2
(n) Single-view method Rowward Hasking
Co no-rwarrding-I1 II: Model Side
9og-i
Goll
2
Quarntin 2
Policy Model
Ro
Venterr
Objective
Unckts Madis
2964 Upstte
Traader Modil
Figure 2: Illustration of Co-rewarding framework: Unlike single-view methods that rely only on internal reward signal on original question (a), Co-rewarding introduces complementary supervision. On the data side (b), paraphrased questions yield pseudo-labels for cross-reference. On the model side (c), teacher model isolated from current policy provides stabilized pscudo-labels for updates.
3.2 TWo INITIATIONS OF CO-REWARDING FRAMEWORK
Co-rewarding-I: the Data Side. Inspired by contrastive learning, such SimCLR (Chen et al., 2020) and InfoNCE (Oord et al., 2018), where two views the same data are encouraged to have similar representations, hypothesize
analogy-invariane inductive property of LLMs in eliciting reasoning capacity: questions that share the same mathematical essence but differ in surface form (e.g.. via paraphrasing, background substitution, or reformatting) should elicit the comparably valid and similar reasoning results. This forms the foundation for a self-referential training signal: contrastive agreement among different question variants can serve as an optimization proxy. Co-rewarding- defines contrastive agreement as principle that aligns model reasoning outputs, treating consistent inter-view agreement as a signal for valid inference. This complements single-view self-rewarding strategies by introducing form of collective validity verification with broader input consideration.
Building upon the discussed contrastive agreement, we initiate our Co-rewarding-/ as illustrated in Figure 2. Formally, its learning objective can be formulated as follows based GRPO:
Jo-rewanding (0) E EzED Noo ( Ro + EurED'
(1 Ro (A)).
(6)
(P)
Inghnon
where the relative advantages are estimated by the cross-refereed supervision as follows:
Å rV,: v.) mean meean((((v,,uun v.)))) A r(v,, v() mewn((((yr)))) v))
(7)
srd((ru((s v)))
std([r(u,, 2))
Specifically, given set of original questions, we utilize the rephrased version that keeps the semantical cquivalence for the model to respond, and then collect the self celf-generated pscudo-labels based on the majority voting mechanism (Shafayat al., 2025) as follows supervise learning the counterparts,
+ arg max (ans(()) = ans(y+)), 3/ + max Y
1 Jans( (1/) ans(y*))
(8)
The overall pipeline can be viewed a dual-path structure with cross-referent process, it may also be compatible with other self- -eneraeed feedbacks (Wang et al., 2022) on the output-side information due the generality of the core idea. While in the current version, we choose the majority voting mechanism in the implementation for the empirical effectiveness and simplicity.
the reward shaping
summarize the pseudo code Co-rewarding-1 in Algorithm 1. Our contrastive objective operates on self-generater reasoning answers, encouraging the model to align its reasoning results different questions that share the similar semantic intent. Formally, for each input question, the signal of Co- fo-rewarding-1 increases when the model's output is consistent with the majority answer obtained from its analogical counterparts, and decreases when it diverges. This contrastive agreement promotes semantic invariance, implicitly increasing the difficulty of reaching trivial solutions to obtain the
4

---

Published as a conference paper at ICLR 2026
reward (e.g., achieving the arbitrary answers but consistent on single input) by involving data-side analogy. We leave a more intuitive case study in the Appendix D.12 to present the rephrased questions.
Co-rewarding-II: on the Model Side. On the data side, our Co-rewarding-1 provides complementary supervision by involving question analogy, while its pseudo-labels are still generated by the current online policy and may depend on rephrasing quality; consequently, supervision remains partially entangled with the policy. Inspired by self- or weakly supervised methods like the representative BOL (Grill al., 2020), DINO (Caron et al., 2021), and Co-teaching (Han et al., 2018), which share the common intuition of introducing an auxiliary network to provide supervision beyond the current model, we initiate Co-rewarding-J from another view of complementary supervision: a model-side strategy that sources pseudo-labels from a teacher reference, which disentangle the self-supervision reward from the online policy. To avoid the heavy cost of adding and maintaining another L.LM in training, Co-rewarding-I1 reuses the GRPO reference model as the teacher to generate the rollouts and produce pseudo-labels. In particular, the teacher is dynamically updated as an exponential moving average (EMA) of the student policy ensure pseudo-label quality improving as the policy improves.
Intuitively, we illustrate Co-rewarding-Il in Figure 2. Its learning objective can be formulated as:
Co-sowarding-nn (0) = E J(tt
(A(A)
(k) ZED ( Taa (112),
C j=
(k) Te (-1zz)
policy cnstent milioon
reforace teacher
where { are policy rollouts and {ū are reference teacher rollouts the k-th training step. and the estimated advantage R(A())) is computed follows:
1(4) v1) mean (r(v) (1) D.)e
((()
arg max g
Jans (7)) = ays(y))), (10)
std( (r(a( )))))
where the pseudo label (( is obtained via majority voting from reference rollouts, and the reference model is updated via EMA with the policy to play role of slowly updated teacher:
(k) (l=-1) 1) + (1 - (1)) Dau a (R) = (Oland Qstaa)
T
ref f
(k)
Feel
(1-a(1))
2
1 cos
K
(11)
where (8) E (0, 1) is the EMA weight, updated according to a cosine annealing schedule from Ostari to Oend, such that the teacher updated rapidly at the beginning and progressively more slowly, thereby evolving smoothly and remaining temporally decoupled from the current online policy.
We summarize the pseudo code of Co-rewarding-11 II in Algorithm 2. This design can be interpreted as a kind of self-distillation, in which slowly updated teacher supervises a faster-moving student. Such paradigm breaks the single-step on-policy feedback loop inherent existing self reffrrearding methods (Zhao et al., 2025b; Prabhudesai et al., 2025; Shafayat al.. 2025), raises the cost of exploiting trivial low-entropy shortcuts or spurious consensus, and offers a stable reward source without introducing an additional L.LM optimizer. In this way, it effectively overcomes reward hacking and prevents training collapse by implicitly seeking a temporal invariance for true reasoning.
Co-rewarding-III: Data-side + Model-side. Given that Co-rewarding- and Co-rewarding-Il II provide two complementary perspectives for constructing stable self- supervuuee signals, a natural exploration integrate both data-side supprviuenoi and model-side self-distillation into unified instanti- ation. We introduce Co-rewarding-Ill which leverages maloggyinnvarian question and its rephrased counterparts while producing pseudo-labels from the EMA-updated refer- ence teacher. Specifically, the teacher generates rollouts for both original and rephrased questions, and the resulting pseudo-label from one side is used to supervise the other. This combination further boosts the resistance of the reward signal to hacking, promoting more stable training dynamics.
between each original
Formally, its learning objective can be formulated as:
T rewandingg 111 (0) = E
Re (A (A()))
ZED fu
Taa (-12), ZED, policy stadent originad questits
(k)
U # (k) (-1x²)
vofocave teadee colloe foaas ceplussed quesioa
+ EE x' e'D,' fu
π (M) Bas
(11=),
S म net
(+1x))
Ro (A)))
(12)
ED,
policy soden oeloor from rephensed quesion
referrnce stacher nilies from original queaion
5

---

Published as a conference paper at ICLR 2026
Table 1: Main Results (%) of Co-rewarding and baselines trained on MATH. Cell background
colors indicate relative performance: darker colors denote better results within each model group.
Additional results of Qwen2.5-3B/77 and Qwen3-1.7B Base trained on MATH refer Table 7.
Training Set: MATH
Mathematiiss
Code
Instruction
Multi- Task
Methods
MATH500
GSMEK
AMC
A1ME24
LiveCode
CRUX
IFEval
MMLU-Pro
Qwen3-3B-Baw
Before
27.82
20,93
23.41
54.75
50.89
52.92
GT--Rwward (Shao et al., 2024)
82.6
87.26
54,22
17:15
30.52
63.25
52.78
57.11
Self-Certainty (Zhao 2025b)
80.74
50.75
15.73
27.20
64,38
50.98
54.17
Entropy (Prahlulesai et al., 2025)
80.2
87.19
49.54
15.63
29.38
62.00
51.81
54,86
Majority Voting (Shufayat al., 2025)
79.8
89.76
49.09
15.83
30.52
63.38
51.80
$6,93
Co-rewarding- (Ours)
81.2
93.70
51.20
15.10
30.81
66.00
55.79
59.95
Co Co-rewarding 1I (Ours)
80.8
92.42
53.46
14.48
30.23
62.83
60.70
57.50
Co-rewarding-11 (Ours)
54.07
63.75
53,69
Qwen3-4B-Banl
Before
21.08
38.88
46.43
47.23
GT-Reward (Shao et al., 2024)
78.6
89.76
51.20
15.00
26.07
55.38
47,80
53.96
Self-Cortainty (Zhao al., 2025b)
71.79
38.86
11:67
22.37
57.00
48.15
48.93
Entropy (Prabludessi et al., 2025)
77.0
88.10
47.44
10.94
25.59
52.88
50.44
49.90
Majority- Voting (Shafayat et al., 2025)
77.4
90.07
45.33
10.10
26.54
57.50
48.78
54.35
Co-rewarding-1 (Qurs)
91.28
50.35
Co Co-rewarding 11 (Ours)
78.0
88.86
45.93
12.17
26.25
55.00
$1.30
53.88
-Co-rewarding-11 (Ours)
90.75
48,80
12.71
26.16
56,00
49.23
53.08
Hama- 2.2.:..-------ttteeee
Before RT.
39.2
65.73
10.54
3.75
9.86
25.37
57.32
31.14
GT- Reward (Shao et al.,, 2024)
47.0
77.94
22.14
11.67
9.57
31.87
47.51
34.32
Self-Ceraintl (Zhao al., 2025b)
43.4
74.91
18.83
6.88
9.95
25.87
54.88
33.34
Entropy (Prabludesai et al., 2025)
43.4
66.19
20.18
6.56
11.66
24.62
54,70
33.52
Majority-Votie Voting (Shafayat et al., 2025)
46.8
78.77
20.48
9.27
11.00
31.25
47.96
33.18
Co-rewarding-1
-Co-rewarding-1l (Ours)
49.8
79.30
22.59
10.73
10.80
30.63
49.90
33.61
-Co-rewarding--l (Ours)
79.91
25.45
10.42
10.43
32.50
46.37
34.50
where the first term supervises the original question via pseudo labels generated from its rephrased counterpart, and the second term, symmetrically, supervises the rephrased question via pseudo labels generated from the original question. The estimated advantages Ro (A()) and R. (Á*(k) are computed in the similar way in Co-rewarding-1 and Co-rewarding-II1 The reference teacher is also updated via EMA, as Eq. (1I) in Co-rewarding-IL. The other formulations and pseudo code of Co-rewarding- III are supplemented in Appendix B.1 and Algorithm 3.
Remark 1. Overall, the two instantiations of Co-rewarding embody our core idea from different to provide cross supervision, while II employs
perspectives: I leverages data-side model-side dif-distillation to stabilize learning. Together, they reflect that stable self-supervised reasoning elicitation can emerge from both the diversity of data perspectives and the disentanglement of supervision signals. Co-rewarding-Ill further explores an orthogonally combined instantiation of these two sides. Moreover, Co-rewarding offers flexible framework, in which key components, such as pscudo-labeling strategies, data rephrasing techniques, teacher model update rules, and policy optimization, can be seamlessly substituted with other advanced approaches (Yu et al., 2025).
EXPERIMENTS
4.1
SETUPS
Backbone Models and Baselines. We employ a diverse set of LLMs from different families and scales in our experiments, including the Qwen2.5 series (Qwen2.5-3B/7B) (Qwen et al., 2025), the Qwen3 series D0wen3-1.7B/4B8BB -Baee) (Yang al., 2025), and the Llama3 series (Llama- 1.2-3B-Instruet (Meta, 2024). Beyond the vanilla GRPO that utilized the GT label for rewarding, we compare our Co-rewarding against several recent state-of-the-art (SoTA) self-reward reasoning approaches, denoted as Self-Certainty (Zhao al., 2025b), Entropy (Prabhudesai et al., 2025) and Majority Voting (Shafayat et al., 2025). The details all basclines are summarized in Appendix C.
Implementation Details. We implement our algorithms based on the VeRL framework (Sheng et al., 2024), and experiments are conducted on x H100-80GB GPUs. For our experiments, we totally use three training sets: MATH (Hendrycks et al., 2021) (7,500 questions), DAPO-14K (Yu et al., 2025) (en-version of DAPO-Male 17k, about 14. questions), and OpenRS (Dang & Ngo, 2025) (7,00 7,000 questions). During RI. training, a global batch size of 128, set the number of

---

Published as a conference paper at ICLR 2026
Table 2: Main Results (%) of Co-rewarding and baselines trained on DAPO-14K. Cell background colors indicate relative performance: darker colors denote better results within each model group.
Additional Results of Qwen3-88 8B-Base and Qwen3-4B-o Base trained on OpenRS refer to Table 8.
Training Set: DAPO-14K
Mathematies
Code
Instruction
Multi- Task
Methods
MATH500
GSMEK
AMC
A1ME24
LiveCode
CRUX
IFEval
MMLU-Pro
Qwen3-3B-Baw
Before RL
27.82
20.93
23.41
54.75
50.89
$2.92
GTT-Rwward (Shao et al., 2024)
86.6
87.19
61.75
24.58
30,52
63.75
53.11
60.27
Self-Certainty (Zhao al., 2025h)
77.63
49.85
15.00
27.77
60.75
50.58
54,24
Entropy (Prahludesai et al., 2025)
79.4
80.82
45.48
15.00
30.14
62.00
51.56
54.57
Majority Voting (Shufayat al., 2025)
78.6
91.66
50.00
11.25
30.33
61.62
51.54
55.65
Co-rewarding- (Ours)
78.4
88.02
51.20
11.88
29.38
62.50
50.17
55.39
Co Co-rewarding 1I (Ours)
80.6
94,01
54.37
16.35
31.66
67.12
53.31
59,83
Co-rewarding-11 (Ours)
92.27
53.77
32.70
66.75
55.85
Qwen3-4B-BB
Before
26.15
38.88
46,43
47.23
GT-Reward (Shao et al., 2024)
83.6
85, 14
$2.86
20.63
18.58
56.88
47,70
55,35
Self-Certainty (Zhao al., 2025b)
68.4
44.81
35.39
8.85
25.88
50.12
45.58
48.84
Entropy (Prabludessi et al., 2025)
76.6
82.79
43.37
12.81
26.35
50.75
48.20
50.22
Majority- Voting (Shafayat et al., 2025)
73.4
64.06
40,81
9.17
26.16
53.00
48.91
$1.06
Co-rewarding- (Qurs)
75.89
43.83
50.12
46.84
51.51
Co Co-rrwarding 11 (Ours)
77.8
91.89
48.49
14.27
26.64
54,87
48.90
52.83
-Co-rewarding-11 IIl (Ours)
79.2
90.45
48,95
15.10
27.58
54.87
50.30
54.79
Homa- 2.2--..-------ttteee
Before RT.
39.2
65.73
10.54
3.75
9.86
25.37
57.32
31.14
GT- Reward (Shao et al.,, 2024)
49.4
78.17
25.90
9.17
10.33
31.37
53.10
33.83
Self-Cerainll (Zhao al., 2025b)
42.4
74.71
17.32
4.79
11.18
28.38
54.50
33.51
Entropy (Prabludessi et al., 2025)
44.0
65.85
17.32
6.56
9.95
25.00
55.78
31.95
Majority- Voting (Shafayat et al., 2025)
42.8
70.96
17.62
8.74
10.14
29.50
54.07
32,95
Co-rewarding-1 (Ours)
70.58
20.93
27.25
53.04
-Co-rewarding-1 (Ours)
49.8
78.62
19.73
8.02
10.43
32.25
$1.92
34,46
-Co-rewarding-11 III (Ours)
48.6
76.95
21.84
9.86
30.50
49.92
34.01
Qweee3-3770-- Bare (MATI)
Qwve15-7B (MAAI))
Groad Tuch
Geced Prab Genrudhlp
80
60
Dosed
20-
150
150
110
Oocual Tec
2000
80
40
bor Sond
Oooad
Overd
Yiur
1000
20
Emtrayy
10
0
5cz
Figure 4: Reward (left) and response length (right) of on validation set. Top: Qwen3-1.78 Base Qwen3-8B-Ban and -lama-3.2-3B-Insttee
Figure 3: Performance curves comparison
trained on and Qwen2.5-7B trained on the MATH set. DAPO-14K. Due to different reward scale from other Bottom: Qwen3-8B- Base and Llama-3.2- methods, Entropy reward is plotted on the right y-axis 3n-tnrtt trained on the DAPO-14K set. of left panels, where the reward is the negative entropy.
rollouts to G = Ō 8 per question for Co-rewarding-L, II and III, and adopt AdamW with learning rephraa rate of x 10-5 In Co-rewarding-I and III, question rephrasing is performed by the open-source Qwen3-32B model. In Co-rewarding- and III, the EMA weight is scheduled from Osan 0.99 to Gend = 0.9999 using cosine annealing. More implementation details are reported in Appendix C.2.
Evaluation Details. To provide comprehensive evaluation of model capabilities, utilize a diverse set of benchmarks spanning mathematical reasoning, code generation, instrructionfoollwwe
and general multi-task abilities. Specifically: (1) Mathematical reasoning: MATH500 (Lightman et al., 2024), GSM8K (Cobbe et al., 2021), AMC (Li et al., 2024a), and AIME24 (Zhang & Math-AI, 2024). (2) Code generation: LiveCodeBench (Jain et al., 2025) release_v6 and CRUX (Gu et al., 2024). (3) dnstruction-folloiin and multi-task abilities: IFEval (Zhou et al., 2023b) and MMLU-Pro (Wang al., 2024). Additional evaluation details are provided in Appendix C.3.
7

---

Published as a conference paper at ICLR 2026
Far Dusenrted)
CS500X Pas3 (N2
AMC Aorgli (Ng)
80
80-
Gooual
Goord Cscowateng.l
Cacceating-1
Comeedy
40-
Edtryy
Max Velk
Kajcay Sarki
20
02 30--
100
Step
200
165
Step
Figure 5: Performance and Stability on GSM8K and AMC. The gains of Co-rewarding arise from its training stability, which supports continuous improvements throughout learning.
Table Ablation study of Co-rewarding, For Co-rewarding-L, ablations train only on original or rephrased data. For Co-rewarding- II, ablation removes EMA updates of the reference teacher.
Truining Set
Methods
MATT1500
GSM8K
AMC
AIME24
LiveCade
CRUX
IFEval
MML.U-Pr
Qwen3-ti--Base
Co-rewarding-1
Majotitt-Votting w/ Union
80.2
93.48
49.70
15.63
31.94
64.88
54.25
59.51
Majority Voting w/ Original
79.8
89.76
49.09
15.83
30.52
63.38
$1.80
56.93
Majoritt-VVoiing... w/ Rephrased
79.2
91.51
50.75
14.17
31.66
60.38
52.24
57.26
Co-rewarding-II
80.8
60.70
57.50
MATH
wwo Updating Reference
79.2
89.46
51.51
13.96
30.62
61.75
56.93
51.85
-aamo-3.2-38--matt
Co-rewarding-1
50.2
79:15
23,80
10.00
11.28
29.88
48.89
33,77
Majority- Settority-Vottnng. w/ Union
Majority-Vottee Voting w/ Original
46.8
78.77
20.48
9.27
11.00
47.96
33.18
Majority- Majtrity-Votting w/ Rephrased
44.0
78.85
21.23
8.85
10.04
17.25
47.84
33.72
Co-rewarding-t II
49,8
79.30
22.59
10.73
10.80
30.63
49.90
33.61
w/o Updating Reference
47.0
78.92
22.29
9.06
5.50
31,25
47.88
33.32
Qwen3-88-Buus
Co-rowarding-I1
80.6
94.01
54.37
16.35
31,66
67.12
53.31
$9.83
DAPO-14K
w/o Updating Reference
78.0
88 -0
$1.66
15.94
30.62
63.75
52:48
$8.01
Llamo-3.2-30 3.-t-t
Co-rowarding-It
49,8
19.73
32.25
51.92
34.46
w/o Updating Reference
45.0
76.72
17.9/
8.02
10.05
30.63
51.33
33.94
4.2
EXPERIMENTAL RESULTS
4.2.1
MAIN PERFORMANCE OF CO-REWARDING
Superior Performance of Co-rewarding over self-rewarding baselines. Table I and Table 2 report the experimental results trained on MATH and DAPO-14K, respectively. We observe that all three Co-rewarding instantiations (I, II, and III) occupy more darker cells in the tables, demonstrating stronger performance than other relf-rewarding So'T baselines. Specifically, Co-rewarding- achieves an average relative performance gain of +4.42% over the best baselines across four mathematical benchmarks and models in Table 1, while Co-rewarding-ll achieves larger average relative gain of + +12.90% in Table 2. Moreover, Co-rewarding- Ill achieves improvements on average of +7.1 1. 1% and 1.72% over Co-rewarding-1 and Co- Co-rewarding-I11 III, respectively, suggesting that integrating data-side cross-supervision with model-side self-distillation can further boost performance. Additional results on other training sets and LLMS are provided in Appendix D.1.
Surpassing GT-Reward on certain benchmarks. Surprisingly, we observe that all three Co- rewarding instantiations (I. and III) outperform GT-Reward in certain cases. For example, on GSMSK, they together achieve an average improvement of +2.77% over GT-Reward in Table 1, while Co-rewarding-Il further delivers larger gain of +5.44% in Table 2. Notably, Co-rewarding-Il reaches remarkably high Pass of 94.01 % with Qwen3-88 8B-Base. This may be because GSM8K relatively easier benchmark, where self-supervised RL is sufficient elicit latent reasoning abilities without GT labels. Additionally, Co-rewarding also shows advantages on coding benchmark CRUX in several cases, possibly due to distribution mismatch between training and evaluation data. This may offer opportunities for self-supervised methods to generalize on par with, or even surpass GT-supervised methods in some cases. These findings highlight the potential of self-supervised RL to elicit reasoning capabilities, particularly with Co-rewarding mitigating training collapse.
Code generalization with preserved general performance. Although trained solely on math- oriented datasets, the models show improvements coding benchmarks, suggesting cross- cross-domain generalization from math to code in self-supervised reasoning elicitation. Moreover, Co-rewarding preserves general instruction-followin and multi-task ability on MMLU-Pro and IFEyal. As shown
8

---

Published as a conference paper at ICLR 2026
Table 4: Detailed performance of MMLU-Pro with Qwen3-8B- Saan-8B-BBse ---e trained on DAPO-14K, More results refer to Appendix D0
MMLLU-Pro (Qwen3-60-Bacc
Methods
biology
bushness
chemistry
computer
ccomomies
health
history
R--aRe
77.96
70.85
60/42
61.95
71.33
51,44
Seff-Certiiny
75.73
58.05
50.53
56.83
Entropy
74.76
59.70
51.33
56.10
67.50 67,90
55.87
48.04
Majuniy- Vaning
75.32
61.47
54.24
58.29
69.67
58.20
49.34
Co rewarding I
76.85
61.22
53.45
59.02
66.82
55.62
48.29
Co-rewarding-1 -1
76.71
68.69
64.58
61.71
68.25
56.85
51.71
Methods
law
math
other
philosophy
physics
psychology
englncering
GT Reward
31.52
73.28
56.28
52.71
61.90
67.30
46.14
Self-Certiinty
63.06
51.63
51.73
Fatropy
78.97
51.51
42.32
Majoniy- Voting
31.16
61.62
52:27
48.
53.27
66.92
40.97
Figure 6: Pseudo label
Co-rewanding-
30.34
66.1
51.73
48.9 18,90
55
66.42
39.63
Co rewarding 11
31.16
72.17
52.49
52.10
03.21
68.17
47.16
accuracy comparison.
Question: is the Irreducible fraction value of 3
what the value of mt n!?
Correct Answer: 142
31
stepiity
To fual od Lets acuaidy de ecucoste dkep skop.
esprattion
Th valoe stees ecrecces aor ract
Find, let's singiily the Iavemest aettin
Stupally the faumussd frathor:
velrs la paklon,, ve soai sinpli givee rpprcudoo sty say and tiie fort the Sactie
3+3 Nan, sntttti th lock Into sxprosite:
Plrst, stoplity the inserment Beriiia:
Sinqiity the frortie::
Nest, ualiitary tark lais the franiiia:
Naw. wwwiiie Eis bask las exprnsion
31
New thto this back how lhe mypemddo::
3+
Nest. stmppity the fracter deaoeataoiee:
Indke
Nov, strpait to eepprectio
Stegiity the
Davikere th ae Majority- Moatrity-Votting
Ss, the cretaciia facki valu ot tae
oppoosionn a Taas, oi ands 21..
31
or al 129 33mm 122
Therofere 109 ooll--3., the velo
the anone EEE
The toction aboady is supplee Iore. becrase
Theeekers (1-- 33 143
vo la- 199 22-112 112
The talaccl 1..
X
Entropy wnd
X
Co-rewarding-I
Co-rewarding II
Figure 7: Case study: An example comparing the generations from Majority-Voiin Entropy, and our proposed Co- Co-rewarding. The results clearly reveal the reward hacking behavior exhibited by Majority- Voting and Entropy, while ours generate the correct answer. Full results refer to Appendix D.13.
in Table 4, Co-rewarding-Il outperforms other self-rewarding baselines in 12 of 14 MMLU-Pro categories, demonstrating that its gains do not come the expense of broader general- -omainn performance. More detailed results of MMLU-Pr and IFEval refer to Appendix 1.10 and D.11.
Importance of stability for performance gain. As shown in Table 2, self-rewarding baselines exhibit noticeably limited performance gain in certain cases, such as Self-Certainty with Swen3-4B-Base on GSM8K. Figure further reflects this by showing that baselines improve quickly the beginning but soon collapse on GSM8K and AMC, whereas Co-rewarding sustains steady progress. This collapse restricts the baselines to effective training on only small portion of the data, preventing further improvements with continued training. These observations underscore the importance of avoiding training collapse in superviied RL unlock further performance gains.
4.2.2 FURTHER ANALYSIS
Co-rewarding alleviates collapse and provides stable self-supervised RL. We use 5,000 questions from the MATH test split as a validation set to monitor training process. Figure 3 shows that all three self-rewarding baselines collapse on both MATH and DAPO-14K. Co-rewarding-I remains stable on MATH but still collapses on DAPO-14K, suggesting that its stability depends on the property of training data. A plausible explanation is that the questions in MATH may provide favorable conditions for promoting diverse rephrasing variability. which is beneficial for the effectiveness of contrastive agreement in Co-rewarding-I. More discussions are provided in Appendix D.7. In contrast, Co-rewarding-I1 consistently maintains stability across datasets, as its design decouples supervision from the online policy and thus breaks the entanglement between supervision and the policy itself.
Co-rewarding attempts to balance exploration-exploitatin Figure 4 shows reward and response length curves. Entropy and Majority- Vooing quickly reach the highest reward, indicating reward hacking rather than genuine reasoning improvement. In contrast, GT-Reward and Co-rewarding exhibit smoother, gradually increasing rewards, reflecting stable training. The response length curves further illustrate this difference: GewRew lengthens responses to explore correct reasoning paths; Majority-Volee collapses to short outputs, restricting exploration; and Entropy collapses its probability mass onto small set of tokens, repeatedly generating them until truncation. Co-

---

Published as a conference paper at ICLR 2026
rewarding instead maintains moderate response lengths throughout training, suggesting balanced exploration-exploitte trade-off. Additional curves for other LLMs are provided in Appendix D.2.
Each part contributes to Co-rewarding. Table 3 summarizes the ablations across two training sets. For Co-rewarding-L. we observe that it typically outperforms all three variants of Majority- -ajority-Voting: models trained only original questions, only rephrased questions, indicates that the crossssuperviiion between original and rephrased questions plays key role in mitigating training collapse, whereas simply adding more data does not resolve the inherent instability of single-view self-rewarding methods. Notably, training only on the original or rephrased data yields comparable results, reflecting that the quality of original and rephrased data is similar. For Go-rewarding-II II, removing the EMA update of the reference teacher model causes clear degradation, highlighting the necessity of teacher updates for improving pseudo-label quality.
their union. This
EMA is essential in Co-rewarding-II for improving pseudo-label quality. Figure 6 compares pseudo-label accuracy across Co-rewarding- II, "w/o Updating Reference", and Majority-Voting. Co-rewarding-Il steadily improves accuracy as training progresses, while "w/o Updating Reference" remains nearly flat, underscoring the role EMA updates in allowing the teacher to co-evolve with the policy and generate higher-quality pseudo labels. By contrast, Eajority-Vottin briefly improves but then collapses to near zero, evidencing reward hacking through consistent yet incorrect outputs.
Case Study of the model reasoning with different learning methods. Figure 7 provides concrete example to illustrate the qualitative difference between self-rewarding baselines and our Co-rewarding. Majority- Voting exhibits reward hacking by boxing incorrect answer "0" to pursue consensus, even though the reasoning steps are correct. Entropy produces repetitive outputs as its decoding probability distribution collapses onto narrow set of tokens during entropy minimization. In contrast, Co-rewarding generates coherent step-by-step reasoning and correctly boxes the final answer, showing its capacity to overcome reward hacking and elicit genuine reasoning. Full results are provided in Appendix 3.13 and additional case studies on code benchmark are discussed in Appendix D.14.
5 RELATED WORK
Reinforcement learning with verifiable reward (RLVR) has recently become a mainstream post- training paradigm for eliciting strong reasoning abilities in LLMs (Guo et al., 2025), achieving remarkably encouraging success particularly on mathematical (Shao et al., 2024) and coding (Luo et al., 2025) tasks. However, RLVR fundamentally depends on high-quality and annotated GT labels to supervise reward signals, which remains a major bottleneck for scalability under the spirit of the scaling laws. To break this limitation, recent efforts have explored RL without external reward from multiple perspectives. For instance, methods such TTRL (Zuo et al., 2025) and SRT (Shafayat et al., 2025) pursue self-consistency to generate pseudo labels for rewards, where agreement among multiple rollouts is treated as optimization objective. Additionally, another technical line such as EMPO (Zhang et al., 2025d), Intuitor (Zhao et al., 2025b) and RENT (Prabhudesai et al., 2025), enhances the L.M confidence by optimizing internal signals of reasoning, such entropy minimization or self-certainty maximization. Different from these studies, Co-rewarding focuses on mitigating inherent training collapse in existing methods and enables stable self-supervised RL training. More detailed discussions of related work are in Appendix A.
6 CONCLUSION
In this work, we introduced Co-rewarding, a self-supervised RL framework that elicits the reasoning capability of LLMs through complementary supervision. Unlike prior self-rewarding methods that entangle rewards with single-view outputs and risk collapse, Co-rewarding establishes stability by decoupling the reward signal from the current online policy with the single-view output. Specif- ically, Co-rewarding-1 leverages contrastive agreement across semantically analogous questions; Co-rewarding-Il employs a dynamically updated teacher to provide insulated pseudo-labels; and Co-rewarding-Ill combines the data-side cross-supervision from Co-rewarding-I and the model-side teacher- based pseudo labels from Co-rewarding-Il to further boost performance. Together, these designs construct cross-refeet reforrbbee reward signals without explicit labels, aligning RL with invariances in reasoning rather than the mere correctness of isolated outputs. We hope this work will inspire further exploration into self-supervised RI. for reasoning advance the development.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT ZZZ, JNZ, ZKZ, XL, XF and BH were supported by NSFC Major Research Plan No. 92570109.
ETHICS STATEMENT
This work complies with the Code of Ethics. It uses only publicly available datasets, involves no human sensitive data, and raises no foresecable risks related to privacy, security, faireess issues. The research is conducted solely for scientific advancement, with no conflicts interest.
REPRODUCIBILITY STATEMENT
We are committed to ensure the reproducibility of our proposed method. A detailed description of our approach is provided in the Co-rewarding Framework section, and the corresponding source code has been submitted in anonymous repository at https: : /gi thub. com/tmlzz monlr-gnnn groulppg Co-rewarding Both backbone models and datasets used in our work are publicly available. Furthermore, all parameters, hyper- pperr-parmettrs, and procedural steps required to reproduce results are thoroughly recorded in the Implementation Details. We believe that these components provide the community with details necessary to verify and reproduce work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877. 2023.
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effective- ness of entropy minimization in Ilm reasoning. arXiv preprint arXiv:2505. 15134, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0880 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi,
Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650 9660-9660, 2021.
Liang Chen, Xueting Han, Qizhou Wang, Bo Han, Jing Bai, Hinrich Schutze, and Kam-Fai Wong.
Eepo: cewpoaattionnnhanneee policy optimization via sample-ther then-forget. In The Fourteenth International Conference on Learning Representations, 2026.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168. 2021.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilss: What works and what doesn't 't arXiv preprint arXiv:2503.162 2025.

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 2024.
Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. Serl: Self-play reinforcement learning for large language models with limited data. arXiv preprint arXiv:2505. 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 18835108666. PMLR, 2023.
Laan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar- Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. International Conference Machine Learning, pp. (6568-16621. PMLR, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.12944 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 2eXx::2402.04792.. 2024.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, and Feng Liu. Trustworthy machine learning: From data to models. Foundations and Trends in Privacy and Security, 7(2-3):74-246, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS. 2021.
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262. 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06744 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025.
Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-
Wooung-WWonnno Woon On. Binary classifier optimiza- tion for large language model alignment. arXiv preprint arXiv:2404.0465: 2024.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. In ICLR, 2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024a.
Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, and Jiantao Zhou. Llm unlearn- ing with Ilm beliefs. In The Fourteenth International Conference on Learning Representations, 2026.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06335, 2025.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In ICML, 2024b.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2024.
Yen-Ting Lin, Di Jin, Tengyu Xu, Tianhao Wu, Sainbayar Sukhbaatar, Chen Zhu, Yun He, Yun-Nung Chen, Jason Weston, Yuandong Tian, et al. Step-kto: Optimizing mathematical reasoning through stepwise binary feedback. arXiv preprint 20070 2025a.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-baser reasoning models. arXiv preprint arXiv:2503.22342 2025b.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 225522, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1 -e-eeo--like training: A critical perspective. arXiv preprint arXiv:2503.20883, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785 2025b.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025.
AI Meta. Llama 3.2:
Revolutionizing edge ai and vision with open, customizable models. Meta Al Blog. Retrieved December, 20:2024, 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. In ICML, 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXin:1807.03745 2018.
2018.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Jing- Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong Kio Chen, Jiacheng Xu. Zongzhang Zhang, and Yang Yu. Language model self ielloimmpoveeentt by reinforcement learning contemplation. The Twelfth International Conference Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2026
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.2266/ 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. et al. Qwen2.5 technical report, 2025. URL org/abs/ 2412. 15115.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2023.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408.0331/ 033144 2024.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.1427. 2022.
Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Exploration versus exploitation in reinforce- ment learning: stochastic control approach. arXiv preprint arXiv:1812.01552,, 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- cry, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 171, 2022.
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with training example. arXiv preprint wrXiv:2504.2057,, 2025a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266-95290 2024.
Yue Wang, Qizhou Wang. Zizhuo Zhang, Ang Li, Gang Niu, Bo Han, and Masashi Sugiyama. What is preference optimization doing, how and why? Arxiv Preprint, 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit approoxxiatiion for sample-efficient rlhf. arXiv preprint arXiv:2405.2 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing 1lm reasoning with rule-based reinforcement learning, arXiv preprint arXiv:2502.14475 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of 1lm performance in machine translation. arXiv preprint arXiv:2401.088 084417, 2024.
14

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint XXi:25505.09388. 2025.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang. Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Shought-augment reasoning with large language models. In NeurIPS, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L.. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi. Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, and Qi Zhang. Two minds better than one: Collaborative reward modeling for IIm alignment. arXiv preprint arXiv:2505. 10597, 2025a.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint ar arXiv:2503. 20..:2293..9377 2025b.
Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang. Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.088 08745, 2025c.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv preprint arXiv:2504.05812 2025d.
Yanzhi Zhang, Zhaoxi Zhang. Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang. Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Il reasoning. arXiv preprint arXiv:2506. 17219, 2025e.
Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024.
Zizhuo Zhang, Qizhou Wang, Shanshan Ye, Jianing Zhu, Jiangchao Yao, Bo Han, and Masashi Sugiyama. Towards understanding valuable preference data for large language model alignment.
In The Fourteenth International Conference on Learning Representations, 2026.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.0333 033335,, 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng. Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint 1955, 2025b.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, al. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2023a.
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1- zero's 's aha moment" visual reasoning on a 2b non-sft model. arXiv preprint 0rXiv:2503.051.2. 2025a.
15

---

Published as a conference paper at ICLR 2026
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiing evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023b.
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In NeurIPS, 2024.
Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information? In ICML, 2025b.
Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint arXiv:2503.22165. 2025c.
Yuxin Zuo, Kaiyan Zhang. Li Sheng, Shang Qu, Ganqu Cui, Xuckai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
16

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
Here we clarify the usage of Large Language Models (LLMs) in this work. For the preparation of this paper, LLMs are limited to the role of a general purppose writing assistant and are not used for research ideation or core content generation. For research methodology. LLM is a core component of our proposed method. Specifically, we utilize the Qwen3-32B model to perform question rephrasing in Co-rewarding-l, which is thoroughly detailed in the Implementation Details section of the main paper. The authors take full responsibility for all content written under their name.
A RELATED WORK
Large Language Model Reasoning. LLMs have shown impressive performance on vast tasks that require reasoning, including solving mathematical problems, writing code, and answering logical questions. One of the key techniques that has improved LLM reasoning is Chain-of-Thought (CoT) prompting (Wei et al., 2022; Zhou et al., 2024; 2025b). CoT encourages the model to generate intermediate reasoning steps before producing the final answer (Zhou et al., 2025c), which has been shown to enhance performance on tasks like arithmetic, commonsense reasoning, and symbolic reasoning. Subsequent work has extended CoT by integrating it with various strategies, including compositional generalization (Zhou et al., 2023a; Khot et al., 2023) and employing structural reasoning approaches (Yao al., 2023a; Besta et al., 2024; Yang al., 2024). In addition, CoT serves as fundamental framework for techniques like fine-tuninig (Zelikman et al., 2022), argentic workflow (Yao al., 2023b), and paving the way for inference-time scaling (Snell al., 2024).
RL for Large Language Models. Several RL algorithms have been developed primarily for alignment tasks (Wang et al., 2025b: Zhang et al., 2026; 2025a). Specifically, DPO (Rafailov et al., 2023), CPO (Xu et al., 2024), and their variants (Li et al., 2024b; Guo et al., 2024; Munos et al., 2024; Hong et al., 2024; Xie et al., 2024) rely on preference pairs labeled by annotators. In contrast, KTO (Ethayarajh et al., 2024) and BCO (Jung et al., 2024) require only single binary label (like or dislike) for each output. Besides, PRM (Uesato al., 2022; Lightman et al., 2024) and Step-KTO (Lin al., 2025a) offer step-by-step guidance by incorporating feedback at each reasoning step rather than focusing solely on the final outputs. Recently, the follow-up work GRPO improves the optimization objective, e.g., DAPO (Yu et al., 2025), Dr. GRPO (Liu et al., 2025a), REINFORCE++ (Hu, 2025), CPPO (Lin et al., 2025b), and GPG (Chu et al., 2025). Another line of research generalizes GRPO to broader applications such multimodal reasoning (Zhou et al., 2025a; Huang al., 2025; Chu et al., 2025; Liu et al., 2025b; Zhang et al., 2025b) and logical reasoning (Xie et al., 2025).
RL without External Reward. RL methods have shown promising scaling capabilities to enhance the reasoning abilities of LLMS (Guo et al., 2025), yet they are often limited by the availability of training data for reward signals (Gao et al., 2023: Liu et al., 2023). Notably, Wang et al. (Wang et al., 2025a) demonstrate that RL can effectively bootstrap LLM reasoning with as little as a single training example, highlighting the potential to minimize or even eliminate reliance on external reward signals during training. Recent efforts leverage distinct strategies for reward assignment. For instance, SIRLC (Pang et al., 2024) and AZR (Zhao et al., 2025a) utilize an L.MM L.MM-as-the -as--a---ttttt the--hueeggeeee approach to assign rewards. In contrast, methods like SRT, TTRL, and their variants (Shafayat et al., 2025; Zuo et al., 2025; Fang et al., 2025; Zhang et al., 2025c) employ self-consiseency (Wang et al., 2022) to generate pseudo-rewards, reducing dependence on external annotations. Meanwhile, INTUITOR, RLSC, and RENT (Zhao et al., 2025b; Li et al., 2025: Prabhudesai et al., 2025) harness the internal confidence scores of LLMs intrinsic reward signals. Additionally, EMPO and its variants (Zhang et al., 2025d; Agarwal et al., 2025) promote reasoning by minimizing entropy of reasoning path, further diversifying the approaches incentivize robust L.MM performance (Han al., 2025).
17

---

Published as a conference paper at ICLR 2026
Algorithm I -o-rewarding--
1: Input: policy model ne. learning rate 77, training dataset D, rephrased training dataset D', total iterations K.
2: Output: trained policy model my.
3: for all iteration = 1, K do
4: Sample mini-batch inputs B C D and B' D
5: for all input question æ E B and x' E B' do
6:
Sample rollouts {y/ C
To x).
7:
Sample rollouts {3/ 1C" ~ TB x²).
8:
Obtain pscudo labels by Eq. (8).
9:
Estimate relative advantages by Eq. (7).
10:
Compute the objective by Eq. (6).
11:
Update t- 0 - VooJCorooarrinng
end for
13: end for
Algorithm 2 Co-rewarding-ll
1: Input: policy model no, learning rate training dataset D, total iterations K
2: Output: trained policy model IT..
3: for iteration & = 1.... K do
4: Sample mini-batch B _ D.
5: for all B do
6:
Sample rollouts {ys ( ~ π TBad (. x).
7:
Update the reference teacher by Eq. (11).
8:
Sample rollouts {ū π ref (- x).
(k))
Obtain pseudo label from (ū, G by Eq. (10).
9:
10:
Estimate the relative advantage by Eq. (10).
11:
Compute the objective by Eq. (9).
12:
Update <- 0 - o (o
13:
end for
end for
Algorithm 3 Co-rewarding-lll
1: Input: policy model Ty. learning rate 7, original training dataset D, rephrased training dataset D', total iterations K.
2: Output: trained policy model Ty.
3: for iteration & 1,... K do
4: Sample mini-batch inputs B C D and B' D'.
5: for all B and x' B' do
6: Sample rollouts [3a),_ ~ Taaa (. x) and (1/) ~ TBad (: x').
7:
Update the reference teacher by Eq. (11).
8:
Sample rollouts (ū, " π ref (-1x) ()) and (7,) ~ = (- x').
9:
Obtain pseudo label from (77) 16
and {ū,}
16 by Eq. (13) and Eq. (14).
10:
Estimate the relative advantages by Eq. (13) and Eq. (14).
11:
Compute the objective by Eq. (12).
12:
Update 0 + 0 - ( Co-rowarding.11 (0).
13:
end for
14:
end for
18

---

Published as a conference paper at ICLR 2026
B PSEUDO CODE OF CO-REWARDING
B.1 FORMULATION OF CO-REWARDING-III
The relative advantages (A and Ry (Atk)) are computed as:
1/00 r( vR) mean (r(T) 36 11)))
Dv
(((K)
arg max
ans (7) = ans(v=)], (13)
std (r( :W:){E)
r(ge (k)
vl) mean( (r (it 10
G 1 (an)(ú,) ans(y=)), (14)
A,(0
(()) arg
std((r( 1)
y
where the pseudo label (( is the majority-vote pseudo label obtained from reference rollouts on rephrased question, and (=(k) is the corresponding pseudo label obtained from reference rollouts on original question. The reference model slowly updated via EMA as in Eq. (11).
B.2 PSEUDO CODE
To intuitively present the pipeline of Co-rewarding, summarize the pseudo codes of Co- Co-rewarding- I, Co-rewarding-Il and Co-rewarding- III in Algorithm 1, Algorithm and Algorithm 3, respectively.
C ADDITIONAL EXPERIMENTAL DETAILS
C.I DETAILS OF BASELINES
We compare our proposed Co-rewarding- and against GT--reward and several recent state-of-the-art (SoTA) self-reward approaches:
GT-Reward (Shao al., 2024): Originally introduced by DeepSeek-R1 (Guo et al., 2025), GT-Reward supervises training using ground-truth (GT) answers, determining whether model rollouts are correct or not, to guide RL optimization.
Self-Certainty (Zhao al., 2025b): This method maximizes self-certainty, defined the KL- divergence between the uniform distribution and the model's 's decoding distribution, serving as reward encourage more confident predictions.
Entropy (Prabhudesai al., 2025): This method minimizes the entropy of the model's rollout distribution, using negative entropy as reward to maximize model confidence.
Majority- (Shafayat et al., 2025): By generating multiple rollouts per question, Majority- Voting selects the most frequent answer assuul-laaell to supervise training.
For all methods, we adopt the widely used GRPO as the policy optimization algorithm.
C.2 MORE IMPLEMENTATION DETAILS
The detailed training configurations are summarized in Table 5, and baseline methods are trained under the same setup for fairness. For the training system prompt, adopt the official default prompt provided by VeRL shown below:
Let's think step by step and output the final
within \boxed
In addition, the semantically analogical questions used in Co-rewarding-I are generated by Qwen3- 32B through a rewriting prompt. The exact rewriting instruction is provided as follows:
You are given math problem. Please rewrite using different. wording and a different real-world scenario, while keeping the underlying mathematical meaning and answer exactly the same.
https://gthhhhcc///ooenngnee
19

---

Published as a conference paper at ICLR 2026
Table 5: Detailed training settings.
Settings
Co-rewarding-1
Co-rewarding-ll
Co-rewarding-lll
Batch Size
128
128
128
Max Prompt Length
512
512
512
Max Response Length
3,072
3,072
3,072
Train Steps
170-220 (MATH), 300-330 (DAPO-14K), 100-130 (Open-RS)
Learning Rate
3e-6
3e-6
3e-6
# Policy Rollout G
8
8
# Reference Rollout &
8
Clip Ratio
0.2
0.2
0.2
Warmup Style
Cosine
Cosine
Cosine
Warmup Steps Ratio
0.1
0.1
0.1
KL Loss Coefficient
0.005
0.001
0.001
Optimizer
AdamW (61 = 0.9, 32 = 0.999,
10-8)
Training Temperature
1.0
1.0
1.0
Evaluation Temperature
0.8
0.8
0.8
EMA Osstrt
0.99
0.99
EMA Crend
0.9999
0.9999
Table 6: Statistics and usages datasets used in our experiments.
Dataset Name
# Data Size
Usage
MATH-Train (Hendrycks et al., 2021)
7,500
Training Set
MATH-Test (Hendrycks et al., 2021)
5,000
Validation Set
DAPO-14K (Yu et al., 2025)
14,109
Training Set
Open-RS (Dang & Ngo, 2025)
7,000
Training Set
MATH500 (Lightman al., 2024)
500
Evaluation Benchmark
GSM8K (Cobbe et al., 2021)
1,319
Evaluation Benchmark
AMC (Li et al., 2024a)
83
Evaluation Benchmark
LiveCodeBench (Jain et al., 2025)
1,055
Evaluation Benchmark
CRUX (Gu al., 2024)
800
Evaluation Benchmark
MMLU-Pro (Wang et al., 2024)
12,032
Evaluation Benchmark
IFEval (Zhou al., 2023b)
541
Evaluation Benchmark
Guidelines:
1. Do not change the math logic or the final answer.
2. Use different words and new centext to make it look like different problem.
3. Avoid copying phrases or sentence structures from the original.
4. Make sure the rewritten questicn is natural, clear, and solvable.
5. Output ONLY between the following markers, and strictly in this format
(no extra explanation):
### RESULT START
ORIGINAL:
<original question>
REWRITE:
crewritten question>
### RESULT_END
20

---

Published as a conference paper at ICLR 2026
Table 7: Supplement Results (%) of Co-rewarding and baselines trained on MATH. Cell background colors: darker colors denote better results within each model group.
Training Set: MATH
Mathematies
Code
Instruction
Multi- Tast
Methods
MATH500
GSM8K
AMC
AIME24
LiveCode
CRUX
IFEval
MML.U-Pro
Qwen2.5-3B
Before RL
19.48
10.69
9.95
18.50
29.83
GT-Reward (Shuo et al., 2024)
65.4
82.18
12,98
6.77
13.93
32.12
33.66
36.74
Certainty (Zhao al., 2025b)
32.22
Entropy (Prabludesai ot al,, 2025)
63.2
80.44
29.67
5.94
9.05
29.00
32.94
35.35
Majority- Vating (Shufayat al., 2025)
64.6
82.41
33.13
5.10
14.03
36.38
35.19
35.50
Co-rewarding: (Ours)
84.53
30.57
5,31
16.40
36.88
33.86
36.38
-o--ewarding-1 (Ours)
65.2
81.72
32.38
4.47
22.25
40.25
32.74
30.79
Qwen2.5-7R
Before RI.
69.4
24.71
15.81
2,81
3.79
26.38
38.19
44.76
GT-Reward (Shao 2024)
45.12
Self-Certiinly (Zhao et al., 2025b)
72.8
84.31
38.55
8.75
12.04
54.12
37.24
43,30
EEtropy (PraMludesai al. 2025)
39.61
Maaority- Voting (Shafayat ot al., 2025)
74.4
84.53
40.96
11.04
15.45
51.00
38.60
43.35
Co-rewarding- (Ours)
74.6
89.61
41.27
10.73
15.73
55.58
42.86
40.51
Co-rewarding-ll (Ours)
89.31
42.77
11.98
47.50
41.82
37.45
Qwen.3-1.78 1.--ate
Before RL
19.56
8.43
4.45
7.50
33.65
33.00
GT-Reward (Shuo et ol., 2024)
69.6
81.57
35.54
8.23
13.74
35.25
36.16
39.12
Certainty (Zhao al., 2025b)
23.04
Entropy (Prablulesai et al., 2025)
63.6
71.79
31.63
6.88
13.74
31.37
35.37
36.67
Majority Voting (Shafayat 2025)
34.25
Co-rewarding- (Ours)
67.6
83.01
32.22
8.65
13.50
32.38
35.56
35.53
Co-rewarding-11 (Ours)
66.2
80.89
33,28
7.50
14.40
32.88
36.94
37.59
60-
10
50
Comaiiyy
Edl Cor1
Condity
Cure
Highly Vila
My
Merdy
Mtacc
100
Figure 8: Performance curves validation set. Left Right: (Qwen2.5-3B, Qwen3-4B-Base Qwen3-8B-Basee -Jama-3.2-3B-Instrucl trained on MATH, Qwen3-4B-Ban trained on DAPO-144
C.3 MORE EVALUATION DETAILS
We conduct the evaluation across a diverse set of benchmarks, spanning matheratticl reasoning, code generation, instruction-followiiii
and general multi-task abilities. Specifically: (1) Mathemat- ical reasoning: We evaluate on MATH500 (Lightman et al., 2024), GSM8K (Cobbe et al., 202 I), and AMC (Li et al., 2024a). For MATH500 and GSM8K, report pass @ accuracy using the lighteval library². For AMC, we use the ttrl3 library and report avg@8 as the metric. (2) Code generation: We assess coding ability using LiveCodeBencc (Jain al., 2025) release v6 and CRUX (Gu et al., 2024). LiveCodeBench is evaluated with its offi fial evaluation library", and CRUX is evaluated via the ZeroEval library"; for both datasets, report pass@ accuracy. and multi-task abilities: We evaluate on IFEval (Zhou et al., 2023b) and MMLU-Pro (Wang et al., 2024), using the Im-evaluat iontioonhae ion-harness library for both. Overall, we summarize the statistics of the datasets used in this paper in Table 6.
(3) Instruction-followiin
hhtpss//gihub..oo//uugggggggi
bbaps///gituub..om/iiiiiii http://tw...oomm/iiiii//u//sooua
Rewand//tee//main//oolttt http:///hhhh.ooom/ilaaaa/////ei/
21

---

Published as a conference paper at ICLR 2026
Table 8: Supplement Results (%) of Co-rewarding and baselines trained on OpenRS. Cell background colors: darker colors denote better results within each model group.
Training Set: Open-RS
Mathematies
Code
Instruction
Multi-Task
Methods
MATH500
GSMBK
AMC
LiveCode
CRUX
IFEval
MMLU-Pro
Qwen.-88B- Base
Before RL
72.40
27.82
20.93
23.41
34.75
50.89
52.92
Reward al., 2024)
89.76
54.97
63.00
52.94
Self-Certainty (Zhao al., 20256)
82.60
85.22
50.00
37.00
64.62
52.12
56.03
Entropy (Prabhudesai et al., 2025)
80.60
87.41
48.95
38.00
61.25
52.53
56,80
Majority- Voting (Shufayat et ul., 2025)
78.00
84.23
51.96
36.75
58.00
51.13
54.92
Co ro-renddding (Ours)
78.20
92.65
50.60
28.91
63.12
53.11
57.21
Co-rewarding- (Ours)
80.00
90.90
53.01
39.75
62.75
52.92
56.55
Owen.T-4F-
Before RL
71.20
26.15
21.08
11.00
38.88
46.43
47.23
GT-Reward (Shao ul., 2024)
78.80
85.22
49.55
33.50
55.12
46.41
50.12
Self-Certainty (Zhao et al., 2025b)
73.20
33.43
35.84
32.50
49.50
46.47
48.24
Entropy (Prabhudesai et al,, 2025)
87.57
42.62
35.00
53.87
-Majority- Voting (Shufayat ul., 2025)
76.00
64.14
44.58
32.25
50.25
46.35
48.75
-C--reardding (Ours)
72.80
83.93
39.41
26.54
53.25
48.11
50.82
Co re-redird (Ours)
89.23
42.32
51.50
48.45
$1.80
Table 9: Performance (%) of test-time training (TTT), Since self-supervised methods are label-free, they can be leveraged during inference for test-time training to further enhance performance.
AMC
LLMs
Methods
2v
pass
avg
pass@116
pass
avg6064
6 64
Before-TTT
15.81
46,99
17.55
66.27
16.34
74.70
17.32
75.90
Self-Certaing
41.57
74.70
39.23
74.70
39.68
78.31
39.95
87.95
Qwen2.5.7B
Entropy
39.57
79.52
Majority- Voting
43.67
63.86
43.67
67.47
43.49
78.31
44.35
85.54
Co-rewarding-I
44.88
60.24
45.33
60.24
45.44
71.08
45.76
73.49
Co-rewarding
78.31
Before-TTT
20.93
61.45
21.31
73.49
19.58
79.52
20.97
86.75
Self-Certannty
49.85
78.31
50.68
78.31
50.41
84.34
49.55
89.16
Qwen3-8B-Ban BBBe-e
Entropy
48.64
74.70
49.92
80.72
49.96
87.95
50.23
89.16
Majority- Majonntt-Vottin
50.00
72.29
50.60
80.72
$1.36
85.54
Co-rewarding-I
52.86
68.67
53.46
74.70
53.24
81.93
53.58
84,34
Co-rewarding-1
48.64
72.29
48.19
73.49
50.19
83.13
49.28
91.57
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 MORE RESULTS ON OTHER TRAINING SETS AND LLMS
Table reports additional results of Qwen2.5-3B and (Dwen3-1.7--Bal and Qwen3-4B-Base trained on MATH, while Table 8 extends the experiments of Qwen3-8B BB---BBBBaa and Qwen3-4B 4B-aaee to another training set OpenRS (Dang & Ngo, 2025). can be observed that Co-rewarding occupies relatively darker areas. Across models and training sets, Co-rewarding-1 and II achieve average relative improvement of +2.23% on GSM8K, with notably high pass@ scores of 92.65% and 90.90% for Qwen3-8B-Ban trained OpenRS, respectively. Moreover, thanks to its stability, Co-rewarding-Il delivers more reliable gains than self-rewarding baselines, which occasionally suffer lower performance certain models benchmarks, e.g., Self-Certainty on Qwen3-1.7B-Bass in Table 7 or Majority- Voting on Qwen3-4B Bree3--B-BBss in Table 8. These results further demonstrate the effectiveness of Co-rewarding.
D.2 MORE CURVES OF REWARD, RESPONSE LENGTII AND PSEUO LABEL ACCURACY
Figure supplements the reward and response curves of Qwen3-4B Bass trained on DAPO- 14k. The trends consistent with Qwen3-88 32---B-Baatt BB-Base and Llama-3.2-2 3.22-3B-Instrun in Figure 4: Majority-Voting and Entropy rapidly increase rewards at carly stage and quickly peak, clear sign of reward hacking. In contrast, GT-Reward and Co-rewarding-Il exhibit smoother, steadily rising rewards, indicating gen-
22

---

Published as a conference paper at ICLR 2026
Qwno3-DB
Corcasnding Majutty Velig
Macetty Wakgeg
Sagator Suaney Roead
Mogemmo Legemtceuut
2000
1:04-
210
Figure 9: Curves of reward (Left), response length (Middle), and pseudo label accuracy (Right) of Qwen3-4B-Bas trained DAPO-144 14k. Entropy reward is plotted the right y-axis due to its different reward scale. Note that entropy minimization is to maximizing the negative entropy.
Table 10: Detailed MMLU-Pro performance on Qwen3-4B-Ban and 2ate-3..----IInstrrce traind on DAPO-14K. Results are reported for each of the 14 categories in MMLU-Pro.
Qwen3-4B-Benn
Methods
biology
business
chemistry
computer sci.
cconomics
engineering
health
GT-Reward
73.50
63.49
59.71
56,34
65.05
42.93
50.86
Self-Certainty
71.41
54.37
45.93
50.73
63.27
35.91
50.
Entrupy
70.99
56.02
50.44
48.29
63.15
34.37
48.41
Majurity- Voting
70.43
55.77
52.83
53.41
62.79
38.09
Co-rewarding-
73.92
59.82
50.71
54.15
64.93
41.49
49.76
Co-rewarding.1 II
72.66
59.95
55.65
53.41
64.10
39.73
50.61
Methods
history
law
math
other
philosophy
physics
psychology
GT-Reward
44.88
26.34
69.80
48.81
44,69
$7.04
65.79
Self-Certainty
39.63
24.43
59.44
43.94
40.08
47.04
59.65
Entropy
40.68
26.43
60.99
45.13
43.69
50.89
61.90
Majority- Voting
40.94
23.43
64.17
43.39
44.09
50.73
63.66
Co-rewarding-1
40.94
23.25
63.73
44.91
42.69
50.58
60.78
Co-rewarding-Il
42.26
24.79
67.58
44.59
41.88
54.19
62.91
Ilama3 2- 2-3B-Instruct
Methods
biology
business
chemistry
computer sci.
economics
engineering
health
GT-Reward
54.81
36.25
25.18
33.41
42.65
21.57
39.36
Self- Self-Certainty
55.23
32,95
27.21
31,95
42.77
20.54
39.12
Entroy
$2.86
31.05
23.94
32.93
41.71
20.43
38.02
Majurity- Voting
56.07
32,95
22.79
30.98
44.19
18.99
39.61
-o-rewarding-
51.88
34.22
22.88
34.88
44.67
19.09
38.63
Co-rewarding-1l
56.21
34.35
27.03
35.61
43.01
19.92
40.34
Methods
history
law
math
other
philosophy
physics
psychology
GTT-Rwward
30.18
22.71
34.20
34.74
32.06
28.33
50.38
Self-Certainty
30.45
24.98
33.38
31.60
29.86
28.56
50.50
Entropy
33.86
32.35
33.01
29.46
24.25
47.50
Majuniy- Voting
32.02
25.25
34.35
34.20
29.86
24.79
48.25
Co-rewarding-
33.86
23.25
32.12
33.01
31.86
25.40
48.75
Io-rewarding-I
32.28
24.34
35.83
36.26
33.27
28.18
49.12
uine learning of reasoning ability. Moreover, Co-rewarding-Il maintains moderate response lengths on Qwen3-4B-Batd further demonstrating its generality in balancing the exploration-exploitaiii trade-off during reasoning training, which a core principle of RI. (Wang et al., 2018).
Additionally, the right panel of Figure 9 presents the pseudo-label accuracy of Qwen3-4B- Bwen3-4B-Base. showing trends consistent with Qwen3-8B-Base and progresses, Co-rewarding-Ill steadily improves pseudo-label accuracy, while "w/o Updating Reference" remains around 25%. Majority-Votie briefly increases accuracy but soon collapses to zero, clearly indicating reward hacking. This highlights our design philosophy of pairing fast policy student with a slowly updated teacher, which decouples supervision from the online policy while enabling the teacher co-evolve with the student, thereby sustaining improvements in pseudo-label quality.
Figure 6. As training
23

---

Published as a conference paper at ICLR 2026
Queaa-- MAT3dM
Qwerk 48 Siore GSMSK
Qwee3-88 Banc AMC
Qwenk-B Sanr CRUX
GSMK Paxglt (No
ANC: Awa8 PN)
CRU Paugi
Trak
Coroentep
S:l Coskeey
Salf to
SalCondey
teneyy
Ineogy
Despy
Velk
Mu
Muiry Vale
Manty-Wale
30
191
190
2/
300
190
300
Step
Stop
Ulwes3-4B- Siae en MATR550
Chwest-4B diase en GOMMe
Qweel-4t-Haae on ANC
Qwen3-4i: Boor in CRUN
(N)
CONSRE Puavill 154
AMC CraStt (50)
Guand Trah
Gond Trsk
Grra Tra
CRUO
Cound But
Sel-Ceratty
ScleCemaityy
8
1aam--322-11 Instrnct MATR1500
Pandi (h0)
CBNMO Passil (hi
AMC Avg 50
CRUX CRUCKPPa Pundl L
Craued
Denieey
kill Donieey
0.
Stp
Sayy
Steg
Sup
Figure 10: Performance curves on benchmarks MATH500, GSM8K, AMC and CRUX across Qwen3-8B-Base, Qwen3-4B-Base, and Llama-3.2-3B 3.2--3B-Instruue trained on DAPO-14K.
D.3 MORE RESULTS OF VALIDATION PERFORMANCE CURVES
As a supplement to Figure 3, Figure 8 presents validation performance curves for Qwen2.5-3B, Dwen3-4B-Bass Qwen3-8B 1ee-8B-BBBaas Llama-3.2-3B I-struuntt trained MATH, well Qwen3-4B- Base trained on DAPO- 14K. Self-Certainty and Entropy collapse rapidly across all settings, as their supervision signals are tied internal confidence entropy and are easily exploited. Majority-Voting also collapses several cases, reflecting that sampling pseudo labels from outputs cannot prevent hacking. By contrast, Co-rewarding-I maintains stability across HIATH-trainet models through data-side contrastive agreement, while Co-rewarding-II consistently provides stability across all models and datasets by disentangling supervision with slowly updated teacher, making hacking substantially harder and optimization more reliable.
D.4 RESULTS OF TEST-TIME TRAINING (TTT)
Thanks to the label-free nature of self-superve supervieed methods, which do not require GT labels, they naturally compatible with test-time training (TTT), enabling further refinement of the model during inference. Table reports the TTT results on the challenging competition-level benchmark AMC across Co-rewarding and other self-rewarding baselines. We observe that Co-rewarding matches or even surpasses existing methods, achieving the best results on 11 out of 18 metries. These findings broaden the applicability of self-supervised RL: beyond post- training for reasoning elicitation, it can also be leveraged inference time further improve performance on specific benchmarks.
D.5 MORE RESULTS OF BENCHMARK PERFORMANCE CURVES
As a supplement to Figure 3 and Figure 5, Figure 10 presents performance curves on MATH500, GSM8K, AMC, and CRUX with Qwen3-8B- BB-eee, Qwen3-4B-Bantl and Llama-3.2-3B 13.2-3B-Iman Instruct.. Consistent with earlier findings, Self-Certainty, Entropy, and Majority- Shotiiity-Voting rapidly collapse across benchmarks and models, while Co-rewarding-Il and GT-Reward sustain continued and stable im-
24

---

Published as a conference paper at ICLR 2026
Table 11: Impact of math training collapse code and multi-task performance. Results are evaluated on models before and after training collapse.
Qwen3-4B-BBan
Mathematics
Code
Multi-task
Training stage
Methods
MATH500
GSMSK
AMC
AIME24
LiveCode
CRUX
MMLU-Pro
Self' Self-Certainty
68.4
44.81
35.39
8.85
25.88
50.12
48.84
Before training collapse
Entropy
76.6
82.79
43.37
12.81
26.35
50.75
Majority-Vone Votting
73.4
64.06
40.81
9.17
26.16
53.00
51.06
SeIf-Certainty
2.8
3.34
2.71
0.00
14.22
8.12
29.71
After training collapse
Eatropy
2.8
2.35
3.46
0.00
18.60
31.75
28.13
Majority-Votie
2.8
4.85
1.36
0.00
24.36
52.75
50.19
Table 12: Difference between original and rephrased questions from background richness, vocab- ulary complexity, and sentence complexity.
Training Set
# Data Size
Background richness
Vocabulary complexity
Sentence complexity
MATH
7,500
+4.91%
+4.79% 1.95%
+9.05%
DAPO-14k
14,100
+4.65%
+4.19%
Table 13: Success rate of different rephraser LLMs: MATH training set rephrased by Qwen3-32B,
Qwen3-8B, and Qwen3-1.7B, respectively.
Rephraser LLM
Training Set
# Original questions
# Rephrased questions
Success rate (%)
Qwen3-32B
MATH
7,500
7,498
99.97%
Qwen3-8B
MATH
7,500
7,477
99.69%
Qwen3-1.73 1.7B
MATH
7,500
2,060
27.47%
provements. These results underscore the link between performance and training stability: stable training enables models to continue improving by effectively learning knowledge from more data.
D.6 IMPACT OF MATH TRAINING COLLAPSE ON OTHER TASKS
We investigate how training collapse occurring on math-oriented training sets impacts the model's performance on code-gende onde-generation and multi-task benchmarks. To this end, we evaluate models trained with existing self-rewarding methods Selff-Cetainnt, Entropy, and Majority- Hajority-Voting)) both before and after training collapse. Table summarizes the results. We observe that training collapses on math-related training affect other tasks (LiveCode, CRUX and MMLU-Pro) in different way for certainty- or entropy-based methods SSelf-Certannty and Entropy) compared with consensus- based methods (Majority- -ottng)).. When collapse occurs on math-oriented training sets, all three methods show substantial performance degradation on the four math benchmarks (MATH500, GSM8K, AMC, and AIME24). However, their impacts on other tasks differ:
For certainty- or entsepy-based methods, the performance on LiveCode, CRUX, and MMLU-Pr also declines after collapse on math training sets. This arises from their reward objectives: maximizing self- certainty minimizing entropy, result in the decoding probability mass becoming highly concentrated on very subset tokens. Consequently, the model produces repetitive outputs, and this repetitive decoding behavior transfers across tasks, leading to degraded performance beyond the math domain.
For the consensus- based method, Majority- Voting shows similar performance before and after training collapse on math-oriented training sets. This may be because its collapses stem from reward hacking at the answer format: the model exploits the \boxed{) structure by consistently inserting an incorrect but self-conse consistent answer maximize reward. This type of collapse weakly affects the intermediate reasoning trace, which largely remains structured. Since code- code- generation and multi-task benchmarks do rely boxed-answer extraction, this type of collapse has limited impact their performance.
25

---

Published as a conference paper at ICLR 2026
Table 14: Impact of rephraser LLM for Co-rewarding-L. Train Qwen3-8B-Ban using data rephrased by Qwen3-32B, Qwen3-8B and Qwen3-1.7B, respectively.
Trained Model
Rephraser LLM MATH500
GSM8K
AMC
AIME24
LiveCode
CRUX
1FEval
MMLUUPPr
Qwen3-32B
81.2
93.70
51.20
15,10
30.81
66.00
55.79
59.95
Qwen3-8B-Base
Qwen3-8B
79.2
92.72
51.51
14.58
30.90
63.12
54.73
59.30
Qwen3-1.7B
78.2
87.41
49.25
12.81
29.57
61.00
53.44
55.85
Qwen2.5-38
Qwent-1.7B-Banl
Qwem3-6B-BBse
70
Voiing Accuray (90
65
Variig Accuncy (9N
80-
Variig Accunazy
60-
155
50-
Micuuity Votig
50-
Mijocity Vidiig
5
Majoclty Voting
Original Date
Original Data
wi Originall Data
Repluased
Reptuaied
Repttaaed
Coveonanding!
40-
Co-eowanding-
66-
Co-rowandigg-
100
350
100
:50
200
100 Step
150
200
Step
Step
Figure 11: Curves of voting accuracy of Mlajority-Voting, Co-rewarding-I and its ablations with Qwen2.5-3B, Owen3-1.7B-BB and Qwen3-8B-BB -B--Base trained on MATH.
D.7 DISCUSSION OF MATH AND DAPO-14K 14K
We leverage Qwen3-235P 2----222BBBB2BBB to score the difference between original and rephrased questions from multiple perspectives, including background richness, vocabulary complexity, and sentence complexity, for MATH and DAPO- 14k. From Table 12, we observe that the rephrasing in MATH exhibits larger changes from the original to rephrased questions than DAPO-14K. This suggests that the questions in MATH may provide favorable conditions for promoting diverse rephrasing variability, which is beneficial for the effectiveness of contrastive agreement in Co-rewarding-l.
D.8 ROBUSTNESS ANALYSIS OF DIFFERENT REPHRASER LLMS
To analyze the impact of different rephraser LLMs for Co-rewarding-l, conduct additional experiments using smaller LLMs instead of Qwen3-32B for rephrasing. To control architectural variability in the rephraser LLMs, employ two smaller LLMs from the same family, i.c., Qwen3- 8B and Dwwen3-1.7B, for rephrasing the MATH training set. Table 13 reports the rephrasing success rate. We observe that rephrasing success rates drop the model size decreases, which expected: rephrasing math questions while preserving the analogical essence a relatively challenging task, and weaker LLMs struggle achieve this goal. This observation supports our choice of Qwen3-32B as the rephraser, as sufficiently capable LLM is required to produce faithful rephrasing.
We then train Co-rewarding-I Qwen3-8B-Bann using rephrased data generated by Qwen3-32B, Qwen3-8B, and Qwen3-1.7B, respectively. The performance is summarized in Table 14. From the results, it can be observed that performance gradually degrades as the size of the rephraser LLM decreases, but not always significantly. Rephrasing with Qwen3-8B maintains reasonably similar performance to using Qwen3-32B, indicating that Co-rewarding- exhibits certain degree of robustness under moderate reductions in rephrasing quality. Notably, rephrasing with Qwen3-1.77 1.7B leads to a substantial performance drop. This degradation is largely attributable to the significantly lower rephrasing success rate of Qwen3-1.7B, which results in a substantial reduction of usable training data and consequently weakens the effectiveness of Co-rewarding-I.
D.9 VOTING ACCURACY ANALYSIS OF CO-REWARDING-1
To demonstrate the stability and efficiency of Co-rewarding-L, we compare its voting accuracy against that of Majority- Voting in Figure 11 and Figure 12. These experiments conducted Qwen2.5- 3B, Qwen3-1.77 1.23--1.7B.. 1.7B-Base and Qwen3-8B- Base models, all trained on the MATH dataset. Across all settings, the Sajority-Vottin method exhibits reward hacking, where its performance sharply declines
26

---

Published as a conference paper at ICLR 2026
Oreec Higuety lakig
Ore
Queeet
Lods
Figure 12: Voting accuracy of Majority-Voting, Co-rewarding-I and its ablated variants across different difficulty levels of questions Top: Qwen3-1.7B-Base Bottom: Qwen2.5-3B.
after reaching an early peak, particularly more difficult questions (levels 2 to 5). In contrast, Co-Rewarding-I maintains a stable voting accuracy on both original and rephrased data. Ultimately, it achieves the highest overall voting accuracy across all models and dataset configurations.
Table 15: Detailed MMLU-Pro performance on Qwen3-8B-Bant and Qwen3-4B-Ban trained on OpenRS. Results are reported for each of 14 categories in MMLU-Pro.
Qwen3-4B-Ben
Methods
biology
business
chemistry
computer sci.
economics
engineering
health
GT-Reward
70.99
59.82
52.30
54.63
65.05
39.01
$1,22
Celf-Certainty
54.50
44.08
49.27
63.63
37.36
Entropy
70.71
58.68
49.03
51.22
63.39
37.46
49.63
Majority Voting
69.60
55.77
47.17
53.17
63.39
36.02
48.78
Co-rewarding-1
69.04
55.39
47.79
53.41
63.86
38.39
50.61
Co-rewarding-1
70.85
58.81
53.27
53.90
66.11
37.15
52.81
Methods
history
law
math
other
philosophy
physics
psychology
GT-Reward
39.63
24.98
65.58
47.84
40.68
54.50
62.53
Self-Certainty
39.63
24.25
58.11
46.65
40.88
46.42
61.40
Entropy
39,90
22.16
62.18
45.02
43.09
50.19
Majority- Voting
40.68
22.52
60.25
46.10
41.08
48.42
60.65
Co-rewarding-
40.68
24.25
62.18
44.37
44.49
49.58
61.65
Co-rewarding-
41.21
25.89
64,91
45.24
39.28
52.27
59.40
Qwen3-8B-Ban
Methods
biology
business
chemistry
computer sci.
economics
engineering
health
GT-Reward
74.76
63.24
55.48
63.17
68.96
41.38
$7.09
Self-Certainty
75.03
63.62
53.62
55.61
68.96
39,83
$7.09
Entropy
75.73
64.39
54,51
58.29
65.05
41.69
55.87
Majority- Voting
76.15
60.20
54.15
56,34
69.91
38.91
55.75
Co-rewarding-I
76.43
65.78
57.07
62.20
69.43
43.14
56.60
-Co-rewarding II
76.84
64.25
54.68
62.43
68.12
42.00
58.06
Methods
history
law
math
other
philosophy
physics
psychology
GT-Reward
50.92
30.25
67.58
$2.49
51.10
$7.20
67.67
Self-Certainty
49.34
28.88
68.02
$1.62
52.10
56.89
66.42
Entropy
50.39
30.43
65.28
51.41
47.09
$4.50
66.67
Majority- Voting
48.03
28.88
63.43
53.68
48.10
52.50
64.66
Co-rewarding-
50.13
29.97
68.54
52.92
50.70
56.66
65.54
Co-rewarding-Il
51.44
30.06
65.80
51.51
52.10
57.58
65.78
27

---

Published as a conference paper at ICLR 2026
D.10 MORE RESULTS OF MMLU-PRO EVALUATION
As complement to Table 4, Table 10 and Table 15 report detailed MMLU-Pro results for models trained DAPO- 14k and respectielly We observe that Co-rewarding consistently preserves general-doman performance across diverse subjects, indicating that though trained on math orientoreetted datasets, its improvements do not come at the cost of broader capabiliiies from other domains.
D.11 MORE RESULTS OF IFEVAL EVALUATION
The aim of IFEval is used evaluate the instruction-foolloinng ability of LLMs. In Table Table 2, Table and Table 8, we report average IFEval performance due to space constraints. Specifi- cally, the evaluation of IFEval includes four metries: lerompp_leve__stritt.. Leonpt.eveeLloocccuuu
inst_leve and inst woose _acc), which apply different levels of answer matching. As supplement, complete results are provided in Table 16. Table 17, and Table 18. The results show that Co-rewarding not only preserves the inherent instruction-follwwee also often surpasses -eward across multiple models. This further confirms that Co-rewarding gains on mathematical and coding benchmarks are achieved without sacrificing instruction-folloingg ability.
ability of base models but
domain
Table 16: Detailed 1FEval Performance on Qwen2.5-3B/7B, Biren--.7B//B/8B-BB and Llama- 3.2-3B-Instruct traind on MATH. Results are reported for loose and strick settings respectively.
Methods
IPEva
Average
Promut Striet
Prompt Loose
Inst. Strict
last. Loose
Average
Prompt Striet
Prompt Loose
Inst. Striet
Enst. Loose
Qhvon25530
Qwen2.5-70
Before RL.
29.83
22.55
27.17
31.89
37.30
38.19
29.57
34,57
41.85
56.76
GT Reward
33.66
25.51
31.42
25.85
41.85
41.49
31.79
39.56
43.65
50.95
Scdf-Cestiintt
32.22
24.40
29.76
34.65
40.05
$7.34
28:47
$4.
40.05
46.04
Entropy
32.94
24.77
30.50
35.13
41.37
40.33
20.13
37.87
43.29
50.00
Majority Vining
35.19
26.25
32.72
37.53
46.24
38.60
29:21
35 35.86
41.61
47.72
Co Co-newandin--
33.86
23.84
31.61
36.00
43.58
41.73
32.35
99.37
44.45
50.72
Co rewarding II
32.74
23.29
29.02
26.33
42.33
41.82
31.79
40.29
43.85
51.31
Qbeea-11.7B. J.78-Buse
Qhven3-4B-Bune
Before RL.
33.65
25.69
30.86
36.45
41.60
46.43
36.04
44.18
48.68
56.83
GT--Reaad
36.16
27.35
31.79
40.64
44.84
47.50
37.34
46.27
49.40
$7.67
Self Certaimy
32.56
24.58
29.20
36.69
41.36
48.15
19.37
05,
49.52
$6.95
Entropy
35.37
25.61
31,40
40.57
48.61
52.52
99:07
Majocity Noting
35.45
26.06
32.16
28.72
48.54
49.78
37.89
47.50
50.36
99.65
Co rewarding
33.56
27.11
31,23
39.32
43.76
50.35
40.57
09:35
51.56
39.83
Co-swwnniing-1
36.94
27.17
33.54
40.05
46.88
$1.30
41.40
40.54
53.12
61.15
L-oma----nnsrurt
Before
OT--Rewand
52.78
41.96
$1.76
54.44
47.41
17.34
SelF-errainty
Entopy
$1.20
52.76
57.67
Majority-Voinng
$1.00
Coscwanding-
55.79
Co-ewanding-1
63.70
65.59
56.00
65.59
30.93
59.35
D.12 ORIGINAL QUESTIONS VS. REPHRASED QUESTIONS
To provide an intuitive illustration, we present several examples of original questions with their rephrased versions in Table 19, We observe that such rephrasings are reasonable and effective, as they preserve the same underlying mathematical essence while presenting the problems in a substantially different surface form. This reflects the high quality of our rephrased data and forms the basis of Co-rewarding-I: by leveraging contrastive agreement across data-invariant variants, the model is encouraged to elicit more robust reasoning ability.
D.13 COMPLETE CASE STUDY
As supplement to Figure 7, we present the complete generation outputs of this case study. The full outputs clearly reveal the reward hacking behaviors existing self-rewarding baselines. Self- Certainty and Entropy fall into repetitive outputs-foo example, Self-Certainty repeatedly generates "Understanding." and Entropy repeatedly produces "Simplify the next fraction" until truncated at the maximum length. This arises because their decoding probability mass collapses onto small subset tokens, leading the model to loop over them. Majority- Voting shows another form of reward hacking by boxing incorrect answer "(" to maximize consensus across rollouts and thereby secure
28

---

Published as a conference paper at ICLR 2026
Table 17: Detailed 1vEval performance on Qwen3-4B/8B-Bas and Llama-3.2-BB-Inst Instructtt traind on DAPO-14K. Results are reported for loose and strict settings in IFEval, respectively.
IFEval
Methods
Average
Prompt Striet
Prompt Loose
Inst. Striet
Inst. Loose
Qwen3-4Bi Bass-eeBBBase
Before RI.
46.43
36.04
44.18
48.68
56.83
GT--Reward
47.70
37.52
45.84
49.76
57.67
Self-Certainty
45.58
35.67
43,99
47.84
54.80
Entropy
48.20
37.71
46.58
50,48
58.03
Majority- Voting
48.91
39.19
47.69
50.24
58.51
Co-rewarding-1
46.84
36.41
45.66 36.66
48.80
56.47
Co-rewarding-I II
48.90
39.56
46.21
51.44
58.39
Qwen3-8B-Ban
Before RI.
50.32
40.11
50.27
51.07
59.83
GT-Reward
53.11
41.59
52.13
54.56
64.15
Self-Certainty
50.58
41.04
49.54
51,68
60.07
Entropy
51.56
41.59
49.91
53.48
61.27
Majonity- Voting
51.54
41.22
51.02
52.64
61.27
Co-rewarding-
50.17
40.67
59.59
Co-rewarding-II II
53.31
41.40
53.23
54.20
64.39
Llama3.2-3B-
Before RI.
57.32
46.77
55.27
60.19
67.03
GT--Reward
53.10
42.33
49.91
57.19
62.95
Sclf-Cortainty
54.50
44.55
51.76
58.03
63.67
Entropy
55.78
45.29
53.23
59.11
65.47
Majority- Voting
42.33
52.50
56.83
64.63
Co-rewarding-I $3.04
42.33
51,02
55.76
63.07
Co-rewarding-II
51.92
41.59
48.24
56.00
61.87
Table 18: Detailed 1vEl Performance on Qwen3-8B/4B-Bas trained Open-RS. Results reported for loose and strict settings in IFEval, respectively.
Mithods
vall
Average
Prompt Striet
Promupt Loose
Inst. Striet
last. Louse
Axerage
Promutt Striet
Prompt Loose
last. Striet
Inst. Loose
(vea3-88- -B- Base
Qww.3-48-Buse
Before
50.32
40.11
50,27
$1.07
36.04
56.83
-G--Rwwrdd
52,83
41.59
$1.02
34.56
62.95
37.34
46.77
49:40
$7.67
Scll-Catainty
Eutoopy
Magoriy- Vining
40.67
53.36
61.15
48.80
56
Oommanting-
$3.95
Os-menninngg
the highest reward. In contrast, both Co-rewarding and GT-Reward yield the correct final answer along with coherent step-by-step reasoning path.
D.14 CASE STUDY CODE BENCHMARK
To understand the behaviors of different methods other tasks, supplement additional case study on CRUX. We observe that Self-Certainty and Entropy fail by producing repetitive outputs, consistent with the collapse pattern math-related benchmarks. In contrast, GT-Reward, Majority- Voting, and Co-rewarding generate coherent reasoning trace and correct solutions.
Question:
is the Irreducible fraction value of
+
3
+
3+
what is the value of m nt?
29

---

Published as a conference paper at ICLR 2026
Table 19: Case studies: Original vs. Rephrased Questions
Original Question
Rephrased Question
Sam is hired for a 20-day period. On days that he works, he earns $60, For each day that he does not work, $30 is subtracted from his earnings. At the end of the 20-day period, received $660. How many days did he not work?
A contractor agrees to job lasting 20 days. For every day the job completed on time, the contractor earns $60. However, for each day the work delayed, a fine of $30 is ap- plied. After the 20-day period, the contrac- tor's s total carnings are $660. How many days the job delayed?
Karen drove continuously from 9:40 a.m. un- til 1:20 p.m. of the same day and covered distance of 165 miles. What was her average speed in miles per hour?
A traveler set off at 9:40 a.m. and reached their destination 1:20 p.m. the same day after traveling total of 165 miles, What was their average speed during the trip in miles per hour?
Solve for x: 2 + 6
A tank is partially filled by two different pipes. One pipe fills half the tank in an hour, and together with another pipe, they fill five-sixths the tank in the same time. pipe alone fills of the tank in an hour, find the value of æ.
second
Co-rewarding-I Answer:
To solve the problem, we need to simplify the given expression step by step and then find the irreducible fraction form. The expression is:
3+
3
3+
First, we simplify the innermost fraction:
1
9
1
10
3
3
3
Now substitute this back into the expression:
3 +
3
+
3
+
10
3
Next, simplify the fraction inside the denominator:
30
33
10
10
10
10"
Now substitute this back into the expression:
10
3
+
=
3
33
33'
10
To add these, we need common denominator:
10
99
10
109
3
33
33
33
33
The fraction 109 is already in its simplest form because 109 and 33 have no common factors 33 other than Therefore, m 109 and n = 33, and the value m + n is:
109 + 33 142.
So, the final answer is:
142
30

---

Published as a conference paper at ICLR 2026
Co-rewarding-Il Answer:
To solve the given expression 3 + 3+ 3+3
we need to simplify it step by step.
First, simplify the innermost fraction:
1
1
10
3
3
3
3
Next, substitute this back into the next fraction:
10
3 + 3+ 13
= 3
3
+
+
3 +
3
3
30
3
33
33
16
10
10
Now, simplify the expression 3
33
10
99
10
109
3
33
33
33
The fraction 109 is already in its simplest form because 109 and 33 are coprime (they have 33 no common factors other than 1).
Thus, the irreducible fraction value of the given expression is 109 33. Therefore, m + n 109 + 142.
33 where m = 109 and
The value of m + i 142
GT-Reward Answer:
AIright, have this problem: The expression is:
1
+ 3+3
And need find the irreducible fraction m that equals this expression, then determine m +
First, need to simplify the expression step by step. looks like continued fraction, so I'll start by simplifying the innermost part and work my way out. Let's denote the entire expression as:
3
3+
**ttep 1: Simplify the innermost fraction**
Start with the innermost fraction:
3 +
3
Calculate this:
9
10
3+
3
3
So now, the expression becomes:
5-34
3
Wait, need to simplify
1
10
3
Now, the expression is:
3
3+ 3
31

---

Published as a conference paper at ICLR 2026
**Step 2: Simplify the next fraction** Now, simplify + 3. 10'
3
30
3
33
3
+
+
10
10
10
10
So, the expression now
E = 3+
33
Simplify +
10
TO
Now, the expression is:
E 3 + 33
**ttep 3: Combine the terms**
Convert to a fraction with denominator 33:
3 99
Now, add the fractions:
99
10
109
E
+
33
33
33
So, the expression simplifies to 109
33 **Step 4: Ensure the fraction is *
Check if 109 and 33 have any common divisors. 109 is prime number (I think), and 33 is 3 x 11. Since 109 isn't divisible by 3 or 11, the fraction 33 already its irreducible form.
**Step 5: Find m + n
Here, m 109 and 33, S0:
+ n 109 + 33 142
Wait, let's double-check the calculations to ensure no mistakes were made. **Verification:** #* Starting over:
E. 3
3+
Innermost:
10
3
3
Next layer:
3
33
+
34
10
10
10
3
Outermost:
1
10
99
10
109
+
3
+
33
33
33
33
10
Yes, checks out. Therefore, m + n 109 142
+ 33
142.
Majority- Voting Answer:
To find the value of m as the irreducible fraction of the expression + n
we need to
34
simplify the expression step by step.
32

---

Published as a conference paper at ICLR 2026
First, let's simplify the innermost fraction:
10
-
3
3
Now, substitute this back into the expression:
1
3
3 3
3+1 10
3
Next, simplify the fraction 3 +
3
10
3
30
3
33
3 +
+
10
10
10
10
Now, substitute this back into the expression:
10
+
3
33
33
10
To add these, we need a common denominator:
10
99
10
109
+
33
33
33
33
So, the irreducible fraction value of the expression
109
Here, m 109 and 33.
33
Therefore, the value of m + is:
m + n 109 + 33
142
Thus, the value of m + n is
Self-Certainty Answer:
Alright, I have this math problem solve:
**Problem:** need to solve a math problem. The problem is about understanding how to
solve math problems. need to understand how solve this problem so I can understand more about math.
### Understanding the Problem
First, need understand what the problem is. need understand math so I can understand more about numbers and how to solve problems with numbers. Understanding math will help understand more about how to understand problems. ### Understanding Math Problems
After understanding the problem, I need to understand more about math problems. Un- derstanding math problems will help me understand how to understand these problems. Understanding these problems will help me understand more about how understand math. ### Understanding Mathematical Concepts
After understanding math problems, I need to understand more about mathematical concepts. Understanding mathematical concepts will help me understand more about math. Under- standing these concepts will help me understand more about how math helps me understand numbers.
### Understanding Mathematical Operations
After understanding mathematical concepts, I need to understand more about mathematical operations. Understanding mathematical operations will help me understand more about how understand these operations. Understanding these operations will help understand more about how these operations help me understand math. ### Understanding Mathematical Operations in Math
After understanding mathematical operations, need to understand ### Understanding Mathematical Methods
33

---

Published as a conference paper at ICLR 2026
After understanding matheratical operations, I need understand about mathematical methods.
### Understanding How Mathematical Methods Help
#### Understanding How These Methods Help Me Understand Math
After understanding more about how these methods help me, need to understand
### Understanding How These Methods Help Me Understand Math Problems
After understanding how these methods help me understand math, I need understand ### Understanding How These Methods Help Me Solve Math Problems
After understanding how these methods help me understand these math problems, need to
understand
### Understanding How These Methods Help Me Solve These Problems
After understanding how these methods help me solve these problems, need to understand
more about
Entropy Answer:
To find the value of as the irreducible fraction of the expression:
3
+ 3+
Let's simplify the expression step by step. **Simplify the innermost **
3
3
+
3 +
3
3
3
3
3
3
3
**Simplify the next fraction:** **
1
+
3
+
1
TO
3
3
1x3
3
10
30
3
30 +
33
3
10
=3
10
=
3
+
10
10
10
10
+
10
10
10
3
** *iimpiffy the next fraction:
+
+
3
+
1
33
3+3
1 x
33
10
99
10
99 +
10
109
3 +
3
3
33
33
33
33
33
33
33
33
33
** Simplify the next fraction:** #*
3
+
3
100
33
31
1
1 x
33
33
3 X 109
33
327
33
327
+
33
360
3
3
3 +
109 33
109
109
109
109
109
109
109
109
Simpllfy the next fraction:**
3 +
3 +
3
360
34
109
34
34

---

Published as a conference paper at ICLR 2026
X 109
109
3 x 360
109
1080
109
1080 + 109
1189
3+
= 3+
3+
+
360
360
360
360
360
360
360
360
360
109
6. ** Simplify the next fraction:**
3
1189
3+
360
7. *impiiyl the next fraction:* **
1
3
3
3927
1189
3+
Simplify the next fraction:**
3 +
3 + 12970
3+
3927
31
** Simplify the next fraction:**
3
3 + 12837
3
12970
3+
3+
10. **impllff the next fraction:"
+
3 3 +
3+
42837
54
11. **Smpplify the next fraction:** **
3 +
3 +
467280
141481
3+
31
34
12. **Simplify the next fraction: **
+
3 +
3
1543321
167280
3+
13. *Simplify the next fraction:**
35

---

Published as a conference paper at ICLR 2026
+
3
3 +
5007243
3+
31
3+
34
3 5097243
3+
1543321
Question:
## Question:
You are given Python function and assertion containing input the function. Complete the assertion with literal (no unsimplitied expressions, no function calis) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete.
[PYTHON]
def (text)
sex_texx = list (text)
for i in '+': if in new new_text:
new text.extt.o remove (i)
return " joinn (nex_tex)) assert f(' (('hboofdeiequ') == ??
I/PYTHON]
# Instruction
Please this question by first reasoning and then previding your answer. Present your reasoning and solution in the following jsen format. Please show your final answer in the 'answer' field, c.g., "answer"; "42".
json
"reasoning"; "answer";
GT-Reward Answer:
"reasoning": "The function 'f' is designed to remove all from the input string 'text', The function first. converts the string into a 1ist of characters iterates through this list, and removes any fur characters it finds. Finally, it joins the remaining characters back into single string. Given the input string "hbtofdeiequ', there no '+' characters in the string, so the function will simply return the original string unchanged. Therefore, the output of the functicn when given the input 'bbtofdeiequ' is the string hătofdeicqu'.",
occurrences the character
36

---

Published as a conference paper at ICLR 2026
Manswer" "hbtoffeiequ"
Co-rewarding-I Answer:
"reasoning": "The function 'f' is designed to remove all occurrences the character 't' from the input string 'text', The functicn first converts the string into list of characters then iterates over the character " and removes it from the list it it exists. Finally, it jeins the 1ist back into a string and returns it. Given the input 'hbtofdeiequ', the character '4' does not appear the string, the function will return the original string unchanged.", "answer": "hbtofdeiequ"
Co-rewarding-Il Answer:
"reasoning"; "The function 'f' takes string 'text and removes all occurrences of the character from it. The function then joins the remaining characters back into string and returns it. Given the input 'hbtofdeiequ', the function will remove all '4' characters (though there are none in the input), the output will be the the asserticn should check the output of #E'"bboofdeieq"") is 'hbtofdeiequ'. ", "answer"; "nbtofdeiequ"
the input string. Therefore,
Majority- Voting Answer:
"reasoning": "The function 'f' is designed to remove all occurrences the character '4' from the input string 'text', The input provided to the function is "bbtofdeiequ'. Since the character 't' does not appear in this string, the function will return the criginal string unchanged. Therefore, the assertion should compare output of the function the original input string.",
"answer"; "'hbtofdeiequ""
Self- Self-Certainty Answer:
solve this problem, need to follow these steps:
1. Understand the given Python function.
2. Determine what the function does with the given input.
3. Complete the assertion with the correct output based on the function's behavior.
37

---

Published as a conference paper at ICLR 2026
Let's 8 go through each step one by one:
1. Understand the given Python function.
2. Determine what the function does with the given input.
3. Complete the assertion with the correct output based on the function's s behavior.
Wo will write the code understand the function, determine its behavior with the given input, and then complete the assertion with the correct output.
Let's write the code step by step. 'pyythnn
Step 1: Understand the given Pythen function
Step 2: Determine what the function does with the given input
Step 3: Complete the assertion with the correct output based
the functicn's behavior
Let's s write the code to follow these steps
"python
Step 1: Understand the given Python function
Step 2: Determine what the function does with the given input
Step 3: Complete the assertion with the correct output based on the functicn's behavior
let's s write the code to follow these steps
Step 1: Understand the given Pythen function
We will define the given function and understand what it does
Step 2: Determine what the function does with the given input We will determine the output of the function for the given input
Step Complete the assertion with the correct output based the function's behavior We will complete the assertion with the correct output
Let's write the code ..1
Let's write the code step by step. 'python
Step Understand the given Pythen function
Step 2: Determine what the function does with the given input
Step 3: Complete the assertion with the correct output based on the functicn's bohavior
Let's write the code to follow these steps
''python
Step 1: Understand the given Pythen function
will aefine the given function and understand what does
Determine what the function does with the given input We will determine the output of the function for the given input
Step 2:
Step 3: Complete the assertion with the correct output based on the function's s behavior We will complete the assertion with the correct output
38

---

Atext (new)_texts = \text (' ('hbtofdeiequ') AJ M Published as a conference paper at ICLR 2026
Let's write the code
Step 1: Understand
Entropy Answer:
jsonn
"To determine the output of the function '2' when the input is 'bbtofdeiequ' we need to follow these steps:
"reasoning":
1. Convert the string 'bbtofdeiequ' into list:
MI
Atext
l'h', 'b' 't', 'o', r 'd', re', 'i, 'e', 9', [u')
V
2. Iterate through the character (+
MI
Atext [for } A in } 4e
AI
AL
Atext (for } \ in '+:
A
M
Atext (for A in (t;
AJ
M
Atext (for } Atext in } (+:
A
AL
Atext (for } \ in '
A]
M
Atex (for } Atext in 44:
A
M
Atext (for } 1 \ in 1+;
AJ
Atext (for } \ in
39


Reviews:
[{"review_id": "8GdsShIPZH", "reviewer": "Reviewer_qDgd", "summary": "The paper proposes Co-rewarding, a self-supervised reinforcement learning framework designed to improve the reasoning ability of large language models without relying on ground-truth labels. The method aims to prevent the collapse and reward-hacking issues common in existing self-rewarding approaches by introducing complementary supervision from two perspectives: (1) Co-rewarding-I, which derives reward signals through contrastive agreement across semantically rephrased questions, and (2) Co-rewarding-II, which employs a slowly updated teacher model to provide temporally decoupled pseudo-labels. Experiments across several reasoning benchmarks (GSM8K, MATH, AMC, etc.) show that Co-rewarding achieves more stable training and in some cases matches or surpasses RL with ground-truth rewards, especially when using Qwen-3 and Llama-3 models.", "questions": "1. The paper states that the two co-rewarding techniques stem from the same core insight but offer complementary perspectives for cross-supervision. However, it remains unclear whether combining them actually yields better performance than using either one alone. If yes, what is the underlying mechanism that makes the combination more effective than a single method?\n\n2. The paper would benefit from additional case studies and quantitative evaluation of the rephrasing models, particularly under low-quality paraphrasing which may change the reasoning results. Since the method claims to stabilize RL, one would expect robustness to varying data-augmentation conditions. Can the authors analyze training stability when rephrasing degrades, and report how performance varies with different levels of paraphrasing noise?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Introduces a clear and well-motivated framework that targets a real weakness in current self-rewarding RL methods: instability and collapse caused by single-view reward signals.\n\n+ Extensive empirical results across multiple models, datasets, and baselines, including ablations that isolate the contribution of each component. \n\n+ The paper is overall well-written, with clear mathematical formulation and structured presentation.", "weaknesses": "+ The reliance on high-quality paraphrasing for Co-rewarding-I is insufficiently examined. The framework may degrade when rephrasing quality is low or domain-specific, yet no robustness analysis is provided. Although the paper claims that rephrased questions should yield similar reasoning outcomes, in practice the reasoning trace can vary a lot depending on how the question is phrased. Thus, how do the authors ensure the meta-transferability of both reasoning and final answers during rephrasing? Moreover, when paraphrasing leads to divergent results, how do the authors obtain the updated groundtruth?\n \n+ Although positioned as scalable and label-free, the method still requires additional compute from rephrasing models and dual rollouts, but the paper does not report cost or latency analysis w.r.t. the baslines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzsqlYky8o", "reviewer": "Reviewer_CbNe", "summary": "This paper tackles the training collapse issue in self-rewarding RL for LLM reasoning. The authors propose Co-rewarding, a self-supervised RL framework that seeks complementary supervision for stability. It is a two-stage process where Co-rewarding-I uses \"contrastive agreement\" via cross-reference between original and rephrased questions and Co-rewarding-II employs self-distillation from a slowly-updated EMA \"teacher model\" to provide stable pseudo-labels, decoupling the reward signal from the online policy.\n\nExperiments show Co-rewarding achieves stable training, avoiding the collapse seen in baselines like Entropy and Majority-Voting. The method outperforms these baselines and, notably, even surpasses training with ground-truth labels in several cases, achieving a 94.01% Pass@1 on GSM8K.", "questions": "1. The evaluation omits high-difficulty, competition-level benchmarks (e.g., AIME 24/25). Given that the training set includes data from DAPO-Math-17k, why wasn't a benchmark like AIME used to more rigorously validate the model's advanced reasoning capabilities?\n2. The design of Co-rewarding-II, which uses a dynamic, EMA-updated 'teacher' as the reference for the KL constraint , seems to violate the fundamental purpose of this term. Traditionally, the KL divergence penalizes deviation from a fixed reference policy, which acts as a stable anchor to prevent catastrophic forgetting . By using a 'moving anchor' that slowly follows the student policy, doesn't this design remove the primary safeguard against significant policy drift?\n3. The paper presents Co-rewarding-I (data-side) and Co-rewarding-II (model-side) as two 'orthogonal' instantiations of the core framework. Given their complementary nature, was a combined 'Co-rewarding-I+II' method ever explored?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper's claims are supported by comprehensive experiments. The authors validate their method's effectiveness across a diverse range of models (Qwen series, Llama-3.2-3B-Instruct)and multiple training datasets (MATH, DAPO-14k, OpenRS). The evaluation is similarly thorough, spanning not only in-domain mathematical reasoning but also out-of-domain tasks like code generation and general abilities (MMLU-Pro, IFEval) . This extensive validation strongly supports the paper's conclusions.\n2. A key strength of this work is the principled and general nature of the proposed Co-rewarding framework. The two instantiations are not ad-hoc fixes but are well-grounded in established, sound concepts: Co-rewarding-I draws its intuition from contrastive learning (analogy-invariance) , while Co-rewarding-II effectively implements a form of self-distillation (temporal invariance).", "weaknesses": "1. Although I like the idea of Co-rewarding-I, it seems not that effective. It relies on additional stronger model to revise the training data but offers marginal improvement in across both training datasets. Sometimes may harm the performance. The author tries to show the effectiveness of rewarding-I with a ablation study but the ablation is not fair in my view point, the fair comparasion should be a model trained on the union of orignal and rephrased instead of training on them seperately since the rewarding-I use both of the datasets.\n2. The ablation study in Table 3 is confusing as it appears to conflate results from different training sets. Specifically, the Co-rewarding-I ablation data (e.g., \"Majority-Voting w/ Original\") matches the MATH-trained results from Table 1 , while the Co-rewarding-II ablation data (\"w/o Updating Reference\") matches the DAPO-14k-trained results from Table 2. This is not explicitly stated in the table's caption or the main text, making a direct comparison difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nuNtfeM6JO", "reviewer": "Reviewer_odXG", "summary": "The paper defines two methods for unsupervised RL finetuning of LLMs for reasoning tasks. The first uses a similarity-based objective across semantic augmentations to train problems; the second uses majority voting across rollouts from a reference policy that periodically tracks the RL policy with an EMA update. The results show that these unsupervised reward signals stabilise training relative to other ground-truth-free baselines and can in some cases outperform using ground truth reward.", "questions": "Why not experiment with a combination of Co-rewarding-I and Co-rewarding-II?\n\nWhy do you think Co-rewarding sometimes outperforms using GT reward? This seems unintuitive and is not currently adequately discussed, only stated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper sets out to define a stable and performant approach to RL finetuning for reasoning without use of ground truth labels and the results demonstrate the effectiveness of the proposed method. Three models are used across in-domain and out-of-domain benchmarks and two separate training datasets are considered. Combined with suitable ablations and analysis of training dynamics (length, reward) this makes for a comprehensive results section.", "weaknesses": "The training datasets used are ones for which ground truth labels are available. It seems important to validate the method in a setting that is better motivated by self-supervised methods (i.e., those without the availability of verifiable rewards during training). \n\nThe written communication of the paper could be improved. There are grammatical errors throughout.\n\nThe related work currently features in the appendix and should be in the main paper text.\n\nThe paper primarily reports pass@1 results, which leaves the reliability of the results uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyMbptiBtR", "reviewer": "Reviewer_pcnh", "summary": "The paper introduces Co-rewarding, a self-supervised RL framework for eliciting LLM reasoning that aims to avoid the training collapse often seen in label-free/self-reward approaches. The core idea is to decouple the reward signal from the current policy's single-view outputs by enforcing invariance views across data-side and model-side. \n\nCo-rewarding-I (data-side) obtains contrastive agreement between semantically analogous (rephrased) questions and the original, using majority-vote pseudo-labels across rollouts to shape a GRPO-style objective $J_{\\text{Co-rewarding-I}}$ with cross-refereed advantages (Eqs. (6)–(8)). Co-rewarding-II (model-side) reuses the GRPO reference model as a slowly updated teacher via EMA (cosine-scheduled $\\alpha(k)$) to generate rollouts and majority-vote pseudo-labels, thus temporally decoupling supervision from the online policy (Eqs. (9)–(11)).\n\nAcross Qwen and Llama backbones, Co-rewarding improves stability and accuracy over recent self-reward baselines (Self-Certainty, Entropy, Majority Voting).", "questions": "1. For I, can you quantify features (lexical diversity, named-entity variation, length) that predict stability across MATH vs DAPO-14k, validating the \"rich background descriptions\" hypothesis?\n2. For AMC/CRUX/IFEval/MMLU-Pro, where gains are small, what failure modes are most common? Would task-aware rephrasing (e.g., code refactorings or spec restatements) strengthen I/II?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The instability of single-view self-reward (reward hacking -> collapse) is articulated and empirically supported, with Co-rewarding showing steadier progress and avoided collapse.\n2. The idea is interesting, differs from (i) internal score rewards (entropy/certainty) and (ii) single-view self-consistency.\n3. The method is simple to implement and seems to be quite practical, it reuses GRPO reference as teacher, and no extra LLM is introduced (beyond rephrasing for co-rewarding-I).\n4. Good experimental results, achieves 94.01% GSM8K Pass@1 on DAPO-14k with Qwen3-8B-Base.", "weaknesses": "1. The paper shows co-rewarding-I collapses on DAPO-14k but II does not; the causes are hypothesized (rephrasing efficacy) rather than directly measured.\n2. The use of Qwen3-32B to generate rephrases may raise a concern of equivalence/label leakage beyond examples; although instructions are provided and case studies look reasonable.\n3. On code/IFEval/MMLU-Pro, improvements are mixed/modest, and there is little task-level analysis (e.g., error modes, solution length vs. correctness).\n4. The mechanisms behind beating GT-reward is not dissected (e.g., improved exploration vs. label noise in GT-reward?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)