Reinforcing General Reasoning Without Verifiers
Authors: Xiangxin Zhou
Zichen Liu
Anya Sims
Haonan Wang
Tianyu Pang
Chongxuan Li
Liang Wang
Min Lin
Chao Du
Venue: ICLR 2026 Poster (2026)
Keywords: General Reasoning
Reinforcement Learning
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: A verifier-free RL method to improve general reasoning for LLMs.

Abstract: The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (**VeriFree**) that bypasses answer verification and instead directly maximizes the probability of generating the reference answer, derived in a principled way from the RL objective. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks.

Published as a conference paper at ICLR 2026
REINFORCING GENERAL REASONING WITHOUT VERIFIERS
Xiangxin Zhou Zichen Liu Anya Sims#1.5, Haonan Wang!+, Tianyu Pang', Chongxuan Lix, Liang Wang2.3, Min Lin', Chao Du Sea La ²UCAS $CASIA
*NUS 5Oxford
RUC
Q Code Link
ABSTRACT
The recent paradigm shift towards training large language models (LLMs) us- ing PrepSeck-RI-Zero-see has led to impressive advancements in code and mathematical reasoning. How- ever, this methodology limited tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering. law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this in- troduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-FR -e-ee--R--Zeo-sste -Zero-sttyle training general reasoning domains, we propose verifier-free method (VeriFree) that by- passes answer verification and instead directly maximizes the probability of gen- erating the reference answer, derived in principled way from the RL objective. We compare VeriFree with verifier- based methods and demonstrate that, in addi- tion to its significant practical benefits and reduced compute requirements, Ver- iFree matches and even surpasses verifier- based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks.
reinforcement learning (RL) verifiable rewards
Qwen3 model performance
Qwen3-8B model performance
70
70
Qwen3-48 48-Base
Qwen3-88 Baan3-8--Base Base
Qwen3 4i--n------ssttttt---e
Ee-Insrrutt-eTTTinkiinn
Qwen3 (ant3-4---niiee EBBinntrruct-Thinking
Qwen3-88 (aar-n----nnsstnn sB-instructtThinnnng
Qwen3 Base
Qwen3-88 Base-Veriteer
Accuracy (%6
50
Qwen3-48-Base BB-Base-VertFree (Ours)
Accuracy (%)
50-
Qwen3-88 Mwenn3-8--Bas--W6 Basa-VariFree (Ours)
40
40
20
20
GPQA
SuperGPQA
Figure 1: The general reasoning capability significantly improves when we apply VeriFree to fine- tune Qwen3 base models on general reasoning dataset. Notably, VeriFree can match or even sur- pass the instruct models and the models RL-tuned from base with specialized LLM verifier.
INTRODUCTION
DeepSeek-R1- ---zer (Guo al., 2025) recently demonstrated that training large language models (LLMSs) using reinforcement learning (RI.) with verifiable rewards can be extremely effective in improving reasoning capabilities. In this RL with verifiable rewards (RLVR) framework (Lambert et al., 2024), the LLM generates a reasoning trace (i.e., chain of thoughts, CoT) followed by a final answer. A rule-based program then extracts and evaluates the final answer, assigning a reward of 1 to the response if the final answer is correct and 0 otherwise. The model is then trained with RL using GRPO (Shao et al., 2024) simplified variant PPO (Schulman al., 2017).
*Euual contribution. *Correspondence Chao Du.

---

Published as a conference paper at ICLR 2026
Verifier-based (RLVR)
As reward signal for model responses
LLM T
if correct
Reward
Ootheriise
Find <'think (end reasoning) und verify LLM amser
Question
<think> reasoning ...,</think>>
LLM Answer
generated remsonnng truce
generated
Verifier
model response
Re ference Answer
VeriFree (Ours)
As reward signal for generated reasoning traces
As weighting for SFT reference answers
LLM Te
Find <'thinio (end reasoning) und patch in reference answer
Question
<think> reasoning
....///hink>>>
fereencce Answer
T
reference
question,
answer
reasoming
generated reasoning
data
patched response
VeriFree enables RI-Zero-style LLM training without requiring access to a verifier. In the case of single correct answer format, VeriFree optimizes exactly the same objective as RI-Zero with a lower variance gradient estimator.
Figure 2:
The simplicity of this approach, coupled with impressive performance improvements in mathemati- cal reasoning tasks, has sparked a wave of follow-up works in this paradigm of RL with rule-based verifiable rewards (Liu et al., 2025c; Luo et al., 2025b; Yu al., 2025), which will refer to the RI-Zero-style training in the following. However, these methods remain limited to domains such as math and code, where rule-based verification feasible. Reasoning is critical far beyond math and coding; however, the difficulty of answer verification in general reasoning tasks poses a major obsta- cle applying this training paradigm broader domains. To address this limitation, we investigate how to extend R1 R1-Zero-style training to tasks where rule-based answer verification is not possible.
natural extension, as explored recent general reasoning works (Su al., 2025; Ma al., 2025), is to introduce specialized LLM as a verifier, similar to the reward model used in RL from human feedback (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022). In these methods, the model-based verifier is queried to determine whether the generated answer is equivalent to the reference answer. Although this approach bypasses the need for rule-based evaluation, it introduces several potential drawbacks (as in standard RLHF): it depends the RI-Zero-style paradigm into optimizing a model-based reward, which makes it vulnerable to reward hacking (Gao et al., 2023), and it adds computational overhead by requiring an additional model to be held in memory and queried during training.
the availability of strong verifier LLM, it converts
In this work, propose alternative: verifier- perifier-free approach that preserves the benefits of the RL paradigm while removing the reliance on explicit rule-based or model-based verification. Starting from principled objective, derive method that directly maximizes the likelihood of generating reference answers. The gradient naturally decomposes into two terms: one analogous to RLVR with likelihood reward signal, and the other resembling to supervised training on reference answers. Fig. 2 illustrates our method, which we term VeriFree, as it does not rely on verifiers.
This approach has several appealing properties. First, when there unique correct answer string, our method is equivalent in expectation to the objective in RLVR, but with lower variance, which can be viewed as form of reward shaping (Ng et al., 1999; Randlov Alstrøm, 1998). Even when multiple valid answers exist, we show empirically that using just one as a reference provides a sufficient learning signal to elicit strong reasoning behavior. Additionally, this framework can be viewed through a variational lens neat way of optimizing over latent reasoning traces.
To make this work in practice, we identify and address several subtle challenges, including effective variance reduction and precise handling of tokenization at the reasoning- answer patching point. We conduct comprehensive ablations to understand the impact of each design choice. We benchmark method across diverse set of general reasoning tasks, and the results are striking: shown in Fig. 1, VeriFree not only matches but often outperforms verifier-based alternatives, while being simpler, faster, less memory-intensive, and more robust.
2

---

Published as a conference paper at ICLR 2026
2 METHODOLOGY
2.1 PRELIMINARIES: VERIFIER-BASHI REINFORCHMENT LEARNING
In RL applied LLMs, the language model is treated as policy we that generates an output autoregressively in response to an input question æ. The goal is typically to optimize xe to maximize a given reward function R(x o):
arg max (. (R(2,o)]
(1)
In RI R1-Zero-style RI., the reward is computed by first parsing the response into a reasoning trace x and final answer y. A verifier then checks against the ground truth reference answer y" and assigns binary reward based on correctness, namely Aeeiiterr (y: " Decomposing the model output as o= (2, we can rewrite the objective in Eq. (1) as:
J x. y²) ( ( æ.*) [Bveiier y }]
which separates the sampling of the reasoning trace and the final answer. To maximize this objective, the model is typically updated using the policy gradient estimator (Sutton & Barto, 2018):
VoJveriter 2z y²)
B g~ (2)) w-- (.).e))
(y y V log (2(2))) Ve log to 2]]
(3)
However, this approach requires evaluating answer correctness via Rvenfier (y: y y² " (y=v') which is often nontrivial. While in domains such as math and code, this evaluation is feasible via rules (Guo et al., 2025; Liu et al., 2025c) or test cases (Gehring et al., 2024), accurate verification in general reasoning tasks is substantially more difficult. As result, recent advances in RI RI-Zero-style training have largely been restricted to verifiable domains, leaving reasoning tasks in general domains under- explored. In light of this, present alternative verifier-free approach which naturally extends this training paradigm to broader reasoning domains.
2.2 VERIFREE POLICY OPTIMIZATION
We begin with the standard objective in Eq. (2) and show that, in the case of a single correct answer, we can derive an equivalent objective that does not require a verifier. Moreover, we compare the gradient estimators of this new objective and the verifier-based objective (Eq. (3)), and demonstrate that our verifier-free gradient estimator has the additional benefit of lower variance.
Starting from Eq. (2) and assuming a unique correct answer such that Reeeiterr y= (== (i.e., exact match rather than semantic equivalence 1l (v=y") )), the VeriFree objective is derived as:
JVerifier (0: x. y²) E Eewaro
E Engost (1laaa)) (N=E²)
E (x
To Te(yle |x, 2) (y=yr)
= =Exwao(112))
To 2o |x. 2)
JVerilree (0; x. y²).
(4)
Ressttee (2:00 (==a0,3))
This can be interpreted as follows: if only one answer y" is correct and receives a reward of (while all others receive 0), then the expected reward given reasoning trace z can be computed directly as the probability assigned to y", effectively marginalizing out y. The corresponding gradient estimator given by (see Appendix B. for full derivation):
V Vo (0: (0: v2) Ez Averitree (z: x. b*) V log (=(2))) + Va log Ne (e (y [x,
(5)
reasoning term
reference answer term
Both the objective and its gradient estimator (Eq. (4) and (5)) are equivalent in expectation to their verifier-based counterparts (Eq. (2) and (3)). Intuitively, the "reasoning term" in Eq. (5) can be
We use '=' to denote semantic equivalence, where multiple expressions can be judged correct. For exam- plc, y '8/5', then *8/5", '1. 6', and "\ (FFac(8)(5)) are all considered correct.
3

---

Published as a conference paper at ICLR 2026
Verifier-based (R1-Zero)
VeriFree (Ours)
Model generates the reasoning trace and y. Extract the answer y.
Model generates the reasoning
Patch the correct answer y²
Check answer using a verifier.
Evaluate probability To [z x).
Reward Rvaiser 1 if correct, 0 otherwise.
Reward Averfree #o |2 x).
Train with gradient estimator V (Eq. 3).
Train with gradient estimator V. (Eq. 5),
Figure 3: pseudocode- comparison VeriFree (ours) and the standard R1-Zero approach.
interpreted as policy gradient where the reward for reasoning trace (i.e., RvenTree (2; the probability that the policy will generate the correct answer y" given z, while the "reference answer term" can be viewed reward-weighte supervised learning term for y given z. We will further elaborate on this interpretation in Sec. 2.3. addition to bypassing the need for a verifier, VeriFree gradient estimator also benefits from reduced variance:
Theorem 1. (Variance Reduction) Let Vetiieer (I, y 2, y) and Gvenrree xz, y" denote the single-sampl Monte Carlo estimators of V VeJVeritter and V Venirree given a and y", respectively. Then we have
Var Φ)) Veerrree (x, y², 2)] < Var=~wol (1) (eeiiiier (x. ,2 v..
(
(6)
This reduction in variance arises from Rao- Blackwelllzttiionn (Casella Robert, 1996). For intuition, the variance in the Monte Carlo estimate of VoJVerifier stems from the randomness in sampling ~ Te(- (x) and y ~ (-12;, z), while for estimating VaJVerrPree we analytically marginalize out y,
thereby removing this source of randomness. We provide full proof in Appendix B.2.
Our gradient estimator Va (0; æ y/) is fully compatible with other variance reduction tech- niques, including RLOO (Ahmadian et al., 2024), GRPO (Shao et al., 2024) reward normalizations, and the PPO (Schulman et al., 2017) clipping operation. As such, we sample multiple responses for each prompt and apply the RLOO baseline to the reasoning term in Eq. (5). We also adopt the cor- rected response-length normalization from Liu et al. (2025c). The final on-policy gradient estimator is follows:
V Jventree (0)
Σ A Vo (olg))o(z;(z) + R. Vo log "o x, 2;)]
where 2; ~ (-(12))), Ri To(y" x 2;), and A, = Toly" zi) - Ejjs To 10, 2;); We also provide the PPO-based off off-policy variant in Appendix
2.3 COMPARISON TO EXISTING APPROACHES
There have been two main prior works that, although derived from different perspective, arrive related alternative gradient estimators: JEPO (Tang et al., 2025) and LaTRO (Chen et al., 2024b).
rensoning term
answer term
V Volivriiierr Ez,y
1 (y=y')
V To(2 +
(y=y')
V log Te (y)
(R1-Zero)
reference answer term
V V Jveitriee
= Eg]
To To(y" x z)
Vo log no To +
TT " x z)
log To x)
(Ours)
VaJJEPO Ez]
log (y/ [x,
V log ne (z)
Vp Tog [x)
(JEPO)
VoJLO Ez
(log Te
(
xx, 2)
log =e(=( )u
log (2( +
V log T x x)]
(LaTRO)
Both JEPO and LaTRO treat the reasoning trace latent variable and extend the standard super-
vised learning objective (log-likelihood) to optimize lower bounds on log( Ez [Te(y" læ, =)]) and log(Ez-ma((1)) [>o(y" "y""a (z, >)),, respectively. The primary difference is that JEPO samples z from the learned policy me, while LaTRO uses fixed reference policy met. Despite originating

---

Published as a conference paper at ICLR 2026
from different perspectives, these methods arrive at similar gradient estimators, shown in above comparisons. However, as reported in Tang et al. (2025). these verifier- free, variational-inferee based methods consistently underperform the standard verifier-based R1-Zero approach. In contrast, our method matches or outperforms the verifier-based baselines. We also include experimental com- parison with JEPO and LaTRO in Appendix F.2.
One possible explanation is that our method exactly recovers the original verifier-basee objective un- der the äinle-correct-answe assumption, whereas JEPO and LaTRO optimize subtly different ob- jectives. For example, JEPO effectively uses a reward of It log नe (y" [ z), , as highlighted in the gradient expressions above. Another distinction lies in the weighting the "reference answer term" Ve log (yy x x). In our method, this term is weighted by the probability xe To (y, z). which is the likelihood of the reference answer given the sampled reasoning trace. In contrast, both JEPO and LaTRO use fixed weight of 1, thereby increasing the probability of y" regardless of the qual- ity of the reasoning trace 2. We hypothesize that this behavior could promote poor reasoning. For instance, suppose the model generates the reasoning trace" " minus apples, finally resulting in a total of apples" when the correct answer is "6". The JEPO and LaTRO objectives would still push the model to output "6" from that flawed trace, reinforcing mismatch between reasoning and answer. Our method avoids this by Nown-weighting contributions from low-quality traces. Due to space constraints, more related work is deferred Appendix A.
2.4 HOw TO HANDLE THE TOKENIZATION PATCHING POINT?
A critical consideration when extracting reasoning traces z from model responses (z, y) stems from the fact that LLMs operate on token sequences, not raw text strings. While human-readable out- puts (e.g..
<answer> \boxed(.... ... </answer>" as in Template 1) suggest splitting reasoning traces # at specific text patterns like "<answer>", such text-based splitting strategy may cause tokenization inconsistencies. For example, the ">" character might be tokenized differently depending on its surrounding context in versus y". While one potential solution is to introduce special tokens to enforee consistent tokenization boundaries, these novel tokens could harm model performance due to their absence from the base model's 's pretraining vocabulary.
Instead, we resolve this by defining the end of the token corresponding to <answer" (i.e.. without ">"), leveraging the fact that the pattern "r>" does not appear in standard tokenizer vocabu- laries. This ensures consistent token-space alignment between sampling and optimization, avoiding instability due to off-policy mismatches (Yao et al., 2025). Notably, this approach is operationally equivalent to setting " "<answer" (instead of "<answer>") as the stop word during sampling, mechanism natively supported by modern inference engines like vLLM (Kwon et al., 2023). In this case, we can sample reasoning traces z directly, instead of first generating the full response (z, y) and then extracting post hoc.
3 EXPERIMENTS
In this section, we empirically evaluate VeriFree across multiple model scales and reasoning bench- marks. We show that VeriFree improves general reasoning capabilities over verifier-based base- lines, achieves higher learning efficiency, transfers to math reasoning tasks without math-specifie supervision, and benefits from its key design choices, including tokenization-au reasoning trace extraction and the RLOO variance reduction technique.
3.1 SETUP
Training. Following the "Zero" setting widely adopted in recent work (Guo et al., 2025; Hu et al., 2025; Liu et al., 2025c; Ma et al., 2025), we directly fine-tune the base LLM, skipping the intermedi- ate stage of supervised fine-tuning (SFT). We implement our training pipeline using Oat (Liu et al., 2024) by instantiating their base modules and incorporate our algorithmic changes. Our experiments are conducted using Qwen3 (Team, 2025) base models across multiple scales, including 1.7B, 4B, and 8B parameters. We adopt the prompt template shown in Template 1. We do not employ KL regularization losses KL penalties in rewards, recent studies suggest that removing KL terms does not have a significant impact (Liu et al., 2025c: Hu et al., 2025). As result, our method does not require maintaining reference model in memory.
5

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy comparison on the MMLU-Pro benchmark.
Method
Len.
Avg.
CS
Math
Chem
Eing
Law
Bio
Health
Phys
Bus
Phil
Bcon
Other
Hist
Qwen3-1,7B-B8 B.B-Bee
618
33.3
34.6
38.9
32.2
21.0
17.3
56.1
33.5
32.0
38.5
21.8
45.7
28.4
44,4
21.0
Qwen3-1.7B (wlo thinking)
893
46.1
49.5
64.4
48.0
35.9
22.9
64.9
38.0
49.7
53.5
33.7
53.9
36.4
51.6
31.2
Qwen3-1.7B (w/ thinking)
56.1
76.4
57.6
27.0
21.9
67.9
47.7
57.5
38.9
64.5
32.3
Qwen3-1.7B 1...B-BBBee-eriffi
875
47.0
48.8
64.4
52.7
38.1
18.7
62.9
41.2
51.9
54.9
31.9
55.2
38.6
53.3
30.2
Qwen Be---BBBB--eriieeee
856
46.9
46.8
64.1
51.7
41.8
20.0
64.0
39.7
52.1
55.6
29.5
53.1
37,5
53.0
29,9
Qwen3-4B-Bar
825
47.2
42.9
67.1
55.5
40.0
22.5
56.9
43.6
55.4
54,9
27.5
52.7
34.3
48.6
34.7
Qwen3-4B (w/o thinking)
838
60.0
65.9
79.1
65.8
45.7
29.0
76.6
57.0
65.1
66.7
48.9
69.2
52.1
64.3
44.6
Qwen3-4B (w/ thinking)
3456
62.7
70.0
84.8
66.6
38.6
28.7
81.3
60.4
67.4
69.2
53.7
75.1
57.8
67.9
49.6
Qwen3-4B-Ban -a--aa--eeriiffeer
921
63.0
66.1
81.3
69.7
52.8
29.1
79.8
62.8
67.6
71.2
48.5
73.1
52.8
68.5
45.4
Qwen 3- VeriFree
64.4
82.2
55.6
30.7
81.7
59.2
71.7
53.4
47.5
Qwen2.5-7B
519
47.8
48.3
59.5
44.4
33.4
25.1
63.6
50.4
48.0
55.9
34.7
60.6
46.0
58.2
38.3
Qwen2,5-7B-Sime S-TBB-SimleeR-ZZo
705
51.2
51.2
52.0
50.2
40.8
30.5
69.5
54.3
52.5
57.3
41.9
62.8
52.6
60.8
42.3
Qwen2.5 Mete2.5-Math--1 NarMaahhPPP (a-h-7B-00 7B-O7t Zero
556
40.5
47.6
47.7
46.9
32.1
18.1
53.6
25.7
49.4
52.9
29.5
54.7
32.8
43.0
22.8
Qwen2.5-7B-Instee
481
55.3
56.6
70.4
55.6
42.7
29.8
69.3
55.1
57.9
63.5
41.5
63.4
53.6
62.4
43.6
General-Reasonerr
867
63.4
73.7
63.3
44.9
35.2
72.0
56.6
61.5
66.7
43.1
68.1
52.8
62.8
47.8
Qwen3-8B
75.0
66.2
46.7
75.9
60.4
48.7
69.0
54.3
47.2
Qwen3-8B (w/o thinking)
1032
61.9
65.6
71.
62.8
462
34.7
79.9
66.1
63.7
55,9
72.9
58.9
52.5
Qwen (w/ thinking)
3952
66.9
71.5
83.8
68.0
38.7
39.
85.2
72.1
69.8
73.3
57.5
79.2
66.3
71.8
57.7
Qwen3-8B Base Veriferr
594
65.9
63.9
81.8
71.1
56.9
35.
81.9
64.9
71.6
74.1
53,9
74.2
58.4
68.4
54.3
Qwen 3-en3-8B-BBe B------BBBBB------ VeriFre
776
67.2
715
85.3
73.5
55.7
37.3
81.9
64.3
73.1
74.1
54,9
74.8
59.6
67.7
54.6
Template 1 (for Ours).
<I <limsttart | | (quest(itu)tnennene laPlesse reason step by step, and put your final answer within <answer> \[boxe[[] < </answer>. < immend >>\<<| im.start |nssissantt
For the 1.7B and 4B models, we conduct fine-tuning for approximately 4,000 policy gradient steps; for the 8B models, we fine-tune for around 3,000 policy gradient steps. During each step, the policy model (i.e., the LLM) generates 8 responses for each question (i.e., group-size=E) with 16 questions processed per step. We use the sampling configurations temperature=1. O, top-p=1, and max tokens=3000 for the rollout process. The responses are then parsed into reasoning traces and model- answers from the training dataset. Subsequently, single forward pass is executed to compute the conditional probability of the reference answer, conditioned on all preceding tokens including the prompt and the reasoning trace. This procedure introduces only minimal additional computational cost, as the forward pass of the LLM does not require autoregressive decoding and does not require storing intermediate states for backpropagation. All collected samples from each step are used for one optimization step. The training is conducted on a single node with 8xH100 GPUs.
answers. We replace the model-predicted answers with the reference
Dataset. To support general reasoning, we begin with the dataset curated by Ma al. (2025), sourced from WebInstruct (Yue et al., 2024). To improve data quality and reliability and reduce size, we retain samples with answers that consist of fewer than seven tokens, and use Qwen2.5- 72B-Instruct (Yang et al., 2024a) to filter out low-quality and noisy data. This process results in approximately 61,000 data samples spanning diverse domains, which we refer to WebData. The category distribution is visualized in Fig. 7.
Evaluation. line with prior work (Ma et al., 2025), we employ multiple-choice questions for eval- uation to facilitate verification. To assess general reasoning abilities, we utilize the following bench- marks: MMLUU-Po (Wang et al., 2024), a challenging multi-task understanding benchmark de- signed to evaluate the capabilities of LLMs across various domains; SuperGPQA (Du et al., 2025), a large-scale benchmark consisting of graduate-level reasoning questions spanning 285 diverse disciplines; and GPQA-Diamond (Rein et al., 2024), which focuses on graduate-level question- answering and is designed to resist shallow pattern-matchine matching and memorization. While our primary focus is not on enhancing mathematical abilities, we also evaluate math reasoning using a suite of standard math reasoning benchmarks. This suite includes MATH-500 (Hendrycks et al., 2021), OlympiadBench (He et al., 2024), Minerva Math (Lewkowyez et al., 2022), GSM8K (Cobbe

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy comparison the SuperGPQA benchmark.
Method
Len.
Avg.
Eng.
Med.
Sci.
Phil.
M.S.
Econ. Mgmt, Socio.
L/A
Hist.
Agron.
Law
Edu.
Qwen3-1.33 1.7B Base
997
17.4
17.7
18.6
16.0
27.4
27.3
20.5
22.4
23,1
15.5
11.1
18.1
20.6
21.3
Qwen3-1.7B (w/o thinking)
1152
23.3
22.6
22.8
24.3
30.3
31.
24.3
27.5
23.8
19.0
18.1
20.8
24.5
28.1
Qwen3-1.7B (w/ thinking)
4799
23.6
21.8
25.3
23.6
33.1
33.7
29.6
27.5
32.2
19.0
18.0
25.0
26.2
31.6
Qwen3-1.33 1.7B Base Verifier
1049
24.5
26.0
23.9
24.4
30.8
26.8
26.9
27.0
26.6
18.9
16.6
22.3
22.6
27.3
Owen3-1.7B-B1 BB-BBBBBB----- VeriFVeriFree
964
24.8
25.7
24.7
24.9
26.5
30.2
25.9
27.9
28.0
20.4
15.9
22.9
25.0
28.9
Qwen3-4B-Ban
902
24.7
25.7
23.6
26.0
23.6
25.4
28.8
28.4
19.6
16.4
16.8
20.6
25.6
24.0
Qwen3-4B (wlo thinking)
1397
31.6
32.0
31.5
32.3
37.5
36.1
37.8
33.7
33.6
24.3
20.6
28.3
31.4
33.5
Qwen3-4B (w/ thinking)
4568
31.7
30.7
33.2
32.1
41.2
31.7
41.7
35.9
32.9
24.5
22.4
30.9
35.7
35.3
Qwen3-4B- Swen3-4B-Base- Verifier
1045
34.3
35.4
35.5
34.5
39.2
41.0
39.1
36.7
37.1
26.6
22.3
33.8
33.1
35.3
Qwen3-4B Boonn--B-BBse--- Base- VeriFree
1451
35.1
36.3
34.5
36.9
35.7
37.1
39.1
38.3
31.5
24.7
22.0
33.0
33.2
34.1
Qwen2.5-7B
716
23.8
27.0
21.8
28.8
31.2
27.6
29.1
22.4
20.8
20.2
24.5
27.4
30.2
850
26.3
26.4
30.5
23.8
32.6
32.2
33.0
31.9
28.7
24.1
21.4
27.2
29.6
32.9
638
21.3
23.1
16.4
21.5
23.1
21.5
25.9
27.2
25.2
17.7
15.9
21.4
18.8
24.8
1aan2.5-7B-nnsrruee
604
28.4
27.7
32.2
27.6
33.7
32.2
32.4
32.9
32.9
24.5
22.1
29.7
30.6
32.4
General- Reasoner-7B
1047
30.8
31.5
32.2
29.9
35.2
41.5
38.4
33.1
35,0
25.5
22.7
28.9
32.5
35.5
Qwen3-88 8B-Base
825
31.0
31.3
34.0
30.6
36.0
37.1
34.7
37.5
35.0
24.2
20.0
28.5
31.4
36.4
Qwen3-8B (wl thinking)
1638
32.4
32.6
36.5
312
39.5
42.0
37.7
37.3
38.5
25.0
22.6
33.2
34.3
38.4
Qwen3-8B (w/ thinking)
4995
35.0
33.6
42.1
33.5
44.4
37.6
44.2
42.7
42.7
27.9
24.9
37.9
38.7
40.7
Qwen3-8B Base Verifier
713
37.1
38.2
39.5
37.2
39.5
39.5
43.0
40.1
38.5
28.9
24.8
34.2
34.8
38.2
Qwen3-88 Base VeriFree
38.0
38.3
39.1
39.6
37.5
41.7
28.6
33.6
al., 2021), AMC and AIME24 (Li et al., 2024). We utilize Math-Verify? check for an- swer equivalence. Except for AIME24, where we employ a temperature=1, and repeat each question 32 times, all other benchmarks are evaluated using temperature=O. O. We use max wow.:tokens-B88 92 for all evaluations. For the multiple-domain benchmarks (MMLU-Pro and SuperGPQA), we report a micro-average. This metric computes a global average by aggregating contributions across all classes, rather than calculating macro-average (which would average per- class scores and assign equal weight to each domain).
Baselines. Our primary baseline, denoted Verifier, is verifier-based approach using the verifier from Ma et al. (2025). The verifier is initialized from Qwen2.5-Math-In. and fine-tuned data generated by Gemini 2.0 Flash to assess equivalence between the reference and predicted answers, conditioned on the question. We apply Dr. GRPO (Liu et al., 2025c) as the optimization algorithm for the baseline, ensuring that all other settings consistent with our approach. Following Ma et al. (2025), the reward definition incorporates additional factors beyond verifier correctness, including format compliance and the length of generated answers. If the format is incorrect (e.g., missing \ [booxed{) } in the model response), a negative reward of -0.5 is applied. Moreover, length penalty of -0. 05 x min (10, abs (Length.of.core www.correet...sse.......... ansser is added.
(Yang et al., 2024b)
We also report the results for the base and the instruct models of Qwen-3- Bwwen-3-1.7B44B/S3 (Team, 2025) and Qwen2.5-7B (Yang et al., 2024a), as well as the checkpoints released by Qwen2.5-7B. SimpleRL-ZZoo (Zeng et al., 2025), Qwenn255-Mtth7B--Ott Reasoner-7B 7B (Ma et al., 2025).² Notably, Qwen3 integrates both thinking mode (for complex, multi-step reasoning) and non-thinking mode (for rapid, cortext-driven responses) within a unified framework. We report results of both modes on Qwen3 instruct models.
(Liu al., 2025c), and General-
3.2
MAIN RESULTS
VeriFree improves general reasoning capabilities. We begin by evaluating the effectiveness of VeriFree in enhancing the general reasoning capabilities of LLMs using the MMLU-Pro and Su- perGPQA benchmarks. Table presents a detailed comparison across model scales and domains in the MML.-Pro benchmark. Starting from base models, applying RL with VeriFree yields substan-
Phtpps: / /1 thub. www/hugg.iie.coooo ionggiingface face MatthVerriy
'Qwe22.5 and Qwen3 use diffeerntt naming conventions in their official release. For base models: Qwen2.5 has no suffix (e.g.. Qwen2.5-7B) whereas Qwen3 adds "-Base" (e.g., Qwen3-88 8B-----BBBoo BB-e). For instruct models: Qwen2.5 uses "IIntruut"" (e.g., Qwen2.5-7B- 7-- while Qwen3 omits the suffix (e.g., Qwen3-8B). We follow these conventions consistently in the paper.
7

---

Published as a conference paper at ICLR 2026
MMLU-Pro Performance
Eval Acc. vs Model Confidence
66
MMLU-Pro Accuracy (%)
0.82
Accuracy (%)
64
64
62
62
VeriFree Verifier
60
500
1000
1500
2000
2500
3000
0.15
0.20
0.25
0.30
0.35
0.40
Training Steps
Model Confidence 7yly |x,
Figure 4: Left: MMLU-Pro accuracy of VeriFree and the baseline fine-tuned from Qwen3-8B base model along training steps. The curve is smoothed by moving average with interval of 384. Right: The dynamics of MMLU-Pro evaluation accuracy and average model confidence To(3) =) along training based on Qwen3-8B base model. Raw data points depicted with more transparency, while smoothed data using a Gaussian filter is shown with less transparency for emphasis. Darker colors represent larger training steps.
tial gains in average accuracy (ranging from 12% 40%), demonstrating that VeriFree effectively fine-tunes LLMs to improve general reasoning performance. See Appendix E for more results.
Similar improvements observed the SuperGPQA benchmark, shown in Table 2, where VeriFree consistently enhances the performance of base models by significant margin. Notably, VeriFree achieves performance comparable to, or even surpassing, that of the instruct model in thinking mode and the Verifier baseline (i.e., the RL-tuned model learned with an additional model- based verifier), without relying on any explicit verification signals.
In addition to accuracy gains, we also observe an increase in response length after tuned by VeriFree, suggesting that the model explores longer reasoning traces to arrive at more accurate answers. which is a behavior reminiscent of DeepSeek-R -7er (Guo et al., 2025). Results on the GPQA benchmark are provided in Appendix E. due to space constraints.
VeriFree leads to better learning efficiency. We compare VeriFree with the baseline that learns from a model-bbsed verifier reward (i.e., Verifier). As shown Fig. (Left), VeriFree consistently outperforms the baseline, achieving higher accuracy with fewer training steps. We attribute this improved learning efficiency to reduced gradient variance, enabled by VeriFree's 's continuous reward signals and the RLOO objective. While both approaches optimize the same reward signal in expecta- tion, VeriFree provides more stable and informative policy gradients, which accelerate convergence and leads better final performance.
Model confidence good reasoning capability proxy. Our analysis based on Qwen3-8B base model reveals strong positive correlation (p 0.82) between MMLU-Pro accuracy and the average model confidence To(y" je, z) during training (Fig. 4, Right). This empirically demonstrates that the model's self-estimated confidence in the correct answer, i.e., To (y" [x, 2), serves an effective metric for quantifying emergent reasoning capabiliiiess in LLMs.
VeriFree learns transferable reasoning skills. To evaluate the transferability of rea- soning acquired through VeriFree, we train model on a dataset with all math-related examples removed, and assess its perfor- mance on both general and math-specific benchmarks. As shown in Fig. 5, Ver- iFree not only improves reasoning perfor- mance on general tasks, as expected, but also demonstrates strong transfer to math benchmarks desippt the absence of math supervision during training. This highlights VeriFreee's i ability to induce general reason- ing capabilities that extend across domains.
Evaluating Reasoning Transferability
70
Qwen
Qwenn
Score (%0
50
40
20
MMILL-PP
GPDA
SuperCPQA
Math-Eval-Sute
Figure VeriFree enhances reasoning transfer to math without math training. When trained only non-math data, the model improves on general bench- marks and effectively transfers to math-specific tasks.
8

---

Published as a conference paper at ICLR 2026
MMLU-Pro Accuracy
Effects of Equivalent Class
46
Swen3-1.78-Ba5e
80
Owen3-1.78-Verfie
44
Owen3-1 Bpven--1.7--orrre 71--Vereee w/ Equiv.
Accuracy
42-
Score (%)
660
40
50
40-
36
VeriFree VeriFree RLOD
30
34
Verifree w/o taken split
0
500
1000
1500
2000
2500
3000
3500
GSMBK
MATH-500
Minerva
Olymplad
Steps
Figure 6: Left: MMLU-Pro evaluation accuracy of VeriFree against ablation variants (w/o RLOO, w/o tokenization-aware split strategy) along training steps. All models are based on Qwen3-1.7B base models. Right: Effects of introducing the equivalent class to VeriFree on model performance.
3.3 ABLATION STUDY
To systematically evaluate method components and offer comprehensive understanding Ver- iFree. we conduct ablation studies based on Qwen3-1.7B base models as follows.
Effects of extraction strategy for reasoning trace z. Our method requires precise separation between reasoning path z and answer y to enable answer replacement. While human-readable splits using <answer> seem intuitive, we instead define z to end at "<answer" (omitting ">"), ensuring consistent tokenization boundaries (see Sec. 2.4). We compare with variant using text- based splitting (denoted as "VeriFree w/o token split") on Qwen3-1.78 via MMLU-Pro (Fig. 6). Our tokenization-awee aware approach achieves superior convergence, while the variant suffers optimization instability due to effectively introducing off-polic data.
Effects of RLOO.. As in Fig. 6 (Left), removing RLOO leads consistent drop in performance throughout training, with final accuracy more than 3% lower than that of the full method. This high- lights the importance of RLOO in stabilizing learning and guiding the model toward better general- ization. Without RLOO, the model converges prematurely and fails to reach the same peak accuracy.
Effects of equivalence class. As mentioned in Sec. 2.1, verifier-based RL typically assesses answer correctness as rewards. Correct answers within specific class often form an equivalence class. Our method, however, utilizes model confidence by focusing on a single reference answer for given question and the model's reasoning trace. To explore the potential advantages of integrating an "equivalence class" into our approach, we conducted ablation studies as follows. We employed model fine-tuned on the MATH-126 dataset (Hendrycks et al., 2021; Lightman et al., 2024) from Qwen3-8B base model through Dr. GRPO (Liu et al., 2025c) with rule-based verification to sample answers MATH-12K, subsequently verifying answer correctness using Math-Verify, This ap- proach enabled us to create an extended dataset with a set of equivalent correct answers for each question. We then fine-tuned Qwen3-1.7B base models using method on both the original and the extended MATH- 12k datasets incorporating equivalence classes. These models are eval valuuaed on GSM8K, MATH-500, Minerva Math, and OlympiadBench assess the impact of including equiv- alence classes. The results, shown in Fig. 6 (Right), indicate that considering equivalence classes offers slight performance improvements, aligning with our expectations. This highlights a minor limitation and motivates future work on algorithms that can better leverage answer equivalence.
4 CONCLUSIONS
In this paper, we rethink reinforcement learning with verifiable rewards (RLVR) for LLMs from novel perspective. By leveraging the gradient equivalence under the unique answer assumption, we derive a new optimization objective that eliminates the need for explicit verification, whether rule- based or model-based Our proposed method, VeriFree, is particularly well-suited for general rea- soning tasks, where rule-based verifiers are infeasible and model-based verifiers are both expensive and vulnerable to reward hacking. Through extensive experiments and ablations, we demonstrate the effectiveness and robustness of VeriFree on wide range of general reasoning benchmarks. We hope our work offers a fresh viewpoint for the LLM RL community and provides a practical approach for building future general-purposs reasoners.

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in Ilms. arXiv preprint
2024.
George Casella and Christian P. Robert. Rao-blackwellisationn of sampling schemes. Biometrika, 83 (1):81-94, 1996.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang. Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. In International Conference on Learning Representations (ICLR), 2025.
Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or Ilms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1301-8322 2024a.
Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arX2:2241.004822, 2024b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and
Gabriel Synnaeve. Rlef: Grounding code 1lms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02088) 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint 1.XXv:2551112944 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.0700 07006, 2025.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: : SRLLtttp///opeeeeeiee //ppeneevee....
net. forum? d=Ou J 6p4ca
10

---

Published as a conference paper at ICLR 2026
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner zeeoooer--zeoo An open source approach to scaling up reinforcement learning on the base model, 2025. URL https: / wwwaa.iiiv..oom org/ abs 2////553..2222
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.01951, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tạ" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXhv:2411.15124 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https: oppnrevie...
Qianchu Liu, Sheng Zhang. Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul Vozila, Tristan Naumann, and Hoifung Poon. X-reasoner: Towards gener- alizable reasoning across modalities and domains, 2025a. URL hwttttss//aaii..ooot 25C5. 03981.
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research-friendly frame- work for Ilm online alignment. https: : / htt.ttt....coooooooo com/ sa - -g// oat, 2024.
Zichen Liu, Changyu Chen, Wenjun Li. Tianyu Pang. Chao Du, and Min Lin. There may not be aha moment in ri-zero-like training - pilot study. https: / oat 1 11m notion. stttiiinsse//// site/ zer- 2025b. Notion Blog.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl ri-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025c.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa. and lon Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025a. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing op-prvee with a 1.5b model by scaling rl, 2025b. Notion Blog.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint arXiv:2505. /4652, 2025.
A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transfommation:: Theory and application to reward shaping. In International Conference on Machine Learning, 1999. URL https: //api..pp..// semant icschol ar.
66.

---

Published as a conference paper at ICLR 2026
OpenAl. Learning to reason with 1lms, 2024. URL -eaminng--o-teaaooo-------
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. http://giwbbb.oo//aaiiiaaaaaaa/a
2025. Accessed: 2025-01-24.
Du Phan, Matthew Douglas Hoffmman David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36:72819 19-7241, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-3 -37411, 2023.
Jette Randlov and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. pp. 463 471, 01 1998.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding with verifiable rewards across diverse domains. arXiv preprint 2.X2:22003.2829, 2025.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Yunhao Tang, Sid Wang, and Rémi Munos. Learning to chain-of-thought with jensen's 's evidence lower bound. arXiv preprint arXiv:2503.19613 2025.
Qwen Team. Qwen3, April 2025. URL httttttt///gennnaae github. wttpuub....oooo/////e///loo
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint 25li:25006..0112.. 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 252::2412.15//55 2024a.
12

---

Published as a conference paper at ICLR 2026
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self eefloomprveeeett. arXiv preprint arX1v:2409.12122, 2024b.
Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. URL https:
/ //fengyao. notion. wtwte//fffppocc---ee
Jiayi Ye, Yanbo Wang, Yue Huang. Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases LLM-as-a-judge. The Thirleenth International Conference on Learning Representations, 2025. URL https: /ttp///eeeee..../////////. net/ forum? witmmmiid=ittaaa 2FITaPFFaa 2Fia jM
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. International Conference on Machine Learning, 2024. doi: 104485500arXXx22201100022
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 challenging questions. arXiv preprint arXiv:2502.13122 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.00 01000 2023.
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629-996 90660, 2024.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejum Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning both effective and efficient.
https: htappps:///kuute /ttp/s/uutttnnnn not not ion, site/ett imm er reesson, 2025. Notion Blog.
Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909v 08593, 2019.
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint 250.::2504.10884, 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A RELATED WORKS
DeepSeek-RR -i-eeekk----ee---ike reinforcement learning. DeepSeek-RI-Zeno (Guo et al., 2025), Tülu (Lambert et al., 2024) and OpenAI's ol (OpenAl, 2024) recently demonstrated that applying RL to learn directly from binary verification-based rewards can be extremely powerful in enhancing the reasoning capabilities of base LLMs. Since then several works have reproduced RI--ero-lik training smaller scales (Zeng al., 2025; Pan et al., 2025; Liu et al., 2025b; Hu et al., 2025; Luo et al., 2025b;a). The aforementioned works all focus on math and coding, domains where verifi- able rewards readily available. By contrast, work aims to extend the R1-Zero-like training paradigm to general domains where verifiable rewards are not available.
Reasoning beyond verifiable domains. Previous work reasoning without access to verifiable rewards has been based around employing an additional LLM to act as proxy verifier or reward model. NaturalReasoning (Yuan al., 2025) introduces large, multi-domain dataset and presents baselines trained using RFT (Yuan et al., 2023) and DPO (Rafailov et al., 2023), leveraging a second LLM reward model, while Su et al. (2025) and General-Reasome incorporate a separate LLM to serve as verifier. X-Reasoner (Liu et al., 2025a) also investigates general reasoning but circumvents the lack of rule-based verification by dropping the R1-Zero-styb training paradigm, instead training via SFT (Grattafiori et al., 2024) responses sampled from more capable models.
(Ma al., 2025) similarly
Self-improving language models. Several works have explored training LLMs using signals based on the model's own outputs. Yuan et al. (2024) propose to prompt the model to judge and rank dif- ferent responses and select the best and worst for preference learning. Chen et al. (2025) leverage the DPO implicit rewards for more efficient and robust self-alignment via iterative DPO (Rafailov et al., 2023). Zuo et al. (2025) use majority voting construct self-labeled rewards for RL to further im- prove dell-trained models during test time, which can be understood as a form of sharpening (Huang al., 2024). The aforementioned works differ from our approach. They do not require reference an- swers or labels, whereas our method requires reference answers in the training dataset. Furthermore, as shown in Sec. 2.2, it is proven that VeriFree is equivalent to RLVR under the single reference an- swer assumption. Another line of research (Phan et al., 2023; Chen et al., 2024b; Tang et al., 2025; Hu al., 2024) approaches LLM reasoning from the direction of variational optimization, treating the reasoning trace as a latent variable. Despite starting from different viewpoint our method has interesting and close connections to this perspective which we discuss detail in Sec. 2.3.
B THEORETICAL ANALYSIS
B. DERIVATION GRADIENT ESTIMATORS
Here we provide derivations of Eq. (5) for the gradient estimator of VeriFree. We also include the corresponding derivation for the standard verifier-based gradient estimator for completeness.
Proof. The gradient estimator for J is derived follows:
V o Jveriiier (0; x. y²)
V Vol. Re
zooe (+1x=)
E Eynna [ (y:: y
V
Σ Reveiiier ( 2/" ( 2=)0())))
2.8
Σ
Rvectteer y yn (oro (o(l)
2.M
Σ
R (y: (olll) (210 wtw.
2.3
E B x)) Bynmo(((2)
Rventter (y::e) (=| lolog
14

---

Published as a conference paper at ICLR 2026
The gradient estimator for J Verifree derived as follows:
V J Verifree (6; xz y)
Vo o B Bo (-12e) (2;
V Ve (-12)) To
x 2]]
V Toly" x 2T
To (3/" xx V (all))/)))
(To (yy x., =) log (00))))))) x =) log To
(
To x) Ve To V oo
Ex (1)
(x mo Volog [>]]
B.2 PROOF OF LOWER VARIANCE
Here provide full proof of Theorem 1, the reduced variance property of VeriFree. We show that the policy gradient estimator derived from Jenniree (0; at, y has variance less than or equal to that of the estimator derived from Neriiterr æ. y²) for any given æ, y" The same relationship will hold for the global objectives averaged over (æ, y²) D.
Proof. The global objective functions are:
Veritizr (f) = E( 1D
E E )- By
= y" [1 (1
Jverirree (0) Fe DDD
B (x To (zz, [=]]]]]]]]]
For given (x, y*) D, the single-sample Monte Carlo gradient estimators are:
Överiteer (x, (o,y". y², v) 1 1 y" Vo log (=(xx) + Vo log log
where: 2 T)() y To( [x. 22) and
&ventrree x. v. 2) To x. 2)
log (2(2) + Vo log (v 2)]
where z ~ (-)r))
The proof relies on the law of total variance and the relationship between Gieniter and GverFrree
First, we show that Gvenrree (x, y x) is the conditional expectation of Gieniteer (x,, y x, y) given y", x. The expectation is taken over y (- x)):
Ey-rs( (-1,x()) Gvettiier (x, y z, v) x v" xx]
E By 1(y 3" [Vo log To (2| + Va moylo =)] x. y),
Σ To(3' xx) = vo
V log (x, x)]
To (2r, 2) Vo lolog (2(2)))) + Va log 1
= Weniiieee (2. 32
15

---

Published as a conference paper at ICLR 2026
We denote By-rr( m.z) Gvennerr y² 2. 2.1))a g,, x] Bylz Veetter (z. y² ,2 2, 8)] for brevity in the following, since æ and y² are already given and fixed. Thus, we have
Ebi= [Geeiieeer (2, y z.y] = Vventreee (x. y ,²,
(8)
The law of total variance states that for random variable W and conditioning variables S, Var(W) Es [as\as[W]]]] Vars|E[W (S]]. Let W Veriter (z, y" 2, y). Given and y", the sources of randomness for Gveriter are and y. Let S be the conditioning variables. The randomness in Verileer given S comes from y ~ To( x. z). Applying the law:
Varz,y [veriiter 2, y)
F= Tex[[arr Varyi= [Gveriter (r, y w)] Var= Fyy [Gveriteer 2, w)]
The expectation E₂ is over m=(-)2)). The conditional variance Varyl= and expectation Eyl are over y ~ Te(-(x, x) for fixed æ, ,².
Substituting the result from Eq. (8) into the law of total variance:
Varz,y [Gveriiie (2r, y y)
Ez Var [Gveriteer (2, y. 2, 2, Varz [Geniireee (1: y", z)]
The second term, Var, VeriFree (x, 8" .>]] is the definition of the variance of the estimator GeriiFeee The first term, Ex y| Veritter x y x) an expectation a variance. Since variance is ways non-negative, Vary [Gveriter (x, y² x)] 2 O. Therefore, its expectation is also non-negative:
Eg
Varyl= (Gveiiiee y², [,, 2 0
Thus, we have:
Varz.y [Gveritier (x. 2, y)] non-negative term) Var, Veniiieee (x, 3² 2)]
This implies:
Varz.1 [eeriirer (x. y 2, y)] 2 Varz [GveniFree (x, ,,,,), =)]
The variance of the policy gradient estimator &venfiiee is less than equal that of O This is an instance of Rao- Blackwellization, where analytically integrating out a source of randomness (the sampling y) by using its conditional expectation reduces variance.
C OFF-POLICY GRADIENT ESTIMATORS
In the main paper we provide an expression for the gradient estimator when the data is fully on- policy. VeriFree is also fully compatible with PPO-style gradient clipping for the case when data is reused to improve sample efficiency. In this case the gradient estimator is:
V VoJventive (0) G
Σ
G
Clip
To (Zi,t x. Zi,ct)
A +
1r'l
Clip
To (y)
D. 2,)
R,
Thas (Z x, 21,(1)
Tea (E
x, zi)
d'11
where Tous is the sampling policy, A Teue zz) Th (y" |æx, 2;), and -. denotes the PPO clipping operation.
T Σ
j
Tow (20 2z), R₁ =
D DATASET DETAILS
The category distribution in WebData (our training data) is visualized in Fig. 7.
16

---

Published as a conference paper at ICLR 2026
Main Categories
Grouped Categories Detail
Other
History
Computer Science
Finance
Grouped
5.4%
9%
6.3%
12.3%
Business
Philosaphy
7%
17.35
Engineering
Physics 20.2%
Law
7.6%
Economics
14.2%
Other STEM
5%3%
9.2%
30.8%
Blology
Chemistry
Mathernatics
Psychology
Figure 7: Category distributions in WebData. A breakdown of the "grouped" category (right) shows detailed distribution of various domains with fewer data samples.
E EXTENDED EMPIRICAL RESULTS
E. MATH-RELATED BENCHMARKS AND
In Tables and 2, provide detailed benchmark results for MMLU-Pro and SuperGPQA using domain name abbreviations. The full nomenclature is as follows:
MMLU-Pro (Table 1):
CS (Computer Science), Math (Mathematics), Chem (Chemistry), Eng (Engineering)) Law (Law), Bio (Biology), Health (Health), Phys (Physics), Bus (Business), Phil (Philosophy), Econ (Economics), Other (Other), Psy (Psychology). Hist (History).
SuperGPQA (Table 2):
Eng. (Engineering), Med. (Medicine), Sci. (Science), Phil. (Philosophy).
M.S. (Military Science), Econ. (Economics). Mgmt. (Management), Socio. (Sociology), L/A (Literature and Arts), Hist. (History), Agron. (Agronomy), Law (Law), Edu (Education).
In Table we present an accuracy comparison across six math evaluation benchmarks and GPQA- Diamond. Models trained with VeriFree demonstrate consistent and significant improvements over the base models, further validating the effectiveness of our approach.
Table 3: Accuracy comparison on math evaluation suite and GPQA-Diamond.
AIME24
AMC
GSMSK
|ATH-500
Minerva
Olympiad
GPQA-D
Method
Acc.
Len.
Acc.
Len.
Acc.
Len.
Acc.
Len.
Acc.
Len.
Acc.
Len.
Acc.
Len.
Qwcn3-1.7B-Bas
1.7 1287
40.0
1001
71.4
58.8
1094
19.5
1071
1737
17.7
Qwen3-1.7B (w/o thinking)
2552
2291
83.3
72.8
1021
27.9
776
1970
19.2
Qwen3-1.7B (w/ thinking)
20.9
7855
5973
89.0
2220
77.4
4525
36.8
5606
6497
17.7
7104
Dwee3-1.7B--BB 1-e---B---BBe------ Base-Verifier
8.4
2317
42.5
1712
81.7
405
66.2
1057
30.5
1357
29.8
1885
36.4
1317
Qwen3-1.77 1-13--..B-BBae--- Beri-reriFree
10.7
1783
37.5
1522
76.2
447
63.8
972
867 21.0
29.8
1484
30.3
1147
Qwen3-1B-BB Base
5.2 1312
50.0
1001
73.1
393
73.4
724
29.4
1027
39.6
1202
24.7
Qwen3-4B (w/o thinking)
20.4 3146
2047
92.1
82.2
1143
41.2
822
49.5
2593
29.8
2127
Qw (w/ thinking)
33.0
7750
6123
2261
84.4
4370
41.5
5352
6432
31.8
6504
Qwen3-4B-BB ---Basa--erriff
15.6
2407
57.5
1566
72.5
377
73.8
1023
24.6
1088
45.9
1576
44.4
1266
Qwen3 4B--BBB--eeee Vrei
16.9
2706
65.0
1904
87.5
682
74.8
1269
25.4
1444
44.9
1899
42.4
1619
Dwen2.5-7B
2.2
869
32.5
922
84.6
260
63.2
582
26.8
819
30.2
963
24.2
595
15.5
1285
$7.5
1097
92.1
331
78.6
690
36.4
795
43.4
1083
23.7
990
Qwen2.5.MM 10022.5MMtt-B-OOA
28.3
1115
846
90.8
79.0
652
33.1
24.7
Qwen2.5-7B. 7-B--s-uttt
993
$2.5
986
78.2
649
690
1123
31.8
General-Reaseee 7B
13.1
1363
52.5
1083
81.7
408
74.6
858
23.5
968
39.3
1303
34,8
1199
Qwen3-8B. 8B-Base
6.5
1213
65.0
917
91.7
304
77.0
671
38.2
884
41.3
1189
38.9
887
Qwen3-8B 8B (w/o thinking)
24.6
2897
62.5
1919
93.3
293
82.2
1087
36.4
751
49.5
2422
27.8
2204
Qwen3-88 8B (w/ thinking)
31.0
7789
62.5
6201
95.3
2203
83.0
4414
43.8
5410
45.2
6554
37.4
6681
Qwen3-8F 8B- Baa--Base-aee Verif-errfier
17.8
1954
57.5
1265
93.4
307
76.2
784
39.0
698
41.8
1201
43.9
733
Owen3-88 --e--------------- VeriFre
25.2
2461
67.5
1542
91.6
375
80.8
1024
30.5
902
50.2
1778
44.4
1080
17

---

Published as a conference paper at ICLR 2026
E.2 EMPIRICAL COMPARISON WITH RELATED APPROACHES
In Sec. 2.3, we have discussed the methodological differences between our work and related ap- proaches, LaTRO and JEPO. In this section, we present an experimental comparison with these methods.
We first evaluate JEPO as a new baseline by fine-tuning the Qwen3 base model on our curated dataset (WebData) as introduced in Sec. 3.1. During fine-tuning, we observe that the number of reasoning tokens in the model's responses progressively decreases, eventually leaving almost no valid reasoning tokens (only answer tokens). The resulting performance is worse than that of the base model, so we do not report these results here. similar trend is observed for LaTRO. hypothesize that this degradation stems from two factors: (1) potential noise in the training data (which preprocessed from Web Instruct and filtered by an LLM), and (2) the lack robustness in LaTRO/JEPO, as these methods apply uniform weighting answer token gradients. This may force the model to prioritize reference answers even when the reasoning quality is poor (see Sec. 2.3 for further analysis).
To further investigate, we evaluate JEPO and our method on the cleaner math 1 12k dataset (Hendrycks al., 2021) for mathematical reasoning tasks. We fine-tune the Qwen3-4B-Ban model using four approaches: (1) RL with rule-based verification (denoted as RLVR), (2) LaTRO. (3) JEPO, and (4) VeriFree. Evaluation across multiple mathematical reasoning benchmarks (see Table shows that our method consistently outperforms JEPO and matches the performance of RL with rule-based verification.
Table 4: Accuracy comparison math tasks. All methods trained math12k (Hendrycks et al., 2021) based on Qwen3-4B-Base. The best results are highlighted using bold text.
Method
MATH500
AIME24
AMC
Minerva
Olympiad
GSM8K
Qwen3-4B-Ban
73.4
5.2
50
29.4
38.6
73.1
RIVR
83.8
18.54
62.5
42.65
50.07
92.7
LaTRO
82.6
16.67
60.0
37.9
45.9
90.8
JLB
83
16.67
60.0
39.0
47.7
92.0
VeriFree
84.6
21.46
62.5
42.65
50.22
92.57
For direct comparison with General Reasoner (Ma al., 2025) which also focuses general domain reasoning, we evaluate their newly released Qwen3-4B-BB (BBasse) MMLU-Pro, SuperGPQA, and GPQA-Diamont These benchmarks de- signed to assess reasoning capabilities across diverse general domains. The results, presented in Table demonstrate that our method matches surpasses other approaches of a comparable model scale.
(also fine-tuned from
Table 5: Accuracy comparison on general domain reasoning tasks. All methods are based on Qwen3-4B-Base The best results are highlighted using bold text. Note that indicates results reported by Ma al. (2025).
Model
MMLU-Pro
SuperGPQA
GPQA-D
Qwen3-4B-Base
51.6
25.4
26.3
Qwen3-4B (non-think)
61.8
32.1
41.7
62.8
32.5
42.9
Qwen3-4H Beon3-4BBBase A---Bsse-Ver Vere-ieri
63.0
34.3
44.4
Qwen3 44---44BBase Bass-VeriFrres
63.5
35.1
42.4
E.3 EVALUATION RESULTS WITH DIFFERENT SAMPLING PARAMETERS
The sampling parameters used in main experiment (detailed in Sec. 3.1) follow the configuration from Ma et al. (2025). Recent works (Hochlehnert et al., 2025; Team, 2025) suggest that greedy decoding may be suboptimal for reasoning models that generate extended thinking traces. To address
18

---

Published as a conference paper at ICLR 2026
this concern, we conduct comprehensive evaluation GPQA-Diamond using various sampling configurations.
We provide additional evaluation results GPQA-Diamond using six distinct sampling configura- tions, including those recommended in the Qwen3 official report (Team, 2025):
S1: temperature=0) (i.e., greedy decoding) and 8k token budget (same Ma et al. (2025))
S2: temperature=0 (i.e., greedy decoding) and 32k token budget
S3: temperature=O.7,tog top- top_k=20, min.p=0, n_repeats=10, and 8k token budget
S4: temperature=0.7, to--p=0.8, 1op_k=20,0 min_p=0, n.repeats=10, and 32k token budget
S5: temperature=066, top 00ppp-0.9,, top k=20, min -0in rerepetts=10, and 8k token budget S6: temperature=0.6, top p=0.95, top k=20, min p=0, n repeats=10, and 32k token budget
We report the results in Table 6. As the results show, our method consistently matches or surpasses both the counterpart with the model-based verifier and also Qwen (w/o thinking) across most settings. Qwen3 (w/ thinking) shows strong results when given extremely large token budgets (32k). Note that Qwen3 (w/ thinking) is trained with multiple stages, including cold start with distillation data, while method and the counterparts with verifiers directly trained from the base model ("zero" setting).
Table 6: Performance comparison with different sampling configurations on GPQA-Diamont The best results are highlighted using bold text.
Model
S1
S3
S4
S5
S6
Qwen3-4B Base
28.79
28.28
29.85
29.39
29.49
28.99
Qwen3-4B (w/o thinking)
36.87
31.31
38.99
40.05
37.88
38.43
Qwen3-4B (w/ thinking)
29.29
47.47
36.26
55.15
35.66
54.19
Been3-4B-BBaee-Veiiff
42.42
43.43
41.57
42.37
42.07
41.41
Qwen33-4B-Base-Werie VeriFree
40.91
45.45
43.03
42.88
43.99
42.88
Qwen3-8B-Banl
37.37
37.88
34.75
36.87
36.62
34.85
Qwen3-8B (w/o thinking)
35.86
32.32
37.02
36.46
36.41
37.83
Qwen3-8B (w/ thinking)
36.36
58.59
37.37
61.26
37.83
62.27
Qwen3-8B- Bwen3-8B-Base-Vere Base-Verific
40.91
44.95
46.52
47.27
45.76
47.68
Qwen3-8B- Veri-rer
48.48
43.94
47.22
46.46
46.01
47.37
E.4 AvG K PERFORMANCE ON SMALL BENCHMARKS
We note that the primary benchmarks in our evaluation are large in scale. For instance, MMLU- Pro includes 12k test samples and SuperGPQA contains 26k. On these large benchmarks, VeriFree generally achieves stronger performance.
We have also added evaluations smaller benchmarks, as shown in Table 7. We report Avg@K, where is chosen such that the total number of samples after repetition is approximately 1k.
Table 7: Avg mrg@k K small benchmarks. The best results highlighted using bold text.
GPQA-D
AIME24
MATH500
Minerva Avg@4
Model
Avg@4
Avg@ 32
Avg@2
42.80
16.04
73.70
24.63
Bwen3-4B-Bass VeriFree
42.29
17.50
75.10
25.54
Qwen3-8B -B---BB--BB------ -B-BBae--errffer
44.07
18.96
77.20
36.50
Qwen3-8BB 3--n3-BBBJue --------BBs---- B-se-aaseeete VeriFFee
45.45
25.52
81.10
34.67
19

---

Published as a conference paper at ICLR 2026
E.5 EXPERIMENTS ON OTHER MODEL FAMILIES
We have extended our evaluation to an additional model family, applying both our method and the verifier-based baseline SetoThinker-BB-Lonn--- (Wang al., 2025), model mid-trained from Llama-3, The performance is compared against the official RLVR-trained reasoning model -oooohe-------o-------
and the corresponding verifier-based baseline. All evaluations are conducted under zero-shot setting. Results are presented in Table 8. Our model can outperform both the official RLVR model and our implemented counterpart with model-based verifier. The base model does not perform well due to poor instructtionfoolloiig abilities.
Table 8: Accuracy comparison on general domain reasoning tasks. All methods are based on D The best results are highlighted using bold text.
Model
MMLU-Pro SuperGPQA
GPQA-D
SetoThinnerrBBB-Loggg
5.2
2.1
0.0
SctoThinkerr33B-Long--2
35.8
18.6
25.3
OctoTThinkrr33B -on-g 3BB-Long-Verifies Verifier
35.5
19.8
28.3
OctoThinker--3 3B 3B-Long- VeriFree
36.2
20.5
30.3
E.6 EXPERIMENTS WITH LONGER ANSWERS
In our main experiments, the dataset preprocessing retains only short-phrase reference answers with token lengths under 7. To investigate whether VeriFree can effectively handle longer answers, we conduct an additional experiment using the training data from General Reasoner (Ma al., 2025), which includes examples with longer reference answers. We align the training and evaluation pro- tocols with General Reasoner, and denote this variant "Qwen3-4B-BBl VeriFre w/ GR data". The results are presented in Table 9.
Table 9: Experiments with longer reference answers. All methods are based on Qwen3-4B-Base. The best results are highlighted using bold text.
Model
GPQA-D
MMLU-Pro
SuperGPQA
Qwen3-4B-Bane 1e1-4---BBee--e Base--eriieee
44.4
63.0
34.3
Bwen3-4B-Basee Base--erreree w/ our data
42.4
63.5
35.1
42.9
62.8
32.5
Qwen3-4B-Band Be1---4--BB----B VeriFree w/ GR data
45.5
63.2
34.5
Even when trained data containing longer reference answers, VeriFrree continues outperform the baselines. It is also noteworthy that our method remains more computationally efficient.
F DISCUSSION
This section provides detailed discussion of the advantages and limitations of our proposed method, VeriFree.
F.1 ADVANTAGES
VeriFree offers distinct advantages over other verification paradigms for general L.LM reasoning tasks. While rule-based verification is effective, its applicability is restricted to specialized domains like mathematics and coding. In contrast, model-based verifiers (e.g.. LLM-as-judge methods) of- fer broader applicability but introduce practical challenges, including substantial resource overhead (e.g.. additional training data, increased computation, and memory usage) and inherent risks such reward hacking and various biases (Ye et al., 2025; Chen al., 2024a). A prominent example is length bias, where LLMs systematically favor longer responses irrespective quality. The fact that methods like General Reasoner (Ma et al., 2025) must explicitly incorporate length-based penalties
20

---

Published as a conference paper at ICLR 2026
(Sec. 3.1) underscores the pervasiveness of this issue. VeriFree cireumvents these issues by directly optimizing the conditional likelihood of the reference answer, eliminating the need for an auxiliary verifier model. So it avoids these pitfalls entirely, providing
robust and efficient alternative.
Although VeriFree trained using only a single reference answer, it remains highly effective even when multiple correct solutions exist. This is because the pretrained base model learns generalized understanding of correctness and reasoning logic, rather than merely memorizing a specific answer format. Much like in supervised fine-tuning, where models develop broad enstruction-followiii ability despite being trained on fixed-style examples, VeriFree captures the underlying principles of valid reasoning, enabling it to recognize correctness across varied expressions. This demonstrates VeriFree's s superiority in generalization without relying on multiple references during training.
F.2 LIMITATIONS
The primary limitation of VeriFree its dependency on direct question-answer pairs for supervision (note that VeriFree does not require any reference reasoning traces). This requirement constrains its applicability in scenarios where such supervised pairs are unavailable. For instance, certain coding tasks, only test cases may be provided without corresponding reference implementations. In such settings, rule-based verification or other methods that do not rely on reference answers may represent the only viable alternatives.
G CASE STUDY
In this section, provide some examples of reasoning traces produced by the model trained by our method on training questions. And we also provide an example with both our model's and base model's responses the test dataset.
Example question in training dataset (category: Chemistry)
Prompt:
<lim\_start) >user
Label or a placard colored black and white with skull and crossbones falls under which class
Please reason step by step, put your final within <answer> \boxed </answer>, < end/>
<lim\_start >assistant
Our Model's Response:
label or placard colored black and white with skull and crossbones indicates the class of hazardous materials known as **Class 6*+, Class is specifically divided into two divisions:
**Division 6.1++: Toxis substances. These materials that cause injury or death ingested or inhaled.
Diiiiision 6.2+*: Infectious substances. These are materials that contain pathogens and can cause disease in humans or animals.
The skull and crossbones symbol is universally recognized as representing toxicity, which correspends **Division 6.1** of Class 6 hazardous materials, <answer> \boxed/Class 6) </answer>
Reference Answer:
Class 6
Example question in training dataset (category: Physics)
Prompt:
i1m_sttart Puser
21

---

Published as a conference paper at ICLR 2026
For what type of object is the pericd-luminosity relation used for determining distances?
Please reason step by step, and put your final within <answer> \boxed|| </answer>, i
Our Model's Response:
The perioč-luminositg relation is fundamental tool astronomy used to determine distances, particularly for a specific type of star known as Cepheid variables.
pulsating stars that exhimit reguiar pattern of expansion and contractien. This pulsation causes them to vary in Luminosity, and the key insight about these stars is that the period of
Cepheid variables
their luminosity variation is directly related to their intrinsic luminosity (absolute magnitude). period of pulsation, the more intrinsically luminous the star is.
other words, the longer the
This period-luminosity relation Leavitt in 1908 by studying Cepheid variables in the Large Magellanic Cloud. Because Cepheids are so bright intrinsically, they can be used as "standard candles" to measure distances within our galaxy and to nearby galaxies.
first discovered by Henrietta
Here's s how the period-luminose relation used to determine distances:
1. +*Observe the Period**: First, astrenemers measure the period of the star's brightness variation, which is the time it takes for the star to go from brightest dimmest back to brightest.
2. DeDeerrinne Intrinsic Luminosity+:: Using the period-luminosig relation, they can then determine the star's s intrinsic luminosity. This relation is usually given in form where graph formula relates period to absolute magnitude (a measure intrinsic brightness).
3. Compare to Apparent luminosity++: The apparent magnitude (how bright the star appears from Earth) be measured di rectly...
4. #Callcllate Distancews: Using the inverse square law for light, which relates the intrinsic luminosity, apparent brightness, and distance, astronomers then calculate the distance to the Cepheid variable.
This method is crucial for measuring distances in astroncmy because it provides a way to gauge hew far away these stars are with a high degree of accuracy. The distances determined using Cepheid variables are themselves used to calibrate other distance measurement techniques, such as those involving supernovae.
Given these details, the type object which the period-luminosity relation is used for determining distances Capheid variables. <answer> (Cephedd variables) </answer>
Reference Answer:
cepheid variable stars
22

---

Published as a conference paper at ICLR 2026
Example question in MMLU-Pro benchmark (category: Psychology)
Prompt (Here we ignore the prompt template and only show the question for brevity):
During the second stage of Kohlberg's s preconventional level
moral development, children obey rules because: Options are:
A. they are taught that rules are unbreakable.
B. they enjoy following the rules.
C. they are rewarded
following rules.
D. they feel they have a personal duty to uphold rules and laws.
E. doing so helps them satisfy their personal needs.
F. they understand the concept of fairness and justice.
G. doing so helps them avoid punishment.
a. everyone else is doing it.
I. they believe in the inherent goodness of rules.
J. they fear societal disapproval.
Our Model's Response:
In Kohlberg's theory of moral development, the preconventicnal level is the first stage of moral reasening, and it is primarily focused on sclf-interest. This level is divided into two stages:
1. Stagae 1: Obedience and Punishment Orientation
In this stage, children obey rules to avoid punis shment. Their reasoning is Fear-based, and they view rules set by authority figures. The focus is the consequences of following or breaking rules, specifically the avoidance of punishment.
2. **aage 2: Individualism and Exchange44
In this stage, which is the second stage preconventional level, children recognize that different people have different needs and interests. They obey rules only when it is in their own interest when they can benefit from doing so. This stage is often described it for me?" stage. Children at this level understand the concept of exchange and may follow rules if it leads to a reciprocal benefit.
the "what's s in
Now, let's s evaluate the options:
(A) they are taught that rules are unbreakable. This does not align with either stage of the precenventionsi level.
(B) thoy enjoy following the rules. Enjoyment of rules is more characteristic of conventicnal levels of moral development.
(C) they are rewarded for following rules, While this could be somewhat related to Stage 2, the primary focus in Stage 2 is on persenal benefit rather than external rewards.
(D) they feel they have a personal duty to uphold rules and laws. This is more aligned with the conventional level, specifically the intermediate stage.
(E) doing so helps them satisfy their personal needs. This could be seen as rolated to Stage 2, but it is not the most precise description.
(F) they understand the concept of fairness and justice.
23

---

Published as a conference paper at ICLR 2026
Understanding fairness and justice is more associated with the conventional and Iost-conventional levels.
(G) they avoid punishment.
This describes Stage 1 of the preconventional Level, not Stage 2.
(H) everyone else doing it. This is more related to conformity and is not a key characteristic of Kohlberg's stages.
(I) they believe in the inherent geedness of rules. This more characteristic of the post-conventional level.
(5) they fear societal disapproval. Fear of societal disapproval is related to the conventional level.
Thee best fit for Stage 2 of the preconventional level is (R) doing helps them satisfy their personal needs, as it reflects the idea of mutual benefit and personal interest. However, (G) they avoid punishment describes Stage more accurately. Given the typical interpretations of Kohlberg's s stages, (E) is the more appropriate choice for Stage 2.
But let's double-check: According Kchlberg, in Stage 2, children's s moral reasoning is based en the notion of exchange and the satisfaction of self-interest. Therefore, (E) is the mest accurate option.
<answer> \boxed(E) </answer>
Base Model's Response:
During the second stage of Kohlberg's preconventional level of moral development, children obey rules because they understand the concept of fairness and justice. This stage characterized by focus on the consequences of actions and the desire to aveid punishment or gain rewards. Chilaren at this stage begin to recognize ze that rules are not absolute and can be negotiated based the situation and the needs of the individual. They start to consider the perspectives of others and the fairness of the rules, which is a significant step towards more advanced moral reasoning.
Therefore, the correct
is:
\boxed(F)
Reference Answer:
H LLM USAGE
We used OpenAl LLM (GPT-5) writing and formatting assistant. In particular, it helped refine grammar and phrasing, improve clarity, and suggest edits to figure/table captions and layout (e.g., column alignment, caption length, placement). The LLM did not contribute to research ideation, experimental design, implementation, data analysis, or technical content beyond surface- level edits. All outputs were reviewed and edited by the authors, who take full responsibility for the final text and visuals.
24


Reviews:
[{"review_id": "16tSb9bfdH", "reviewer": "Reviewer_9QRi", "summary": "The paper proposes an RL framework called VeriFree, which extends the R1-Zero paradigm (DeepSeek-R1-Zero) to general reasoning tasks without requiring explicit verifiers. Existing methods like RL with verifiable rewards (RLVR) rely on rule-based or model-based verifiers (often another LLM), which are infeasible for open-ended domains such as law, medicine, or economics. VeriFree addresses this by eliminating the verifier entirely while preserving the core benefits of RL-based fine-tuning. In R1-Zero-style training, a model generates a reasoning trace (z) and a final answer (y). A verifier gives a reward = 1 if y matches the reference answer y*, else 0. VeriFree reformulates this process: instead of relying on a verifier, it directly maximizes the model’s probability of generating the reference answer. The resulting gradient estimator combines: (a) a reasoning term weighted by the model’s likelihood of producing the correct answer, and (b) a supervised term weighted by the same likelihood.\n\nThis makes it equivalent in expectation to RLVR under single-correct-answer cases but with lower gradient variance (proved via Rao-Blackwellization). The approach integrates variance reduction (RLOO) and tokenization-aware patching to ensure stable optimization.\n\nFor experimental evaluation, Qwen3 (1.7B, 4B, 8B) base models are fine-tuned with VeriFree using the Oat framework. Benchmarks used are MMLU-Pro, GPQA, SuperGPQA (for general reasoning) and MATH-500, OlympiadBench, Minerva Math, GSM8K, AMC, AIME24 (for mathematical reasoning). On MMLU-Pro and SuperGPQA, VeriFree matches or surpasses verifier-based RL baselines while being simpler and more compute-efficient. The accuracy gains are +12-40% over base models, and it achieves faster convergence and higher final accuracy than verifier-based RL.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written and structured well. \n\n- The concept of verifier-free RL eliminates reliance on rule-based or LLM verifiers, making RL training scalable to open-ended reasoning domains. Further, it is derived directly from the RL objective; proven equivalence to RLVR under certain assumptions with formal variance reduction.\n\n- Strong empirical results: Matches or surpasses verifier-based methods across multiple benchmarks while being simpler, faster, and less memory-intensive. The proposed method demonstrates reasoning transfer to unseen domains and improved convergence behavior.", "weaknesses": "- In Tables 1 and 2, it is unclear why the accuracy of Qwen3-4B-Base-Verifier is lower than that of Qwen3-4B-Base-VeriFree. In VeriFree, the LLM output is first parsed into reasoning tokens and a generated final answer (y). Then, the generated answer is replaced with the gold-standard final answer (y⁎). If only one answer (y⁎) is correct and receives a reward of 1 (while all others receive 0), then the expected reward for a reasoning trace z can be computed directly as the probability assigned to y⁎, effectively marginalizing out y. In contrast, when using a verifier, y⁎ will receive a reward of 1 from the verifier, but all other correct answers (for questions with multiple valid answers) will also receive a reward of 1. Conceptually, this should make the verifier-based approach perform better than VeriFree, as it trains the model to obtain a reward of 1 for all correct answers rather than a single reference answer.\n\n- The paper mentions that \"when multiple valid answers exist, we show empirically that using just one as a reference provides a sufficient learning signal to elicit strong reasoning behavior.\"\" However, it is not clear why this should be the case. In the VeriFree framework, the model is trained to maximize the reward only for a single correct answer.\n\n- The theoretical equivalence holds only when there is a unique correct answer, limiting its applicability to open-ended or multi-valid-response tasks. In such cases, the model may overfit to the reference answers instead of exploring diverse reasoning paths, thereby reducing interpretability.\n\n- For the baseline, it is unclear why a model-based (i.e., LLM-judge) reward model was used. For math and code tasks, rule-based or test-case-based evaluation is feasible and far more reliable than an LLM judge. The paper claims that VeriFree demonstrates strong transfer to math benchmarks; therefore, for math- or code-specific benchmarks, a stronger rule-based verifier should have been used as a baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dvKfHKEzxY", "reviewer": "Reviewer_r1nM", "summary": "This paper propose a novel VeriFree framework that bypass answer verification and assign rewards according to the probability of generating reference answer.\n\nUnique answer assumption might not hold generally.", "questions": "1. Does VeriFree performs better than existing methods that rely on verifiers such as DAPO and ORZ. As RLVR exhibits good generalization capability, could it be possible that training on mathematical problems can yields better results?\n2. How does VeriFree works on other model families, considering the data contamination risk of Qwen3 model family.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed VeriFree framework is simple to implement and bypass the need of verifiers.\n2. Experimental results demonstrate that VeriFree achieves comparable results to the verifier-based baseline and shows good transferable reasoning skill gains.", "weaknesses": "1. Evaluation reliability: Single run pass@1 results are too noisy for small benchmarks like MATH-500, Minverva etc. The author might have to report Avg@k and other statistics for robust demonstration.\n2. Missing baselines: many existing probability- or frequency-based baselines are missing, such as TTRL.\n3. The assumption that only single accurate answer exists might be problematic due to the flexibility of language (e.g., 'A is larger than B' and 'B is the smaller one'). The author might need to conduct more analysis on this assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mB0LXBPxc", "reviewer": "Reviewer_2Bnh", "summary": "This paper proposes VeriFree, a verifier-free methodology for general reasoning tasks beyond rule-based verification. The authors explicitly test VeriFree on a variety of domains such as economics, chemistry, and health (among others) to examine performance on multi-task understanding, and `graduate student level reasoning'. In addition, for the case when there is exactly one correct answer to questions, the authors provide theoretical analysis demonstrating that the approach of VeriFree has lower variance than methods which use a verifier, including reinforcement learning with verified rewards (RLVR).", "questions": "The authors' comments on the points raised in the Weaknesses section, above, will be helpful.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(+) The paper presents an interesting approach to bypass the need for verification for more general reasoning tasks that might not be rule-based. \n\n(+) The claims are supported by analysis (for one specific setting- also see Weaknesses, below) and experimental evaluations. \n\n(+) The performance of VeriFree is comparable to verifier-based methods on a large variety of tasks, providing a promise of improved computational efficiency resulting from eliminating reliance on a strong verifier. \n\n(+) Sec. 2.3 comparing the gradient estimator in VeriFree to estimators from alternate approaches offers particularly helpful and revealing insight into the working of VeriFree (for the single correct answer setting).", "weaknesses": "(-) The authors claim that VeriFree is the first verifier-free methodology for this class of problems. It is not clear if benchmarking against verifier-based methods only adequate. \n\n(-) What does `rule-based answer verification’ exactly mean? It is reasonable to conjecture that some of the domains mentioned might support rule-based answer verification (e.g., based on statutes of laws). This question becomes even more relevant since the authors state in the Limitations (in the Appendix) that VeriFree is constrained by its dependency on question-answer pairs for supervision. \n\n(-) It is not clear if the single answer assumption is a reasonable one to make for domains beyond rule-based verification. Specifically, one would assume the potential ambiguities in answers to questions in these domains is a characteristic that has necessitated use of a verifier. \n\n(-) From Fig. 1, the claim of VeriFree surpassing instruct models and models tuned with a specialized LLM verifier are somewhat strong, given that the accuracy numbers in the bar graphs for VeriFree are only very marginally higher in 5 out of 6 graphs and comes in second best in one graph. At best, one can conclude that VeriFree matches the accuracy performance of these models. \n\n(-) In Line 107, the paper claims that VeriFree is simpler, faster, less memory-intensive, and more robust than verifier-based alternatives. It is not clear which experiments in the paper correspond to evaluations of memory intensiveness and robustness. \n\n(-) In Tables 1 and 2, it is not clear to me how the quantity in the Avg column is computed. Checking the numbers, it does not appear to be equal to the mean (average) of the numbers in the same row in the columns on the right. For example, in Table 1, for Qwen3-8B Base-VeriFree, the Avg reported is **67.2**, while **mean(71.5,85.3,73.5,…,45.6) = 66.3**. If it is indeed the case that the Avg should correspond to the mean of the entries in the same row, then this discrepancy extends to many of the other numbers in both tables. \n\n(-) While VeriFree performs better when looking at the average accuracy, for the 4B experiments in Table 1, VeriFree obtains a higher accuracy than Verifier in only 8 out of 14 domains. Similarly, for the 8B experiments in Table 2, VeriFree obtains higher accuracy that Verifier in only 7 out of 13 domains. \n\n(-) Minor typo: Line 183 - marginalizes —> marginalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1kN4unMxxQ", "reviewer": "Reviewer_tixt", "summary": "This paper introduces VeriFree, a method that extends R1-Zero-style reinforcement learning to general-domain, non-mathematical reasoning by using the model's own probability of the ground-truth answer as the reward signal, thereby eliminating the need for explicit verifiers. The authors evaluate VeriFree on broad reasoning benchmarks akin to GeneralReasoner and show that it outperforms a strong LM-based verifier baseline.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is simple yet effective, offering a clear mathematical derivation and concise comparisons with existing algorithms.\n- By tackling general-domain reasoning rather than confining itself to mathematics, this work addresses a pressing gap and distinguishes itself from the current wave of math-only RL studies.", "weaknesses": "- The VeriFree method uses implicit reward induced by model confidence, which is unavoidably an imperfect measure. Where budget is unconstrained, practitioners will still prefer a heavyweight but higher-fidelity verifier; the method therefore does not raise the performance ceiling of RL reasoning, it merely lowers the cost floor to some extent.\n- All reported experiments rely on single-word or short-phrase ground-truth answers; leaving open the critical question of whether the method can be applied to problems with long answers and coding tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)