General-Reasoner: Advancing LLM Reasoning Across All Domains
Authors: Xueguang Ma
Qian Liu
Dongfu Jiang
Ge Zhang
Zejun MA
Wenhu Chen
Venue: NeurIPS 2025 poster ()
Keywords: large language model
reasoning
reinforcement learning
Primary area: applications

Abstract: Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs).
Particularly, the "Zero" reinforcement learning introduced by Deepseek-R1-Zero, enables direct RL training of base LLMs without relying on an intermediate supervised fine-tuning stage.
Despite these advancements, current works for LLM reasoning mainly focus on mathematical and coding domains, largely due to data abundance and the ease of answer verification.
This limits the applicability and generalization of such models to broader domains, where questions often have diverse answer representations, and data is more scarce.
In this paper, we propose General-Reasoner, a novel training framework designed to enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing a generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of-thought and context-awareness. We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 benchmarks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that General-Reasoner outperforms existing baseline methods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.

General-Reasoner: Advancing LLM Reasoning Across All Domains
Xueguang Ma, Qiian Liu,
Dongfu Jiang, Ge Zhang, ZZeuun Ma, Wenhu Chen Vector Institute, TikToo, Singapore, MM-APP
University of Waterloo,
x93ma@uus
https://tiger- ai ht
Abstract
Reinforcement learning (RL) has recently demonstrated strong potential in enhane- ing the reasoning capabilities of large language models (LLMs). Particularly, the "Zero" reinforcement learning introduced by Deepseck- -aasseckk----e Ri-Zero, enables direct RL training of base LLMs without relying stage. Despite these advancements, current works for LLM reasoning mainly focus mathematical and coding domains, largely due to data abundance and the ease of answer verification. This limits the applicability and generalization of such models broader domains, where questions often have diverse answer representations, and data is more scarce. In this paper, we propose GENERAL-REASONEE training framework designed enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of of-thought and them on wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 bench- marks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that GENERAL-REASDINE REASDNER outperforms existing baseline meth- ods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.
intermediate supervised fine-tuning
a novel
We train series of models and evaluate
1E.00.3
70
00.
64.2
Score (%)
50
40
43/5
36.333
24.2
30
MMLU--Pr
GPOA
SuperGPOA
TheoremQA
Math Related
SRIpECHL LOMO
Ganno 148 Base
Owen3 140 Instruct (hink)
General Resso OwS-188 168
Figure Effectiveness of our GENERAL REASOOEE trained with diverse verifiable reasoning questions using model-based verifier compared to bascline methods on various reasoning tasks.
'The first two authors have equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent advancements in large language models (LLMs) have demonstrated strong potential in many expert-level tasks by following user instructions. One of the most significant breakthroughs in recent work is the use of reinforcement learning (RL), such as PPO and GRPO, to improve the reasoning capabilities of LLMs.. A particularly noteworthy development is the "Zero" reinforcement learning setting proposed by DeepSeek-R1 R1-Zero [1. which shows that training a base LLM directly via RL can unlock powerful reasoning capabilities without relying
supervised fine-tuning step. RL for LLM reasoning has inspired multiple open-source efforts, such as SimpleRL [2], DAPO [3) and DeepScaleR [ [] which leverage variants of GRPO 66] to further enhance reasoning performance, either in the zero RL setting or via continuous reinforcement learning on models with supervised
fine-tuning or distillation.
However, existing methods primarily focus on training and evaluation in mathematical reasoning or coding tasks. This narrow focus can be attributed to two main reasons: (1) DATA ABUNDANCE: it is easier to harvest large-scale mathematical data from the Internet due to the abundance of international math and coding competitions and exams; (2) ANSWER VERIFICATION mathematical domains allow for easy and reliable answer verification using straightforward rule-based methods (e.g.. exact numeric matching, symbolic equation comparison), which provide accurate reward signals. However, this reliance on rule-based verifiers restricts the generalizability of the resulting models. Real-world reasoning often spans multiple disciplines, such as science, finance, and the humanities, and involves complex, long-tailed answer representations that cannot be reliably verified through heuristic rules. In fact, training solely on mathematical or coding data does not guarantee generalization to other reasoning domains. For example, although S1/S1.1 1 (7) significantly improves mathematical scores, it degrades performance MMLU-Pro [ by 4-6%.
To address these challenges and broaden the applicability and robustness of RL-based reasoning models, we propose training framework designed to enhance the reasoning capabilities of LLMs across diverse, non-mathematical domains, while preserving their strengths in mathematics. Our approach tackles the two aforementioned problems:
All-Domain Reasoning Dataset (WebInstruet-veriffee) A major bottleneck for scaling rea- soning beyond mathematics is the lack of publicly available, high-quality reasoning datasets that span multiple domains and offer reliably verifiable answers. To address this, construct a large-scale, diverse-domale dataset by carefully crawling and filtering high-quality reasoning questions from web resources, based on WebInstruct [9]. We then employ state-of-the-art LLMs to automatically select questions whose answers can be reliably verified, significantly expanding the training scope disciplines such as physics, chemistry, social sciences, and finance. Our carefully curated dataset contains approximately 230K high-quality reasoning questions, providing a robust foundation for general reasoning training across multiple complex domains.
All-Domain Answer Verifier (General-Veriffer): In addition to scaling data, existing rule-based verifiers struggle with the diverse answer types commonly found in broader domains, including short string answers, LaTeX expressions, and other structured forms, limiting the effectiveness of reasoning training beyond mathematics. We introduce compact generative verifier model (1.5B parameters) explicitly trained to verify short answers in a context-award thereby providing robust and reliable reward signals for RI. training. This approach improves the flexibility and scalability of RI. training across diverse reasoning tasks.
manner,
Using this diverse, verifiable reasoning data along with the generative model-based verifier, we are able to train a series of GENERAL-REASONE models in the Zero-RL setting from various base models. To validate the effectiveness of our approach, we conduct comprehensive evaluations across 12 challenging reasoning benchmarks beyond mathematics, including MMLU-P Pro [8) GPQA [O), SuperGPQA M. TheoremQA [22, and BBEH [13], as well as standard mathematical reasoning benchmarks such as MATH-500 14] GSM8K [15), and Olympiad [66.. GENERAL-REASONEN typi- cally boosts performance general benchmarks like MMLU-Pro and SuperGPQA by approximately 10%. On math benchmarks,
can even slightly outperform math-focused RL frameworks such as SimpleRL 2 benefiting from cross-domain generalization. Our best model SEIERALL-READNEER-OOE
is able to match or beat GPT-4o in various benchmarks.
In summary, work makes the following key contributions:

---

1. We construct and release a large-scale, high-quality dataset (WebInstruct- verified) of verifiable reasoning questions with short-form answers spanning diverse domains, significantly broadening the training resources available for general reasoning.
2. We introduce a compact, generative model-based verifier Gieneral-Verifier specifically trained for Shain-of-thoughe answer verification, effectively replacing tule-based methods and enabling robust and scalable RL training across multiple complex reasoning domains.
3. Through extensive evaluations,
empirically demonstrate the superior performance and general- ization capabilities of GeNERAL-REAsONE in the Zero-RL setting, providing a strong baseline and valuable insights for future advancements in RL-driven general reasoning.
2 Related Works
2.1 Large Language Models for Reasoning
Advancements in large language models (LLMs) have demonstrated substantial potential in effectively performing diverse tasks by following user instructions. However, expert-level tasks, such as solving complex mathematical problems [I2] or addressing practical STEM challenges (8) require robust reasoning capabilities, Recent work on chain-of-though. reasoning has shown that LLMs can significantly enhance their performance by explicitly decoding their thought processes during inference [17]. Subsequent studies further investigate the scaling of reasoning process, termed "test-time scaling," " better enhance the reasoning capability of LLMs [0).
Following this direction, commercial models such as OpenAT's 01 series have exhibited impressive performance reasoning-intensive evaluations 21,, including mathematical Olympiad problems. Meanwhile, recent open-source models, such as Qwen (22), QWQ [23). and Deepseek R1 []. have also achieved competitive results, narrowing the performance gap with state-of-the-ar commercial counterparts. This progress opens new avenues for academia to explore advanced optimization
strategies aimed at further improving the reasoning capabilities of LLMs 3 24
2.2 Zero Reinforcement Learning for LLMs
Typical methods for enhancing the reasoning capabilities of LLMs usually involve first performing supervised fine-tuning with chain-of oh-thht (CoT) data on a base model [25]. followed by reinforce- ment learning (RL) further improve reasoning and generalization performance. However, recent studies exemplified by Deepseek-R1-Zen learning to strong base model can effectively uncover significant reasoning capabilities without ini- tial supervised fine-tuning. This "Zero Reinforcement Learning" approach has attracted considerable attention within the research community, inspiring efforts to replicate and enhance the method on other powerful base models such as Qwen (26).
[4] have demonstrated that directly applying reinforcement
key advantage of zero RL its efficiency in collecting verifiable question-answer pairs, which eliminates the need for complete reasoning chains as training targets. This efficiency provides greater flexibility in enhancing reasoning performance without complex data collection efforts, enabling models to self self-improve more effectively. However, existing works on zero RL have mostly focused on mathematical reasoning (2). While there are concurrent works 22 28 aim to expand reasoning capabilities to broader domains, our work seeks to do so by leveraging a model-based verifier to scale diverse reasoning data across domains and provide comprehensive evaluations demonstrate effectiveness.
General Reasoner
3.1 Diverse Verifiable Reasoning Tasks
To facilitate robust reasoning capabilities across wide range of domains beyond mathematical problems, we construct a large-scale, diverse, and high-quality dataset composed verifiable reasoning tasks. Our dataset-buidding pipeline illustrated in Figure
The initial data source is from WebInstruct dataset (), which consists of around million naturally occurring, web-crawled instructions from high-quality resource websites, including platforms like
3

---

Subject Physics
Q
Extract
Tag LLM
Dificuitt University
Solve
Remove
230k diverse, verifiable pairs
A
LLM
All v Too easy
Ans. Type
Expression
All X: Risky
Webinstruct Answer
QA Pairs
8 x CoT Solutions
Webinstruct-Verffee
Figure 2: Data creation pipeline: consists QA mining, Extraction and Verification.
StackExchange and various educational portals. Despite WebInstruct's suitability for general instruc- tion tuning, the majority of documents are not directly usable reasoning tasks due to the lack of explicit verifiable answers or require a reasoning process.
To address this, we first trace the entries in WebInstruct back to its original web page to re-crawl precise question-answer pairs. During this process, remove the questions lacking clearly identifi- able human-writte answers on the original source websites. Many websites require membership or complex interaction show the answers, which will filtered out. This careful selection aims to ensure retained entries are verified by humans, enhancing the dataset's reliability and correctness.
Next, further leverage Gemini-1.5-Pro 29 state-of-the-art LLM,, extract questions explicitly identified as having clearly verifiable short answers for single-turn questions. This step yields an intermediate dataset of approximately million verifiable reasoning questions across various disciplines. Subsequently, we apply Gemini-2.0-F2 including the answer type, subject category, and difficulty level. Recognizing the skewed ratio of mathematical tasks, we specifically filter out mathematics problems that are labeled as easier than university-leved ensure more balanced and challenging dataset distribution.
to annotate each question with metadata,
Flost
Mathamatiss
Expresiion
Plysics
Muligle Choice
Chemisry
11.4%
Inseges
Sering
21.5%
Other
Bonlean
Ecunomies
13.6
Permettag
BEA
History
Fuctinn
Biology
3.77
100k
ZAN
430%
2.0
186%
Answer Type Distribution
(b) Domain Distribution
Additionally, recognizing that web-crawled data inherently contains noise, such as questions that are either unsolvable trivially easy, implement further rigorous filtering refine the dataset quality. Specifically, Remini-220-FFaas generates eight candidate solutions for each question, allowing us to apply the following quality control criteria:
We exclude questions for which all eight candidate solutions fail, effectively removing ambiguous noisy questions that likely arise from crawling errors incomplete source content. We also exclude overly simplistic questions for which all eight candidate solutions are correct, ensuring the dataset maintains sufficient complexity and presents meaningful challenges for robust reasoning and generalization during RL training.
The Gemini-2.0 2.0-lahh generated solutions are also later utilized to train our proposed model-based verifier, which will be discussed in detail in the next section.
Eventually, the processed dataset contains approximately 230k reasoning questions spanning di- verse answer formats, including multiple-choice, numerical expressions, matrices as highlighted Figure 3al Figure 3b further illustrates balanced domain distribution of our curated dataset, encompassing disciplines such as mathematics, physics, chemistry, finance, and various other hu- manities and social sciences fields. This rigorous data curation process produces challenging but reliable dataset for training generalizable reasoning capabilities in large language models.

---

Table 1: Examples of reasoning questions where the model provides correct answers, but the rule- based verifier fails to recognize their correctness, while the model-based verifier succeeds.
Example
Example 2
Example
Question
Consider the line perpendicular to the surface s² at the point where = and 1. Find a vector parametric equa- tion for this line terms of the parameter I.
Find the partial pressure in so- lution containing ethunol and propanol with total vapor pres- sure of 56.3 torr. The pure vapor pressures are 100.0 0 torr and 37.6 turr, respectively, tion has mole fraction of 0.300 of ethanol.
What is the work done to push I kg box horizontally for meter surface with coefficient of friction of 0.5?
solu-
Ground Truth Answer
x= St, y= 2= 17 30.0 tont, 26.3 torr
J.9
Student Answer
81, 4
1
17-1
The partial pressure of ethanol is 30.0 torr and the partial pressure I-propanol 26.32
4.9 N-m
Rule Based Verifier
False
False
False
Model Based Verifier
True
True
True
3.2 Model-Based Verifier for GRPO
Preliminary. We adopt Group Relative Policy Optimization (GRPO) [6] following the recent
advancements such as DeepSeek-R1 M. Given question-answer pair (q, a), behavior policy Te.
samples a group of G individual responses (ot) The GRPO objective updates model parameters 0 as follows B
JGRPO (0) D( -o.
1 o.
min (ri:(0)A,,, clip(ri,(())) +c),,.) -BDKt ( -Dol (Tall
Σ
G 1-1
0i
where
To(0r, 1 q, 0i,et) (0)
A Ti mean( (R))
(2)
Tawd (oi, q. or,er)
(t)
In this work, our design on the model-based verifier specifically affects how the reward R is computed.
Limitations of Existing Reward Model. Traditional reward models are trained through human feedback or preference assessment, returning scalar values based on the entire output to indicate overall quality. Although intuitive, these models are suffering from being hacked by the policy model, and usually require the reward model have a large parameter size be effective and robust.
In contrast, rule-based verifiers, widely used mathematical reasoning due to simplicity, evaluate only the final answer, allowing models greater freedom explore diverse reasoning paths. However, these rule-based approaches encounter several critical limitations when extending beyond mathematics:
Rigid Matching Criteria: Rule-based methods typically require exact matches or adherence to rigid structures, failing to recognize semantically equivalent answers that differ in representation. Semantic Insensitivity: They are ineffective at interpreting answers that vary semantically, such equivalent expressions answers expressed in different units or formats.
Lack of Generality: Adapting rule-based verification to a wide range of disciplines and diverse answer formats can be difficult, limiting their applicability and scalability.
Generative Model-Based Verifier. We introduce a compact generative model-based verifier specif- ically trained to robustly assess answer equivalence across diverse domains. Ideally, state-of-the-an -art large language model (LLM) like Gemini-2.0 could verify answer equivalence; however, such solutions are computationally expensive and impractical for large-scale RL training.
Instead, we leverage our dataset creation pipeline, specifically Gemini-2 0 O-generateed candidate answers and verification annotations, to train compact 1.5B --r-amrmtttrr generative verifier model.
5

---

This verifier, initialized from Owen2.5-Math-1e. 1.5B (26], is fine-tuned to assess student- generated short answers (extracted from the response) s against ground-truth references g generative manner, whose inference process is formulated as:
y Ply 9,9,8); I = yeots [sep], Vtabel
where I is a chain-of-thought reasoning process yoot with a final binary prediction Ulabet (true/false) on whether the student answer is equivalent to the ground-truth in the question context.
This verifier integrates seamlessly into our reinforcement learning pipeline, providing robust, accurate reward signals. Empirical analysis confirms that our model- based verifier achieves high agreement with Gemini-2.0- 2.0-Flash, substantially outperforming traditional rule-based approaches.
4 Experiments
4.1
Training
We follow the Zero RL setting, directly conducting reinforcement learning (RL) from base large language models without an intermediate supervised fine-tuning stage. Specifically, we initialize our models using the base model from Qwen2.5 family (7B and 14B) and the newer Qwen3 family (4B and 14B) [22], and apply the GRPO algorithm. Please note that we pick the general Qwen model instead of the Qwen-math math model maximize the general performance, which could lead to seemingly lower performance on mathematical benchmarks. Reward scores during training are calculated as follows:
If the solution extraction fails (c.g.. no boxed answer or summarization such as "the solution is:"), the reward -0.5.
If the solution passes verification, the base reward is 1, with tongth-based penalty applied to discourage excessively long generations. In our early experiments, we notice the model can hack the model-based verifier by putting long content in answer box if there is no length penalty.
penalty -0.05 min abs (len_of dile_of__rooe oww.oof.ddoddooooooooo ttrth len_of_are ____sserr))
Training is conducted on 4 nodes with 8xH100 GPUs per node for up to 700 steps for Qwen2.5 series models. For model initialzied with Qwen3-Bass we train up to 400 steps. Please see Appendix [A.5] for detailed hyperparameters of each model checkpoint. During training, the average model response length increases from approximately 700 tokens to around 1000 tokens. The total training time is around 2 days for the 4B/7B model and around 4 days for the 14B model. Our implementation is based on the verl repository
2
4.2 Evaluation
To evaluate the models' general reasoning capabilities, we conduct a comprehensive assessment across several challenging benchmarks:
MMLU-Pro g]:
A robust and challenging massive multi-task understanding dataset tailored to rigorously benchmark large language models' capabilities.
SuperGPQA [I: A large-scale benchmark targeting graduate-level reasoning across 285 diverse disciplines. All the questions are verified to be not found on Google Search.
BBEH [13] A new benchmark that extends BIG-Bench Hard [30] by introducing more challeng- ing tasks for better evaluation of complex reasoning.
GPQA I0: Graduate-level question answering designed to be resistant to shallow pattern-
matching or memorization. We use the diamond split in GPQA.
TheoremQA [2] Graduate-level question answering designed to require knowing corresponding
theorems. It covers math, physics, EE&CS, and Finance. It's a general reasoning benchmark. Math-Related Tasks: A suite of standard math reasoning benchmarks, including MATH- 500 14. Olympiad [16], Minerva (31,, GSMSK 15), AMC, AIME24, and AIME25. We use the
limplee-evallacealuuutioo framework, and use GPT4o
32 to check the answer equivalence.
http:://ggthhhl.ccc//mmm////sppp

---

Table Aceuracy comparison our GENERAL-REASDNES with baseline methods on general reason- ing benchmarks. MMLU-Pro, SuperGPQA, TheoremQA and BBEH contain multiple subfields.
Model Name
Backbone
MMLU-Pro Miem
GPQA-D
SuperGPQA Macro (discipline)
TheorenQA Acc
BBEH
54.4
38.8
11.4
QwQ-32B
Swen2.5-32R--mme
52.0
54.5
43.6
48.4
22.6
GPT-4o
74.6
46.3
43.6
ol-mini
80.3
60.0
45.2
53.1
DeepSeek- R1
DeepSeek V3
84.0
71.5
59.9
59.1
34.9
Models
Qwen3-4B
Qwen3-4B-Instren (non-think)
Qwen3-4B-o Base
61.8
41.7
32.
42.0
14.9
SONERRLLREASONER---M
Qwen3-4B-Bass
62.8
42.9
32.5
48.3
12.2
Mudels
Qwen2 5-7B-Base
29.1
Qwen2.5 1in25.-77 5- Instrucl
Qwen2.5 7-- Base
57.0
33.8
30.7
36.6
12.2
Open- Honn-Reasnne---ee
Qwen2.5.7B. BBsB
Nemotron-Cros'' CrossThink
Qwen2.5-77BBB 5-B-Bas
57.8
38.5
29.
SimplRL-Qwen22. 5-7--oo
Qwen2.5: 5-B-Base
51.5
24.2
29.9
38.0
GENERAL GENERAL-REASDNERR REASONER-77
Qwen2.5-7B Base
58.9
38.8
34.2
45.3
12.5
14B Models
Qwen2 5 14B Base
53,3
32.8
30.7
33.0
10.8
Qwen2 1.402.5514-Innstrmne
Qwen2.5. Base
41.4
1appleRL-Qwen255-1444
Qwen2.5- (ven2.5-1444 Base
64.0
39.4
35.7
40.8
13.6
GENERAL-REABOWER ReABoNER-QW2.5-11
Qwen2.5-144 1ann2.-14B-BBas
66.6
43.4
44.3
15.2
Qwen3 14B.
Qwen3-14B- Innttrrcccttt (non-think)
Qwen3- 1113-148-Baeee (B-Base
70.9
54.8
39.8
42.4
19.2
GENERAL- REA.AA-RRRASONE-OW--HAAA
Qwen3-14B-BBa
70.3
56.1
39.9
54.4
Table 3: Math task accuracy across datasets. Most evaluations are based greedy decoding except for AIME24 and AIME25 (averaged over sampling runs following SimpleRI. (2).
Model Name
AVG
MATH-500
Olympiad
Minerva
GSMSK
AMC
AIME24
AIME25
Models
Qwen3-4BB Base
40.3
34.8
Qwen3 4B Instruct (non think)
54,2
80.4
49.0
57.0
92.0
62.5
22.5
16.1
GRNERAt.-RRASSE REASONER-44
53.4
80.6
47.7
$7.7
92.2
60.0
20.0
15.4
7B Models
Qwen2.5-7 2-nn2.5-7BBBBB 5-7B-Bse
34.7
60.2
28.6
36.0
83.
30.0
3.8
1.4
Qwen2.5 Instruct
75.0
39.4
48.4
74.0
41.9
49.6
90.7
60.0
15.2
7.5
GENERAL. REASONER-7B
48.5
76.0
37.9
$4.0
92.7
13.8
10.4
14B Models
Qwen2.5 14B Base
37.0
65.4
33.5
24.3
91.6
37.5
3.6
2.9
Qwen2.5-144 5-14B-Insttt
49.9
77.4
44.7
52.2
94.5
57.5
12.2
11.0
SimpleRL. Qwen2.5 14B Zoo
50.7
77.2
44.6
$4.0
94.2
60.0
12.9
11.8
GReERAL. ReAsoxEn-Qu2.55.
Qwen3-14B-Bas
49.9
74.6
44.3
55.9
93.2
14.7
11.4
Qwen3-14BB 14B- (non-think)
57.0
82.0
52.4
59.9
93.9
57.5
28.5
25.1
GEHBRAL- SENSRAL-REASSOEER-OHHH REABONER-QQ.3-148
58.8
83,8
51.9
68.0
94.4
70.0
24.4
19.2
4.3 Baselines
Our main baselines are listed as follows: 1) Qwen2.5 and Qwen3 family (2) We include these model results to understand the gains of our RL training. SimpleRL 2 a comprehensive collection of math-based RL models, 3) Open-Reasoner-ZZrro 33 strong RI reasoning model to replicate DeepSeek-R1 which also focused on math training, Nemotron-Croste Crons [27]: an all-domain reasoning model. This model is the closet to general purpose reasoning model.

---

4.4 Main Results
Table 2 summarizes the performance of our GENERAL- REASONEE across general reasoning bench- marks. Overall, RENERAL.-REASOEER with Zero RI. consistently outperforms both base and super- vised fine-tuned models across the Qwen2.5 and Qwen3 backbones.
For models initialized with Owen2.5-7B-Bart surpassing both the base model at 47.7% and the instructed model at 57.0%. These gains also extend to GPQA and SuperGPQA. Similar improvements appear with the 14B backbone: GENERAL-REASON Qw2.5-14B reaches 66.6% on MMLU-Pro, outperforming Qwen2.5-14B-BB. 14B-tust at 62.7%. It also shows strong results on math-related benchmarks, achieving high average scores for both 7B and 14B variants, as shown in Table Compared to other reinforcement learning methods, GENERAL-REASUN CrossThink across MMLU-Pro, GPQA, SuperGPQA, and BBEH. The trend holds with 14B models, where REASRNE achieves the best overall results.
SINEEAL-REASOONEE achieves 58.9% MMLU-Pro
at 53.3% and Qwen2.5-
consistently outperforms both SimpleRL and Nemotraon-
Stronger results are observed when initializing example, GENERAL-REASONER-- surpasses Qwen2.5-7B after Zero RL, reaching 62.8% on MMLU- Pro versus 58.9%. This demonstrates the efficiency and transferability of our training method across model scales. The best performing version is CENNERALRRESSNEER-AA on GPQA and 54.4% on TheoremQA, matching the performance of commercial models like GPT- 4o, which scores 50.0% and 43.6% respectively, despite relying solely Zero RL. Compared to Qwen3-14P3 1-5--11--nnttue (non-think), which undergoes post-training via distillation from a much bigger teacher model, our model maintains an advantage many benchmarks. Notably, even in non-think mode, the Qwen3-Instrus model continues to generate CoT outputs. While some performance gaps remain relative to closed-source or closed-data systems, results underscore the promise of Zero reinforcement learning when combined with domain-diverse reasoning data and a compact, model- based verifier. Although performance gap remains some benchmarks compared to closed-source or closed-data models, our results underscore the promise of Zero reinforcement learning when combined with dovain-diverse reasoning data and a compact generative verifier.
with Qwen3 backbones. For
which achieves 56. %
Notably, our model does not exhibit overthinking. During training, the average response length grows to around 1,000 tokens, which is significantly shorter than methods like DeepScaleR 4 where outputs can be around 32k tokens. As an example, on the Computer Science split of MML.U-Pro, DeepScaleR-1.5B.. -eeeSealleR-..5B--reeiew requires 18 minutes on 4xH100 GPUs achieve 35% accuracy. In contrast, our GeneRaL- REASONERR-4 finishes in just 1.5 minutes with a higher accuracy of 61 %.
Analysis and Ablation Study
5.1 Impact of Data Abundance
Table 4: Model performance trained with the diverse domain reasoning data vs. math-only data.
Backbone
Data
MMLU-Pro
GPQA
SuperGPQA
Math-Related
Qwen2.5-7B. BBBeas
Full
58.9
34.3
34.2
48.5
Qwen2.5-78 Base
Math Only
56.9
32.8
29.8
49.1
Qwen2.5-14BB Bren2.55-14-Base
Full
66.6
43.4
39.5
53.9
Qwen2.5-144BBBan
Math Only
64.8
38.9
35.6
48.6
To quantify how domain diversity in the training data affects reasoning performance, we compare two variants of our Zero RL setup: one trained on the full, diverse-domain dataset and one trained exclusively on math-related questions. Table reports results for both Qwen2.5-7B-BB -B-Bes and 2wwen..5-14B-.a. backbones under these two conditions.
For the 7B backbone, restricting training to math tasks yields a one point gain on the Math-Related benchmark (49.1% vs. 48.5 48.5%), but comes the expense of general reasoning: MMLU-Pro. GPQA and SuperGPQA each drop by roughly 2 percentage points when compared to the full-data model. In contrast, the model trained on the full dataset achieves stronger performance across all benchmarks, demonstrating more general reasoning capabilities. The 14B backbone exhibits even clearer benefit from data diversity. With full data, REEVERAL-REASDNER-- SEEEER--Q..51114 outperforms its math- only counterpart every metric, improving both general benchmarks. These results confirm that
8

---

60
Table 5: Zero RL training using model- based verifier versus the rule-based verifier on the Owen3-44-Base model for 120 step.
(94)
Dataset
Model-Based
Rule-Based
52
Model-ooe basedd
MMLU-Pro
60.1 1
58.1
26
60
100
120
GPQA
39.4
37.9
Step
SuperGPQA Math-Related
30.5
30.1
Figure 4: MMLU-Pro evaluation score at differ- ent training step using model-based verifier and rule-based verifier.
50.4
50.0
(Ul bazed
rulo based
0.8
modelbbwed
<: Fode-bhened
0.4
0.2
U88
Flant
ceter
Streg
Besideo
Perrentage
Moltipie Chocce
Integer
Hasrte
Fracton
0,0
Physics
Iepoaering
Chemittry
STMM
Heath
Computer Scarced
lostory
Low
Feunce
Cater
Busices
Ecoeenicc
Phlcospory
Cateer
Figure 5: Agreement comparison between rule-based and model- based verifiers against Gemini-22 Flash, conditioned on Gemini verifying answer correct. Left: Agreement by answer type. Right: Agreement by task subject.
training diverse reasoning domains enhances general reasoning while maintaining improving mathematical reasoning.
5.2 Impact of General-Verinee Veriffer on Model Training
We conduct an ablation study to verify the effectiveness of our model-based verifier by comparing models trained using either our model-based verifier or a traditional rule-based verifier Both setups train the Qwen3-4B-Bas training with the model-based verifier achieves higher accuracy across all evaluated benchmarks compared to the rule-based verifier.
model for 120 steps under identical conditions. As shown in Table 5
Figure 4 provides detailed comparison of performance progression the MMLU-Pro dataset throughout training. We observe that the rule-based verifier reaches an early plateau at around step 60 (approximately 58%), while the model-based verifier continues improving. achieving about 60% accuracy by step 120. This demonstrates that the model-based verifier effectively guides reinforcement learning, enabling the model leverage diverse training data better and ultimately achieve stronger overall reasoning performance.
5.3 Verifier Agreement with Gemini-2.0-Fii and Human Judgment
To further assess the advantage of the our model-based verifier over the rule-based verifier, we study their agreement with Gemini-220-Flaan As discussed earlier, a critical limitation of the verifier is its high false-negative rate. In sample of 50k nnswerveerifiaation pairs that Gemini deemed correct, the rule-based verifier achieved only 22.2% agreement on average, whereas our model-based verifier achieved 78.7%.
We further analyze agreement across different answer types as shown TigureeHeeft False negatives most prevalent in unstructured answer types such Lists Strings. Even in structured formats like multiple choice, variations like answers being expressed as textual descriptions rather than option letters can cause discrepancies. In Figure F domain-level analysis shows that the model-based verifier is particularly beneficial for non-math STEM fields like Physics and Engincering, where
'The rule-based verifier is implemented based on httppp://gettah...mmm//gggggg///

---

answer formats are diverse. In contrast, fields such as Economics tend to use more structured answers like multiple choices, which narrows the performance gap between the verifiers.
Table 6: Agreement rates between human judgment and different verifiers 200 samples, grouped by answer type. Scalar Verifier use the same training data as General-Verifie but predict probability of Trus/False scalar value without any chain-of-thoughh
Answer Type
Rule- Based
Gemini Basee
Scalar Verifier
General Verifferr (Ours)
Boolean
0.750
1.000
1.000
1.000
Expression
0.516
0.935
0.806
0.806
Float
0.585
0.892
0.754
0,877
Fraction
0.750
1,000
1.000
1.000
Integer
0.852
0,963
0.926
0,963
List
0,556
0.778
0,778
0.889
Matrix
0.667
1,000
0.833
0.833
Multiple Choice
0.773
0.864
0.864
0,909
Other
0.000
1.000
0.500
1.000
Percentage
0.600
1.000
0.600
0.700
String
0.312
0.938
0.812
0.812
To further substantiate the reliability of our verifiers, we performed a human evaluation of 200 randomly sampled question-answer pairs, comparing the agreement rates of the rule-based, Gemini- based, scalar-based and our General-Verifer against human judgment. As shown in Table [6] our general verifier achieves higher agreement with human annotators than the rule-based and scalar approaches, and is competitive with the Gemini-based verifier across diverse answer types.
5.4 Comparison with Supervised Fine-Tuning
Table 7: Comparison between Zero RL training and supervised fine-tuning (SFT) WebInstruct- Verified.
Method
MMLU-Pro
SuperGPQQ
initialize with Qwen3-4BBBHal
Base
0,516
0.254
w/ WebInstruct SFT
0.439
0.227
w/ WebInstruct- Verified SFT
0.440
0.281
GEneeRAL-REasOeER -EAAASSNNN----3--BBB Qw3-4B
0.628
0.325
initialize Qwen3-14B-BB
Base
0.642
0.365
w/ WebInstruct SFT
0.447
0,254
w/ Weblnstruct- SFT
0.533
0.323
GENVRRAL-REASDNER-OU..B.
0.703
0.399
We conducted ablation experiments to directly comparing trained via supervised fine-tuning (SFT) on both WebInstruut--eriifee and the original WebInstruct datasets. Table 7 reports reporrsss accuracy MMLU-PRO and SuperGPQA of the variants, showing that SFT on either WebInstruct- Verified or the original WebInstruct dataset does not reliably improve performance over the base model and may even cause degradation. In contrast, RL-based training yields significant gain, highlighting the effectiveness of our proposed framework.
against models
Conclusion
In this work, we introduce a training framework for enhancing the general reasoning capabilities of large language models by leveraging a model-based verifier scale verifiable training data across diverse domains. By applying GRPO training directly to base LLMs using our curated, high-quality, verifiable dataset and a generative model-based verifier, we demonstrate competitive reasoning performance against models that require additional supervised fine-tuning stage. Our approach achieves strong generalization across various challenging domains while preserving superior effectiveness in mathematical reasoning tasks.
10

---

References
[1] DeepSeek-Al Team. DeepSeek-R1: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. httpp::tt/////riivvooo http:/awwwwvvrraa////aaaaaaa/a/e abwrg//bb//250111 12948
[2] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL httphhhtp///et arxiv org/ abs/ 2203.250..... 18892
[3]
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjin Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An soun-source Ilm reinforcement learning system at scale, 2025.
URL htttts:////art: org/ hts///bs//000
14476
[4] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li. Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
[5] Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025. Notion Blog.
[6] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: //axxvv. orgoorgg/... abs/
2402.03300
[7] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL ftttss///arxiv
abs/2501.
2501. 19393
[8] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pi pro: A more robust and challenging multi-task language understanding benchmark.
The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:// openreview.o. wttpevviiwneeeeeeeee wwt.:eeefffooooeeeee frretfformmm urum? 1d=y100M622r33
[9] Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. MHm Scaling instructions from the web. In The eighth- Annual Conference on Neural Information Processing Systems, 2024. URL /ttp::tt///pperereiie haca:p/s//ppaarrroeeeeoooo/o//a/
[10] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level moogle-proff q&a benchmark, 2023. URL (tRL|hhtss:// http//axxvvvv.//e wwwxxiv..oog/aae wot./oo/////////eeeo 12022
[11] M-A-P. SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines, 2025. URL https://arxiv. org/
14739
[12] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889-7901, Singapore, December 2023. Association for Computational Linguistics. doi: 18.18653/v 1t122223.mmmy. main.489. URL https://aclantholl
wwg///////222. 20223 emnlp-main 489/
[13] Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoe V. Le, and Orhan Firat. Big-bench extra hard, 2025. URL URLhtttss: //tp///xxvvv.///// org/ abs/2502.115
11

---

[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL wtt.hattpss//ooppevviieeee http:/ewwwwwweeeeeeeeeeeeeeeeeee whee/forummidd///0 http://orruiidd//yttttaasaasse
[15] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL BtRLhttps:/ /tttps://arxivv org/ abs/2110. 14168.
[16] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad- bilingual multimodal scientific problems, 2024. https::tt:///tre
org/ httg//bb//402.. /2022.2222...2 14008
[17] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Ch prompting elicits reasoning in large language models, 2023. URL https: org/ abs/ 20
[18] Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025. URL 24235
https://arxix: org/
[19] Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self calibration 2025. httthttss///rxxit org abs////3......//// 0000
[20] Edward Yeo, Yuxuan Tong, Xinyao Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in LLMs. In ICLR 2025 Workshop on Navigating and Addressing
Data Problems for Foundation Models, 2025. URL http::://ppaueerreeeeeee/fffrrrr
AgtQ1hMQOV
[21] OpenAl Team. Openai ol system card, 2024. URL httpss//araiv..or///b//2211111..
[22] Qwen Team. Qwen2.5 technical report, 2025. URL
[23] Qwen Team. Q-QQ-32B: Embracing the power of reinforcement Iearniin March 2025. URL https:://wenlnn wwiithub..o/blo.gee 1ttthuub.i//bloggggg-32///
[24] Xiaomi LLM-Core Team. Mimo: Unlocking the reasoning potential of language model from pretraining to posttraining. TrXiv:2505.07605 2025. URL https://axxiv..rrr org. http///rriivooor/a////5555 abs//5/b///5 07608
[25] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human http:://arriv..orr//a///00000 org/ 2211222111102.222
feedback, 2022.
/tttps://arxiv...
[26] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via selrovmmprveemnnt, 2024. URL
[27] Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. org/ abs abs/ 13941 https://arxi::
Scaling self-learning beyond math reasoning, 2025. URL
[28] Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding with verifiable rewards across diverse domains, 2025.
httpst:::: / //tpxiv..vv////////a///aaa/aaaaa org/ abs/2503, 23829
12

---

[29] Gemini. Gemini: A family of highly capable multimodal models, 2024. URL https://arxix
org/abs/2312. 11805
[30] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thoughh can solve them, 2022. URL
https://arxiv org/ abs/ /210 09261
[31]
Aitor Lewkowyez, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL
net/forum"s orum weerrmmiddIITTTaae IFXTEEEZRR
[32] OpenAl. GPT-4o system card, 2024. URL https://arxin
hrgg/abs/4410 21276
[33] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeree An open source approach to scaling up reinforcement learning on the base model, 2025, URLhttps:
24299 24290
13

---

A Appendix
A.1
Limitation
This work focuses on the Zero RL setting, where models are trained directly from base language models without intermediate supervised fine-tuning. However, the proposed integration model- based answer verification and diverse-doman reasoning data is not tied to Zero RL. Exploring its impact when combined with intermediate stages, such as supervised fine-tuning or distillation promising to have further performance gain.
Additionally; our study targets general language reasoning tasks across diverse domains. We do not specifically focus on code reasoning or olympiad-level math competitions. Integrating these specialized reasoning domains into framework is valuable avenue for future extension.
In this work, the General-Verifier focuses on verifying short answers. Future work could explore extending it to long-form answers and further investigate the risks associated with reward hacking.
A.2 Broader Impact
This work improves the general reasoning capabilities of large language models, making them more applicable to real-world scenarios where complex, multi-domain reasoning is essential. By enabling LLMs go beyond math and code, our approach allows Al systems better support humans in tasks such as decision-making, analysis, and problem-solving across diverse fields, ultimately increasing their utility in production settings and high-impact applications.
A.3 Detailed Results
We provide detailed evaluation results on MMLU-pro, SuperGPQA and BBEH are listed in below. Following previous work, the average reported in MMLU-Pro is micro-average, and for SuperGPQA and BBEH, we report macro-average across subtask/domain.
Table 8: Per-domain accuracy comparison different models on MMLUUPrr.
Model Name
Asg
CS
Math
Chem
Eng
Law
Bio
Health
Phys
Bus
Phil
Econ
Other
Psy
Hist
Owen3 Base
GENERAL REAScoIEn
62.8
61.7
81.6
69.9
50.9
29.2
79.8
61.0
70.0
71.4
48.9
71.1
51.3
56.3
47.8
Owen2 5- Base
47.7
51.7
58.9
45.1
34,5
25.5
64.6
46.6
50.0
55.4
34.9
57.1
47.6
$4.8
39.1
Owen2 5- -BB-nnstuct
57.0
57.1
71.4
57.3
43.1
31.3
71.4
55.4
60.7
61.8
41.9
67.1
51.8
63.0
48.3
SimploRL Qwell
54,9
GRMFAAI--RRae RR48OVFn-77
58.9
61.5
71.7
66.2
Owen2 5- 14B Base
53.3
54.6
63.0
$2.8
36.0
31.9
71.3
56.5
$2.9
61.1
46.1
64.8
50.1
51.0
44.4
Owen3-1 1-B Base
61.2
70.2
77.4
67.3
48.7
33.2
80.8
65.5
68.7
70.7
53.5
74.4
60.7
69.7
$1.6
Qwen2.5 5- 5-44--nmstaa
SimpleR1
Qww.2. SempleRl-wee22...
5-148-7on
66.1
66.9
49.8
37.2
79.5
67.7
69.5
70,9
DENERAL- RAATAAI-RAAscsn--OOC Sww.oounn--Q...... 14B
66.6
69.8
78.8
67.5
$4.8
39.7
81.7
64.3
71.4
71.9
56.7
74.4
64.00
75.2
59.1
GENERAL REAscHenn Qw3-14B
70.3
73.9
86.5
76.4
55.5
39.9
83.3
70.5
76.1
76.6
$8.3
78.7
66.2
73.3
58.0
Table Per-domain accuracy comparison of different models on SuperGPQA.
Medel Name
Avg.
Eng.
Med.
Sci.
Phil.
Mil. Sci.
Ecan.
Mgnnt.
Socio.
Lit/Arts
Hist.
Agron
Law
Edu.
Qwen3-4B-BBB BBse
25.4
27.3
26.7
26.5
23.9
30.7
29.3
23.1
19.9
15.3
23.1
29.3
28.7
REASOMEn-4D
24.4
Qwen2.5 5-78-Busc
26.7
25.1
23.8
29.7
28.8
29.2
31.7
28.0
21.5
18.2
25.6
27.7
31.6
Quen2. Insmeetttt
30.7
20.2
31.2
27.9
32.3
36.1
32.9
33.7
36.4
24.8
20.5
27.4
51.3
35.1
SimpleRI Qwee2 S-78-Zoo
29.9
28.0
31.3
26.0
34.9
32.2
32.7
31.5
34.3
25.0
23,
27.2
29.4
33.5
34.2
32.3
34.5
31.1
36.3
38.3
36.5
41.3
25.1
23.3
34.2
39.9
Q. 5-14B-Bus
32.3
36.1
QuenT-
36.5
36.1
37.8
41.8
40.3
28.8
36.7
33.6
36.4
37.2
Qwen2. 5- 148--nstt
35.8
35.6
37.1
34.1
38.6
36.
41.8
39.5
39.2
30.7
26.6
12.2
56.1
37.4
SimpleRL.
CompleRl-Qwea2-22 Qwen2 14B-2oo
35.7
34.2
36.7
33.0
36.0
40.0
39.9
41.3
38.5
30.8
26.7
31.1
38.1
38.2
SEASOSER 0w22.11111
39.5
36.5
35.6
41.8
42.9
42.7
32.8
29.7
37,9
39.6
45.3
BRUERSI--SRAROVER... 2.a00
41.5
29.2
14

---

Table 10: Per sub-task accuracy comparison of different models on BBEH.
150
15
438
444
11
43
980
5/
135
48
160
165
140
4.1
283
115
13
18.5
11:0
364
63
04
-33
482
218
00
33
28.5
13
95
96
18
153
83
120
36
180
140
14.5
214
18
465
-89
185
93
400
3.5
130
90
18
258
13
140
15
183
98
13.
360
183
225
23.
140
220
183
A.4
Verifier Input-Output Example
Verifier Input:
User: ### Question: Solve the Bernoulli equation
dy dx y 5. 2
###l Ground Truth Answer:
-2² + Cz
### Student Answer:
y For the above question, please verify if the student's answer is equivalent to the ground truth answer. You don't 't need redo the question yourself, just check if the student's answer equivalent to the ground truth answer. If the student's answer is correct, output "Final Decision: Yes". If the student's answer incorrect, output "Final Decision: No".
Cr x²
Verifier Output:
Assistant: Ground Truth Answer:
= + Ca
2. Student Answer:
1
y
Cr
x²a
We rewrite the ground truth answer
Ca 2?
y Cr x²
Comparing this with the student's answer, that they the Final Decision:
A.5 Detailed Hyper-Parameters
We provide the detailed hyperparameters for training our GENERAL-REASONER variants in Table 11] The difference in batch size configurations between the Qwen2.5 and Qwen3 series is due to the limitations in earlier versions of vLLM (e.g.. v0.6.3), which did not support engine sleep and wake mechanisms for properly loading and unloading the verifier model. As result, we had to dedicate 2 GPUs to the verifier and 6 GPUs to policy model for each node. With the newer vLLM version (v0.8.5), we are able to share GPU resources more efficiently by leveraging proper parameter loading and unloading.
15

---

Table 11: Hyperparameter settings for General-Reasoner variants.
paraneter
GENERAL-REASOHEN 7B
GENERAL REASONZn Qw25-14B
GENZnAL REASDKEn-4B
GESERAL REASCHER Qw3-14B
p_nodes
4
4
5_gp%
SxH1100
BxH1100
SxH1100
8xH1100
backbone, ii
Quam/Owen2.5577
QweniQwen225
Qwemn\Qwen3-4B-
QwantQwen3-1 J4BBBBBBs
version
0.6.3
0.6.3
0.8.5
0.8.5
train, batch size
768
768
1021
1024
1024
1024
1024
1024
rew_respone__1ee
4006
ing_iingrrate
fe-7
apo_nini_baat batch_size
192
192
256
256
apo_nicro_batedd batch_ size
4
4
4
clip_ratio_lon
0.3
0.3
clip_ratio_big
ks, loss_coef
0.0001
0.0001
0,0001
0.0001
kl_loss__ypp
low _l
low_var_ki
low. vur kil
low var
1,0
1.0
0.7
0.7
X
colc
0.001
0.001
0.001
0.001
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated from web-crawled data; and (2) developing generative model-based answer verifier. These are covered by the methods and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
We include the discussion of limitations in Appendix A.1
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification:
This is empirical paper; thus, there is theoretical result included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all the high-level information reproduce models in Sec 4. and Sec 4.2 and due to space limitation, more detailed information are included Ap-
pendix A.5
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA].
Justification:
We will release the code, data, and model checkpoints based acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wttpssubiiiiioooooosoooooiooe for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/iips.cc/public//uue
Codeeuumiiii wettsbbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide training details in S.c Sec 4.2 as well as Appendix A.5
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA].
Justification: We follow the evaluation setting of commonly used benchmarks such as MMLU-Pro, the model comparison is based on the average of multiple sub-tasks evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the required computation resources in Sec 4.1 and Appendix A.5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss//worrss.../paaal///taaaas
Answer: [Yes]
Justification:
We have reviewed and made sure our paper conforms the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included the discussion broader impacts in Appendix A.2 A.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this paper poses such risks.
Guidelines:
21

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: have properly cite the usage existing assets and respect the license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssiiipssiiiiaea... phcerssithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The dataset we create is described in details in Sec 3.1
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
22

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "RfMgti6xQj", "reviewer": "Reviewer_nUFT", "summary": "This paper proposes **General-Reasoner**, a reinforcement learning framework for improving the general reasoning abilities of language models across diverse domains—ranging from math and science to social science and finance. Unlike standard RLHF pipelines, the method performs reinforcement learning **without any supervised fine-tuning**, a setting the authors call “Zero-RL.”\n\nThe paper introduces two main components:\n\n1. **WebInstruct-Verified Dataset**: A large-scale, high-quality, and verifiable multi-domain QA dataset collected and filtered using LLMs (e.g., Gemini) to ensure answerability and consistency.\n2. **Generative Verifier**: A lightweight, CoT-based model trained to semantically evaluate the correctness of answers. It serves as a reward model during RL training via GRPO (Group Relative Policy Optimization).\n\nThe method is tested on 12 benchmarks, including MMLU-Pro, GPQA, SuperGPQA, and various math datasets. It achieves state-of-the-art or highly competitive performance across the board—even with smaller models (e.g., Qwen3-4B)—without using any instruction-tuning.", "questions": "1. Why does Zero-RL work so well at 4B scale in your setting? Could you provide analysis (e.g., with/without verifier, or smaller RL steps) to explain?\n\n2. Have you tested whether the model is overfitting to the generative verifier's patterns? For example, can it fool the verifier with shallow patterns?\n\n3. How robust is the generative verifier to answer formatting, over-precision (e.g., significant figures), or semantic ambiguity?\n\n4. How would you position this work in relation to DeepSeek Zero or other recent Zero-RL efforts?", "limitations": "No limitation section", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Conceptual clarity and practical significance**: The method aims to train a general-purpose reasoning agent without SFT, a meaningful contribution given the costs and biases introduced by supervised pipelines.\n\n- **Innovative dataset and reward design**: The use of LLM-filtered, verifiable multi-domain questions and a generative verifier offers a new approach to constructing diverse and semantically grounded reward signals.\n\n- **Strong empirical performance**: General-Reasoner consistently outperforms base models and strong RL baselines across diverse tasks and domains, even when using small-scale models like 4B.\n\n- **Readable and well-structured**: The paper is clearly written, with logically organized sections, clear diagrams, and helpful qualitative examples.\n\n---\n\n## Weaknesses\n\n- **Related work on Zero-RL is not acknowledged**: Recent work such as **DeepSeek Zero** also demonstrates RL training without SFT. While General-Reasoner offers distinct innovations (e.g., the generative verifier and diverse domain coverage), the paper does not acknowledge or contrast itself with these methods, leaving its novelty under-specified.\n\n- **Unclear why Zero-RL works at small scale**: The paper shows that 4B models trained with Zero-RL achieve strong results, but does not investigate *why*. Are the gains due to data quality, the verifier, the GRPO algorithm, or something else? No ablations or training curve analysis are provided to support this important finding.\n\n- **Potential risk of reward hacking is not addressed**: Model-based reward signals are known to be vulnerable to “hacking” (i.e., models learning to game the verifier without improving actual reasoning), as highlighted in recent work (e.g., arXiv:2505.22203). The paper does not evaluate whether this risk affects its results, nor does it include human or external evaluation to verify reasoning quality.\n\n- **Verifier correctness is assumed but not deeply evaluated**: The paper claims that the generative verifier is more flexible than rule-based ones, which is likely true. However, it does not assess whether the verifier enforces domain-specific norms—e.g., in a chemistry case where it marks “26.32” correct instead of “26.3,” violating significant figure rules. This could lead to inaccurate training feedback.\n\n- **WebInstruct-Verified dataset lacks transparency**: While the data curation pipeline uses multi-answer voting and LLM filters, the thresholds for acceptance, coverage across domains, and error rates are not reported. No human validation is used to ensure that retained samples are factually or pedagogically sound.\n\n- **No statistical significance or variance reporting**: The paper reports average accuracies but omits standard deviations or confidence intervals. Given that generalization is known to be sensitive to seeds and sampling, this limits trust in the results.", "was_revised": true, "final_justification": "I carefully read the rebuttal, other reviewers' comments, and the manuscript once again. I still found that the technical convergence of the paper to prior art is limited (the rebuttal on these points is not satisfactory), and needs a clearer explanation on why the paper can increase general reasoning ability, with a precise distinction from prior works. However, I also agree that the experimental results presented in the paper are worth noting for the community, and thus increase the score from 3 to 4.", "rebuttal": "We sincerely thank the reviewer for acknowledging our strengths in innovative dataset creation and reward design, as well as our empirical performance. We also appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:\n\n1. **“Related work on Zero-RL is not acknowledged” (W1, Q4)**\n    \n    Our work builds upon recent Zero-RL methods, including DeepSeek Zero and SimpleRL, as acknowledged in the Introduction and Related Work (e.g., line 26, line 102). However, our primary contribution differs in both motivation and execution. While existing Zero-RL approaches primarily focus on mathematical reasoning, we explicitly aim to enhance general reasoning across diverse domains by significantly expanding verifiable training data and evaluations. We will further clarify this distinction in the related work section.\n    \n2. **“Unclear why Zero-RL works at small scale” (W2, Q1)**\n    \n    Figure 4 presents the training curve for the Qwen3-4B model. The initial rapid gains likely stem from the model aligning its answer formats with expectations, while continued improvements increasingly depend on data quality and verifier effectiveness. We will include more detailed discussions results section.\n    \n3. **“Potential risk of reward hacking is not addressed” (W3, Q2)**\n    \n    In the early stages of our experiments, we did observed risks of reward hacking. The policy model tended to inject long content into the latex box for short answers. This resulted in increasing training rewards but poor test performance. \n    \n    As mentioned in line 200, our reward design thus includes penalties for answers that significantly deviate in length from the ground truth short answers. This constrains the model's ability to inject hacking content intended to mislead the verifier.\n    \n    Our comprehensive evaluation across various downstream reasoning tasks demonstrates that our verifier-guided training produces robust performance improvements, suggesting that the resulting model is not affected by the potential risk.\n    \n4. **“How robust is the generative verifier to answer formatting” (W4, Q3)**\n    \n    In Figure 5, we evaluate the agreement between our smaller generative verifier and the Gemini-based verifier, demonstrating high alignment between the two.\n    \n    However, we acknowledge the reviewer's concern regarding the assumed correctness of the verifier.\n    \n    Regarding the reviewer's example of a chemistry case where \"26.32\" is marked correct instead of \"26.3,\" this illustrates precisely why we need a model-based verifier that makes judgments conditioned on question context. \n    \n    To assess verifier quality, we manually evaluated 200 randomly sampled question-answer pairs, comparing the verification agreement of different verifiers with human verification. Below are the results:\n    \n    | Answer Type | rule_verifier | gemini_verifier | general-verifier (ours) |\n    | --- | --- | --- | --- |\n    | Boolean | 0.750 | 1.000 | 1.000 |\n    | Expression | 0.516 | 0.935 | 0.806 |\n    | Float | 0.585 | 0.892 | 0.877 |\n    | Fraction | 0.750 | 1.000 | 1.000 |\n    | Integer | 0.852 | 0.963 | 0.963 |\n    | List | 0.556 | 0.778 | 0.889 |\n    | Matrix | 0.667 | 1.000 | 0.833 |\n    | Multiple Choice | 0.773 | 0.864 | 0.909 |\n    | Other | 0.000 | 1.000 | 1.000 |\n    | Percentage | 0.600 | 1.000 | 0.700 |\n    | String | 0.312 | 0.938 | 0.812 |\n    \n    These results demonstrate the high accuracy and robustness of our generative verifier across diverse answer formats. This human evaluation further confirms the significant advantage of our generative verifier compared to traditional rule-based methods. We will incorporate this detailed analysis based on human evaluation into the revised manuscript.\n    \n5. **“No statistical significance or variance reporting” (W6)**\n    \n    Our evaluation protocol adheres to established practices in prior research. For high-variance datasets like AIME24/25, we report performance averaged over 32 runs, consistent with methods such as Simple-RL. For comprehensive benchmarks like MMLU-Pro and SuperGPQA, we report averages across large test sets containing more than 10,000 queries each, following the protocols established by the original datasets.\n    \n6. **“WebInstruct-Verified dataset lacks transparency” (W5)**\n    \n    We would like to highlight that the WebInstruct-Verified dataset will be publicly released. To further improve transparency, we will also provide the unfiltered dataset and detailed outputs from the Gemini model used in data curation. This additional information will ensure complete transparency in our dataset creation process.\n    \n\nWe hope our response has addressed your main questions and concerns. Your suggestions have helped us improve the paper. We welcome further discussion during the reviewer-author discussion period."}, {"review_id": "lMwbHP5sBN", "reviewer": "Reviewer_87sm", "summary": "This paper proposes General-Reasoner, a new Zero-RL training paradigm for improving the reasoning capabilities of LLMs across diverse domains. To achieve this, a large-scale, diverse-domain dataset with high-quality reasoning questions from web resources is constructed. Additionally, a generative verifier model is trained for evaluating answers, providing robust and reliable reward signals for RL training.", "questions": "n/a", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tThis paper tackles an important gap in LLM reasoning that most prior work has primarily focused on mathematical and coding domains due to the abundance of data and the ease of answer verification. \n-\tThe curated dataset, comprising 230K high-quality reasoning questions across various domains (e.g., physics, chemistry, finance), is a significant contribution and will likely benefit future research in general-domain reasoning.\n-\tExperiments are comprehensive, involving a wide range models, such as Qwen2.5 and Qwen3 family, RL models, reasoning models, and multiple evaluation benchmarks (e.g., MMLU-Pro, SuperGPQA). General-Reasoner outperforms baselines on most datasets. Additionally, ablation studies effectively analyze the impact of data abundance and model-based verifier.\n-\tThe paper is well-written and easy to follow.\n\n**Weaknesses**\n\n-\tThe technical novelty is relative limited, but the main contributions lie in the curated dataset, the new Zero-RL training paradigm with a compact model-based verifier, and thorough experiments.\n-\tThe data collection process heavily relies on Gemini models. Specifically, Gemini-1.5-Pro is used to extract questions, Gemini-2.0-Flash is applied to annotate each question, and solutions are generated by Gemini-2.0-flash. The potential issues introduced by these models, such as factual errors or biases, are not addressed. There is a lack of quality control in the collected data. It would benefit from including human evaluation, even on a small subset, to assess data quality.", "was_revised": true, "final_justification": "I have read the authors' response and agree that the limitation in human-based quality control should be discussed in the paper.\n\nI will retain my score to reflect my support for this work.", "rebuttal": "We sincerely thank the reviewer for the valuable and encouraging feedback acknowledging our contribution in tackling an important gap in LLM reasoning. Below is our point-by-point response to the comments:\n\n1. **“The technical novelty is relative limited, but the main contributions lie in the curated dataset, training paradigm and experiments.” (W1)**\n    \n    We agree with the reviewer that our primary contributions emphasize providing a comprehensive and scalable framework rather than technical novelty. Specifically, we contribute a diverse, high-quality dataset for RLVR training across multiple domains, alongside a cohesive training paradigm that integrates \"verifiable data\" with a \"small model based verifier.\" This integration enhances scalability across various data sizes and domains while simplifying the overall approach.\n    \n2. **“There is a lack of quality control in the collected data.” (W2)**\n    \n    We acknowledge that the absence of human-based quality control is a limitation in our dataset curation process. Our approach relies on automatic filtering, including dropping questions that Gemini models consistently fail to solve across multiple (8) attempts. This filtering strategy helps ensure that questions in the final dataset are solvable and free from errors. We will add a discussion about implementing further human quality control in the limitations section."}, {"review_id": "Ylut1QS99g", "reviewer": "Reviewer_MFcj", "summary": "General-Reasoner introduces a zero-RL fine-tuning framework that leverages a newly curated, 230 K-question, multi-domain reasoning dataset (“WebInstruct-verified”) and a 1.5 B-param generative verifier to provide reward signals across diverse answer formats; by directly applying policy optimization to base LLMs (Qwen2.5-7B/14B and Qwen3-4B/14B), it achieves consistent ~10 perc gains over baselines on 12 STEM and humanities benchmarks.", "questions": "1. Did the authors compare the verifier against Gemini generated answers?\n2. What if the verifier is trained only with open LLMs? All curation/verifier training described uses Gemini, no ablation with only open-source models.\n3. Is generative verifier better than scalar RLVR?\n4. Over-reliance on a single verifier model backbone: are there any ablations on alternative verifiers?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\nThe data collection effort could be valuable to the RL community. It contributes to the endeavor of upscaling verifiable rewards \n\n## Weakness\nThis work is a solid engineering effort that scales verification-based RL across broader question types. But its core contributions are largely in data collation, not in new RL algorithms or understanding model reasoning. This work definitely going to make a solid dataset track paper if it were submitted to that track. \nThey don’t isolate the RL fine-tuning from the dataset or verifier. Without showing how much of the performance is from increased data or from RL dynamics, it's hard to credit RL for true reasoning gains.\nAmong all baselines, the main complaint is that there is no standalone “Distill” comparison: Across all reported results (e.g., Tables 6–8), the paper only evaluates un fine-tuned “Base” models, supervised-fine-tuned “Instruct” models, and zero-RL variants (SimpleRL, General-Reasoner). There is no pure knowledge-distillation / direct SFT (and other no-RL) baseline to isolate the effect of the RL procedure itself.", "was_revised": true, "final_justification": "Still, my perspective is that the paper’s main contribution lies in the dataset curation. While I recognize its value, I believe the work aligns more naturally with the scope of the dataset track. As such, I find it less compelling as a main track paper. My borderline accept rating reflects this—acknowledging the merit of the dataset, but with reservations about its fit for the main track.", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution to scaling RLVR across diverse domains. We appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:\n\n1. **“They don’t isolate the RL fine-tuning from the dataset or verifier.”**\n    \n    In Table 4, we isolate the effect of dataset diversity by fixing the verifier and comparing performance between the full training dataset and a math-only dataset. The results clearly demonstrate that scaling training data across diverse domains improves general reasoning capabilities. Similarly, in Table 5, we isolate the effect of verifier quality by fixing the dataset and comparing our model-based verifier against a rule-based approach. This comparison highlights the advantages of using a model-based verifier for handling diverse domains. Through these controlled ablation studies, we separately examine the impact of dataset scale and verifier quality on model performance.\n    \n2. **“There is no pure knowledge-distillation / direct SFT (and other no-RL) baseline to isolate the effect of the RL procedure itself.”**\n    \n    Our primary goal is to evaluate the proposed Zero-RL framework, where the model learns exclusively from questions and verifiable answers without supervised CoT labels. Therefore, our main baselines include Zero-RL methods, such as SimpleRL.\n    \n    We compared our approach against models like Qwen2.5-14B-Instruct, which were trained with supervised fine-tuning (SFT). Our model consistently outperforms these models, highlighting the benefits of our RL-based training. This is particularly significant considering our training data consists of questions with verifiable answers but no explicit CoT supervision.\n    \n    To further enrich our analysis, we conducted additional experiments:\n    \n    - Training models directly with SFT to generate answers without reasoning steps.\n    - Training models with the original WebInstruct dataset (without verifiable answer extracted), which provides a relatively direct comparison between CoT-SFT and Zero-RL approaches.\n    \n    The following table presents additional experimental results for Qwen3-4B-Base and Qwen3-14B-Base models:\n    \n    | Model | Post Training | MMLU-PRO | SuperGPQA |\n    | --- | --- | --- | --- |\n    | Qwen3-4B-Base | - | 0.516 | 0.254 |\n    | qwen3-4b-webinstruct-verified-sft | SFT | 0.4397 | 0.281 |\n    | qwen3-4b-webinstruct-sft | SFT | 0.4386 | 0.227 |\n    | General-Reasoner-Qw3-4B | RL | 0.628 | 0.325 |\n    |  |  |  |  |\n    | Qw3-14B-Base | - | 0.642 | 0.365 |\n    | qwen3-14b-webinstruct-verified-sft | SFT | 0.5332 | 0.323 |\n    | qwen3-14b-webinstruct-sft | SFT | 0.4465 | 0.2542 |\n    | General-Reasoner-Qw3-14B | RL | 0.703 | 0.399 |\n    \n    These results show that SFT on strong base models like Qwen3-4B-Base and Qwen3-14B-Base may even have a negative impact, while RL over our curated dataset consistently improves performance compared to the base model.\n    \n3. **“Did the authors compare the verifier against Gemini generated answers?”**\n    \n    Yes, our verifiers were evaluated against Gemini-generated answers. We will add more detailed information about this comparison in Section 5.3 of the manuscript.\n    \n4. **“How is generative verifier v.s. scalar RLVR.”**\n    \n    To clarify the effectiveness of our generative verifier compared to scalar verifier, we trained a scalar verifier with the same backbone and training data and evaluated its **agreement with the Gemini-based verifier** (matching Figure 5). The scalar verifier judges answers based on the probability of Pr(True).\n    \n    | Answer Type | rule_verifier | scalar_verifier | model_verifier |\n    | --- | --- | --- | --- |\n    | Boolean | 0.233 | 0.851 | 0.882 |\n    | Expression | 0.192 | 0.542 | 0.743 |\n    | Float | 0.105 | 0.477 | 0.755 |\n    | Fraction | 0.699 | 0.839 | 0.924 |\n    | Integer | 0.460 | 0.680 | 0.829 |\n    | List | 0.065 | 0.472 | 0.736 |\n    | Matrix | 0.515 | 0.669 | 0.840 |\n    | Multiple Choice | 0.336 | 0.761 | 0.869 |\n    | Other | 0.106 | 0.365 | 0.681 |\n    | Percentage | 0.277 | 0.502 | 0.683 |\n    | String | 0.187 | 0.668 | 0.826 |\n    \n    These results demonstrate that our generative verifier consistently outperforms the scalar verifier across all answer types. This performance advantage can come from the generative verifier's ability to leverage chain-of-thought reasoning during the verification process.\n    \n    In addition to automatic evaluation comparing against the Gemini verifier, we also conducted human evaluation on 200 randomly sampled question-answer pairs to assess the **agreement with human judgment** as presented below:\n    \n    | Answer Type | rule_verifier | gemini_verifier | scalar_verifier | general-verifier (ours) |\n    | --- | --- | --- | --- | --- |\n    | Boolean | 0.750 | 1.000 | 1.000 | 1.000 |\n    | Expression | 0.516 | 0.935 | 0.806 | 0.806 |\n    | Float | 0.585 | 0.892 | 0.754 | 0.877 |\n    | Fraction | 0.750 | 1.000 | 1.000 | 1.000 |\n    | Integer | 0.852 | 0.963 | 0.926 | 0.963 |\n    | List | 0.556 | 0.778 | 0.778 | 0.889 |\n    | Matrix | 0.667 | 1.000 | 0.833 | 0.833 |\n    | Multiple Choice | 0.773 | 0.864 | 0.864 | 0.909 |\n    | Other | 0.000 | 1.000 | 0.500 | 1.000 |\n    | Percentage | 0.600 | 1.000 | 0.600 | 0.700 |\n    | String | 0.312 | 0.938 | 0.812 | 0.812 |\n    \n    These results further confirm that our general verifier is more effective than the scalar verifier.\n    \n5. **“Alternative verifiers? What if the verifier is trained only with open LLMs? “**\n    \n    Our main contribution is scaling Zero-RL training across general domains, with the verifier being a secondary contribution. Future work could explore alternative training strategies or using open-source LLMs for verification, as well as testing various combinations of verifier models. We will acknowledge this limitation and outline these future directions in our manuscript.\n    \n\nWe hope our responses have addressed the reviewer's main questions and concerns. We welcome further discussion during the reviewer-author discussion period."}, {"review_id": "lxoZwNMoBf", "reviewer": "Reviewer_JeZ6", "summary": "This paper tries to extend RL training of LLMs beyond verifiable domains like math and coding where rule-based rewards can be applied. \n\nTo do this, they first constructed a general-domain training set by filtering WebInstruct-verified, and then trained a small generative reward model as the answer verifier for the general-domain QA task so that computing the reward can be inefficient. The small generative reward model is based on Qwen2.5-Math-1.5B and distilled from Gemini-2.0. They then trained the model with standard GRPO (without any warmstart with SFT). \n\nThe trained models (named General-Reasoner) gets decent improvement as compared to the Base models on MMLU-Pro, GPQA, SuperGPQA, and BBEH in general. And in fact, they get improvement on math benchmarks too probably due to the learned reasoning behaviour.", "questions": "- What's the SFT data for \"Qwen3-4B-Instruct\" in the second block of Table 2? Is that just one of the official relased checkpoints from Qwen?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Weakness:\n\n- I think at this point we all agree that Qwen models are outliers in the sense that their mid-training probably made them extremely suitable for RL. So I do think it's a fair ask to request additional experiments on non-Qwen checkpoints to verifiy the generalizability of the proposed method. \n\n- You might want to justify the novelty of the work a bit more: For example, in \"Demystifying Long Chain-of-Thought Reasoning in LLMs\", there are also RL experiments on general-domain QA data (Section 5) and they also used WebInstruct data. What's the fundamental difference apart from your own small verifier model? \n\nStrength:\n\n- Thorough experiments and a trendy topic \n- Good empirical results\n\n---------------------------------------------------------------\nUpdate on 7/4: I was prompted by the AC to add more reasons for rejection, so I'm adding more thoughts here. Quite frankly, I think the paper is offering some interesting empirical results for the RLVR community, and the execution is generally ok; it just doesn't quite hit the scientific standard I was hoping for in a NeurIPS paper. I will share some more thoughts/questions that will hopefully be helpful: \n\nThe paper has two main contributions: (1) curating a general-domain training set for doing RLVR beyond math tasks; and (2) training a small model-based answer verifier as the reward model in RL, rather than using rule-based rewards. \n\nFor (1): \n\n- In your filtering step, you mentioned you only kept examples where you can find human-written answers in the original websites. How exactly was this step implemented? Did you use an LLM filter to check the original websites?  \n\n- Quite frankly, a significant portion of the dataset is still math problems (34%). \n\n- How do you justify the quality of the final dataset? And how do you justify its necessity? Do we really need RL data on general-domain questions? Because if you look at the paper \"Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning\", in their Figure 1, even if you RL on math data, you still get improvement on non-math tasks. If you wanna claim this new training dataset is a contribution, shouldn't you do some data ablation to justify its importance? Does training on your filtered WebInstruct data offer additional gains beyond simply training on existing math data? \n\nFor (2):\n\n- I like Table 5 in the sense that it's clearly telling me what's the advange of model-based verifier vs rule-based verifier. It'd be nice if there could be some results of using an \"oracle\" reward too. For eg, by using Gemini as the verifier there. In a way, I really like the various ablations being done in the paper \"Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning\". They showed clearly the gains and potential reward hacking of using model-based verifiers. Do you think your work is offering additional new insights beyond their results? (Apart from experimenting it on different datasets?) \n\nIn general, I would happily raise my score if you could tell me concretely the list of scientific insights that your paper is offering beyond what's already known by previous RLVR papers. For to give you some few-shot examples, here are some examples extracted from the papers I mentioned above:\n\n- RL training on math can get transferable improvement on other non-math benchmarks or even non-reasoning benchmarks (from \"Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning\") \n- Using model-based verifier in RL training can be better than rule-based verifiers, but in some cases, there is reward-hacking (from \"Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning\". \n- You can do RL on general-domain data (WebInstruct) with either model-based verifier or rule-based verifier, ideally on a filtered set of prompts with short answers, and it can get improvement on general benchmarks like MMLU-Pro (literally Takeaway 5.2 from \"Demystifying Long Chain-of-Thought Reasoning in LLMs\"). \n\nIn general, I was hoping for a scientific paper beyond \"we gathered a new training dataset by filtering an existing dataset, and finetuned a new model with existing training algorithms, and it works slightly better on some benchmarks\". Or at least this shouldn't be the narrative of the paper IMHO. Happy to discuss more during rebuttal!", "was_revised": true, "final_justification": "The additional comparison of SFT vs RL on WebInstruct (SFT version vs RL version of the dataset) is interesting. \nAlso, I did not realize the fact that some of the related works are released AFTER this paper. \nI raised my score to 4 accordingly.", "rebuttal": "We sincerely thank the reviewer for highlighting our empirical results and our contribution in scaling data for RLVR training across diverse domains. We also appreciate the valuable feedback and comments. Below is our point-by-point response addressing each comment raised:\n\n1. **“What's the SFT data for \"Qwen3-4B-Instruct\" in the second block of Table 2? Is that just one of the official relasedcheckpoints from Qwen?”**\n    \n    Yes, the Qwen3-4B-Instruct baseline is one of the official released checkpoints from Qwen. We will add further clarification in the experiment setup. \n    \n2. **“How exactly was [kept examples where you can find human-written answers] implemented?”**\n    \n    Thanks for raising this question, here what we trying to say is not guarantee “human” answer in contrast to “synthetic” answers. As the source data is from webcrawling rather than LLM generated, it is naturally “human-written” contents, however, we use Gemini-Pro to ensure that the completeness of human-written answers for the extracted questions. We will update the description to improve the clarity.\n    \n3. **“a significant portion of the dataset is still math problems (34%). “**\n    \n    While our dataset encompasses diverse reasoning domains, we maintained a significant portion (34%) focused on math problems. This decision is based on the fact that many STEM-domain tasks depend on mathematical reasoning abilities. Nevertheless, we highlight that 66% of our dataset covers non-mathematical domains, which represents a substantial departure from existing works that focused only on mathematical reasoning.\n    \n4. **“How do you justify the quality of the final dataset? shouldn't you do some data ablation to justify its importance? “**\n    \n    In Table 4, we compare performance when using our full training dataset versus a math-only subset while keeping the verifier fixed. The results demonstrate the effectiveness of scaling training data across multiple reasoning domains. Our method also outperforms previous approaches like SimpleRL, which primarily focuses on math-only data, further validating the importance of incorporating diverse reasoning tasks.\n    \n5. **“insights beyond existing / concurrent works”**\n\n    1. **Demystifying Long Chain-of-Thought Reasoning in LLMs**\n        \n        *Preprint Date: Feb 5th, 2025 (concurrent work within 3 month)*\n        \n        We would like to clarify that the WebInstruct dataset used in the referenced work was designed for supervised fine-tuning (SFT). In contrast, our WebInstruct-Verified dataset specifically targets high-quality RLVR training data. To demonstrate the effectiveness of our approach, we conducted additional experiments comparing SFT training with WebInstruct against RL training using WebInstruct-Verified on Qwen3-4B and Qwen3-14B.\n        \n        | Model | Post Training | MMLU-PRO | SuperGPQA |\n        | --- | --- | --- | --- |\n        | Qwen3-4B-Base | - | 0.516 | 0.254 |\n        | qwen3-4b-webinstruct-sft | SFT | 0.4386 | 0.227 |\n        | General-Reasoner-Qw3-4B | RL | 0.628 | 0.325 |\n        |  |  |  |  |\n        | Qwen3-14B-Base | - | 0.642 | 0.365 |\n        | Qwen3-14b-webinstruct-sft | SFT | 0.4465 | 0.2542 |\n        | General-Reasoner-Qw3-14B | RL | 0.703 | 0.399 |\n        \n        The results demonstrate that using our curated data with the proposed training framework is more effective than applying SFT over WebInstruct. Based on these findings, we conclude the following takeaway:\n        \n        **Insight**: *The curation of extract verifiable question-answer pair to create diverse domain RLVR training data is more effective than directly apply on crawled instruction following data such as WebInstruct.* \n        \n    2. **Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning**\n        \n        *Preprint Date: May 28th, 2025 (follow up work of ours)*\n        \n        This referenced work is actually a follow-up study to ours, and notably, our verifier model was extensively employed in their experimental comparisons. Their experiments validate that our high-quality model-based verifier performs well compared to other verifiers. We highly appreciate their additional analyses and insights.\n        \n        Our primary contribution is scaling Zero-RL training across general domains and evaluating the broader improvements in reasoning capabilities of LLMs. While an extensive characterization of model-based verifiers is beyond the scope of our current work, especially since their study specifically addresses mathematical reasoning, we aim to clearly deliver this key takeaway:\n        \n        ***Insight**: Model-based verifier have important role in help scaling the RLVR training data across diverse domains.*\n        \n    3. **Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning**\n        \n        *Preprint Date: July 1th, 2025 (follow up work of ours)*\n        \n        Their work includes General-Reasoner in its major evaluation, which demonstrates strong transferability. We find their analyses of transferability particularly impressive.\n        \n        Returning to our research, our findings align with evidence that mathematical reasoning enhances general LLM capabilities. However, we demonstrate significant additional benefits from scaling RL training to include diverse reasoning tasks. Comparing SimpleRL-14B with Qwen2.5-14B-Base on MMLU-PRO shows that math-focused RL improves general reasoning performance. Our General-Reasoner-Qwen2.5-14B further surpasses these results, highlighting the substantial gains from broader-domain reasoning training. Additionally, Table 4 demonstrates the improvements achieved by incorporating diverse data compared to using math-only data.\n        \n        ***Insight**: While math reasoning enhances general LLM capabilities, scaling training to include diverse reasoning domains significantly increases effectiveness.*\n        \n    \n    We sincerely thank the reviewer for bringing up these comparisons, which enrich the discussion of our paper. We hope that the above analysis of the suggested comparisons clearly demonstrates the specific contributions and insights our work provides."}, {"review_id": "DzcZ80Ff9f", "reviewer": "Reviewer_nfc4", "summary": "This paper introduces General-Reasoner, a novel training paradigm designed to enhance the reasoning capabilities of large language models (LLMs) across diverse domains. It addresses the limitations of existing methods, which primarily focus on mathematical and coding tasks. The authors construct WebInstruct-verified, a large-scale, high-quality dataset spanning multiple disciplines, and develop General-Verifier, a generative model-based verifier that replaces rigid rule-based verification. This approach enables robust reward signals for reinforcement learning (RL).\nComprehensive evaluations demonstrate that General-Reasoner outperforms baseline methods on general reasoning benchmarks such as MMLU-Pro and SuperGPQA, while maintaining strong performance in mathematical tasks. The work contributes a scalable framework for improving LLM reasoning across a broad range of domains.", "questions": "1. Clarify the innovation of the General-Verifier: Specifically, how does it differ from existing verifiers in terms of architecture, training, or performance?\n2. Extend training steps in Figure 4: The sharp performance rise at step 120 may be coincidental. Could you train for more steps (e.g., 200+) or until convergence to validate robustness?\n3. Separate Qwen3-14B and Qwen2.5 results: Grouping them together may obscure performance differences. Please disaggregate the results for fair comparison.\n4. Expand baseline comparisons: Include more results for Qwen3-14B baselines to better contextualize General-Reasoner’s improvements.\n\ntypos: There is a typo in the OpenReview abstract where '\\model' is not rendered correctly.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "## Strengths\n\n1. The paper introduces a novel model-based verifier that significantly improves answer verification across diverse domains, addressing a key limitation of rule-based methods.\n2. The construction of a large-scale, high-quality dataset (WebInstruct-verified) enhances the scope of reasoning tasks beyond mathematics, contributing valuable resources for future research.\n3. Comprehensive evaluations across multiple benchmarks demonstrate the model's robust performance and generalization capabilities, particularly in non-mathematical domains.\n\n## Weaknesses\n\n1. The use of a model-based verifier (or reward model in RL) to replace rule-based verification is not novel, as evidenced by prior works like [1][2][3], and has been applied even in math-focused tasks.\n2. GRPO already employs model-based reward systems (as shown in [4], Table 10), making the proposed modification less innovative.\n3. Table 4 shows only modest improvements on MMLU and GPQA when using diverse data versus math-only data, and the comparisons with Qwen3-14B lack strong baselines, raising doubts about the method's impact.\n4. The answer-pair verification approach is limited to QA-style tasks with short answers, failing to generalize to broader tasks like summarization, writing, or translation.\n\n[1] VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models\n\n[2] xVerify: Efficient Answer Verifier for Reasoning Model Evaluations\n\n[3] Compassjudger-1: All-in-one judge model helps model evaluation and evolution\n\n[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", "was_revised": true, "final_justification": "Based on the author's rebuttal, points 3 and 4 have been resolved, so I have decided to raise my overall rating.", "rebuttal": "We sincerely thank the reviewer for highlighting the contribution of our datasets and the effectiveness of the model-based verifier in RLVR training. We also appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:\n\n1. **The use of a model-based verifier (or reward model in RL) to replace rule-based verification is not novel, as evidenced by prior works like [1][2][3], and has been applied even in math-focused tasks.**\n    \n    The works mentioned, such as VerifyBench [1] and xVerify [2], are very close concurrent works to ours, published within 3 months of the submission deadline. Their preprints were actually released after the first version of our preprint. \n    \n    Beyond timing considerations, our work's main contribution is scaling RLVR data across diverse domains, with the generative model-based verifier serving as a solution that effectively supports this data scaling. This differentiates our approach from previous works (e.g., [3]) that only focus on improving answer verification in the math domain. Our work provides a scalable framework that advancing the field beyond narrow, domain-specific reasoning enhancements.\n    \n2. **GRPO already employs model-based reward systems**\n    \n    While model-based reward systems aren't entirely novel, our work specifically focuses on developing a lightweight, generative model-based verifier that enables scaling across diverse domains.\n    \n3. **The sharp performance rise at step 120 may be coincidental.**\n    \n    We thank the reviewer for raising this question. To further investigate, we retrained the model-verifier-based General-Reasoner-Qwen3-4B corresponding to Figure 4 for 500 steps, with the results shown below:\n    \n    | Steps | MMLU-PRO |\n    | --- | --- |\n    | 40 | 0.5767 |\n    | 80 | 0.5865 |\n    | 120 | 0.5958 |\n    | 160 | 0.6061 |\n    | 200 | 0.6140 |\n    | 240 | 0.6082 |\n    | 280 | 0.6173 |\n    | 320 | 0.6142 |\n    | 360 | 0.6239 |\n    | 400 | 0.6234 |\n    | 440 | 0.6319 |\n    | 480 | 0.6312 |\n    \n    This demonstrates that the performance improvement with the model-based verifier is consistent and not coincidental. \n    \n4. **Expand baseline comparisons: Include more results for Qwen3-14B baselines**\n    \n    Our main baselines in this work are the ZeroRL approaches such as SimpleRL. However, since Qwen3-14B is a newly released model, no baselines from previous works are available for it. The comparisons with ZeroRL baselines focus on the Qwen2.5 models, while our Qwen3 experiments demonstrate that our data and training framework also bring improvements to these recent stronger models.\n    \n    To further enrich the baseline comparison, we trained two SFT baselines: 1) we directly used WebInstruct-Verified data to conduct SFT with questions and extracted short answers, and 2) we used the original WebInstruct (without processing it as verifiable answer data with open answer form) to conduct SFT. The results are shown below:\n    \n    | Model | Post Training | MMLU-PRO | SuperGPQA |\n    | --- | --- | --- | --- |\n    | Qwen3-4B-Base | - | 0.516 | 0.254 |\n    | Qwen3-4b-webinstruct-verified-sft | SFT | 0.4397 | 0.281 |\n    | Qwen3-4b-webinstruct-sft | SFT | 0.4386 | 0.227 |\n    | General-Reasoner-Qw3-4B | RL | 0.628 | 0.325 |\n    |  |  |  |  |\n    | Qwen3-14B-Base | - | 0.642 | 0.365 |\n    | Qwen3-14b-webinstruct-verified-sft | SFT | 0.5332 | 0.323 |\n    | Qwen3-14b-webinstruct-sft | SFT | 0.4465 | 0.2542 |\n    | General-Reasoner-Qw3-14B | RL | 0.703 | 0.399 |\n    \n    The results demonstrate that SFT training on strong base models actually degrades performance. In contrast, our RL-based training framework consistently delivers significant improvements over the base model.\n    \n5. **Separate Qwen3-14B and Qwen2.5 results**\n    \n    We appreciate your suggestion and will separate the Qwen3 and Qwen2.5 results for the 14B models into different groups to enhance clarity in the presentation.\n    \n6. **The answer-pair verification approach is limited to QA-style tasks with short answers, failing to generalize to broader tasks like summarization, writing, or translation.**\n    \n    This work focuses on improving LLM reasoning capabilities specifically within the scope of reinforcement learning with verifiable reward (RLVR) for reasoning-intensive tasks. Tasks like summarization, writing, and translation are not reasoning-intensive and fall outside the scope of our research. We will expand this discussion in the limitations section.\n    \n7. **typo in OpenReview abstract**\n    \n    We thank the reviewer for their careful reading. We will fix the typo. \n    \n\nWe hope our response has addressed your main questions and concerns. We welcome further discussion during the reviewer-author discussion period."}]

Author rebuttal:

Meta-review: This paper presents a pipeline for constructing a large-scale reasoning dataset from web-crawled data. Using this pipeline, the authors curate 230K high-quality reasoning questions spanning a wide range of reasoning domains. The dataset is then used to train a domain-agnostic model-based verifier for evaluating answers. Within a Zero-RL training framework, the dataset and verifier yield reasoning models with improved accuracy across multiple domains.

While the method itself is not highly novel, the data curation pipeline is sound, the paper is well written, and the empirical results are comprehensive. The proposed dataset and verifier should prove valuable for researchers and practitioners working on reasoning problems.

Given the quality of the dataset, the strength of the results, and the potential impact on the field, AC recommends acceptance.
Decision: Accept (poster)