Title: General-Reasoner: Advancing LLM Reasoning Across All Domains
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

reasoning

reinforcement learning
Abstract: Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs).
Particularly, the "Zero" reinforcement learning introduced by Deepseek-R1-Zero, enables direct RL training of base LLMs without relying on an intermediate supervised fine-tuning stage.
Despite these advancements, current works for LLM reasoning mainly focus on mathematical and coding domains, largely due to data abundance and the ease of answer verification.
This limits the applicability and generalization of such models to broader domains, where questions often have diverse answer representations, and data is more scarce.
In this paper, we propose General-Reasoner, a novel training framework designed to enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing a generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of-thought and context-awareness. We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 benchmarks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that General-Reasoner outperforms existing baseline methods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces General-Reasoner, a novel training paradigm designed to enhance the reasoning capabilities of large language models (LLMs) across diverse domains. It addresses the limitations of existing methods, which primarily focus on mathematical and coding tasks. The authors construct WebInstruct-verified, a large-scale, high-quality dataset spanning multiple disciplines, and develop General-Verifier, a generative model-based verifier that replaces rigid rule-based verification. This approach enables robust reward signals for reinforcement learning (RL).\nComprehensive evaluations demonstrate that General-Reasoner outperforms baseline methods on general reasoning benchmarks such as MMLU-Pro and SuperGPQA, while maintaining strong performance in mathematical tasks. The work contributes a scalable framework for improving LLM reasoning across a broad range of domains.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper introduces a novel model-based verifier that significantly improves answer verification across diverse domains, addressing a key limitation of rule-based methods.\n2. The construction of a large-scale, high-quality dataset (WebInstruct-verified) enhances the scope of reasoning tasks beyond mathematics, contributing valuable resources for future research.\n3. Comprehensive evaluations across multiple benchmarks demonstrate the model's robust performance and generalization capabilities, particularly in non-mathematical domains.\n\n## Weaknesses\n\n1. The use of a model-based verifier (or reward model in RL) to replace rule-based verification is not novel, as evidenced by prior works like [1][2][3], and has been applied even in math-focused tasks.\n2. GRPO already employs model-based reward systems (as shown in [4], Table 10), making the proposed modification less innovative.\n3. Table 4 shows only modest improvements on MMLU and GPQA when using diverse data versus math-only data, and the comparisons with Qwen3-14B lack strong baselines, raising doubts about the method's impact.\n4. The answer-pair verification approach is limited to QA-style tasks with short answers, failing to generalize to broader tasks like summarization, writing, or translation.\n\n[1] VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models\n\n[2] xVerify: Efficient Answer Verifier for Reasoning Model Evaluations\n\n[3] Compassjudger-1: All-in-one judge model helps model evaluation and evolution\n\n[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\nQUESTIONS:\n1. Clarify the innovation of the General-Verifier: Specifically, how does it differ from existing verifiers in terms of architecture, training, or performance?\n2. Extend training steps in Figure 4: The sharp performance rise at step 120 may be coincidental. Could you train for more steps (e.g., 200+) or until convergence to validate robustness?\n3. Separate Qwen3-14B and Qwen2.5 results: Grouping them together may obscure performance differences. Please disaggregate the results for fair comparison.\n4. Expand baseline comparisons: Include more results for Qwen3-14B baselines to better contextualize General-Reasoner’s improvements.\n\ntypos: There is a typo in the OpenReview abstract where '\\model' is not rendered correctly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to extend RL training of LLMs beyond verifiable domains like math and coding where rule-based rewards can be applied. \n\nTo do this, they first constructed a general-domain training set by filtering WebInstruct-verified, and then trained a small generative reward model as the answer verifier for the general-domain QA task so that computing the reward can be inefficient. The small generative reward model is based on Qwen2.5-Math-1.5B and distilled from Gemini-2.0. They then trained the model with standard GRPO (without any warmstart with SFT). \n\nThe trained models (named General-Reasoner) gets decent improvement as compared to the Base models on MMLU-Pro, GPQA, SuperGPQA, and BBEH in general. And in fact, they get improvement on math benchmarks too probably due to the learned reasoning behaviour.\n\nSTRENGTHS AND WEAKNESSES:\nWeakness:\n\n- I think at this point we all agree that Qwen models are outliers in the sense that their mid-training probably made them extremely suitable for RL. So I do think it's a fair ask to request additional experiments on non-Qwen checkpoints to verifiy the generalizability of the proposed method. \n\n- You might want to justify the novelty of the work a bit more: For example, in \"Demystifying Long Chain-of-Thought Reasoning in LLMs\", there are also RL experiments on general-domain QA data (Section 5) and they also used WebInstruct data. What's the fundamental difference apart from your own small verifier model? \n\nStrength:\n\n- Thorough experiments and a trendy topic \n- Good empirical results\n\n---------------------------------------------------------------\nUpdate on 7/4: I was prompted by the AC to add more reasons for rejection, so I'm adding more thoughts here. Quite frankly, I think the paper is offering some interesting empirical results for the RLVR community, and the execution is generally ok; it just doesn't quite hit the scientific standard I was hoping for in a NeurIPS paper. I will share some more thoughts/questions that will hopefully be helpful: \n\nThe paper has two main contributions: (1) curating a general-domain training set for doing RLVR beyond math tasks; and (2) training a small model-based answer verifier as the reward model in RL, rather than using rule-based rewards. \n\nFor (1): \n\n- In your filtering step, you mentioned you only kept examples where you can find human-written answers in the original websites. How exactly was this step implemented? Did you use an LLM filter to check the original websites?  \n\n- Quite frankly, a significant portion of the dataset is still math problems (34%). \n\n- How do you justify the quality of the final dataset? And how do you justify its necessity? Do we really need RL data on general-domain questions? Because if you look at the paper \"Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning\", in their Figure 1, even if you RL on math data, you still get improvement on non-math tasks. If you wanna claim this new training dataset is a contribution, shouldn't you do some data ablation to justify its importance? Does training on your filtered WebInstruct data offer additional gains beyond simply training on existing math data? \n\nFor (2):\n\n- I like Table 5 in the sense that it's clearly telling me what's the advange of model-based verifier vs rule-based verifier. It'd be nice if there could be some results of using an \"oracle\" reward too. For eg, by using Gemini as the verifier there. In a way, I really like the various ablations being done in the paper \"Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning\". They showed clearly the gains and potential reward hacking of using model-based verifiers. Do you think your work is offering additional new insights beyond their results? (Apart from experimenting it on different datasets?) \n\nIn general, I would happily raise my score if you could tell me concretely the list of scientific insights that your paper is offering beyond what's already known by previous RLVR papers. For to give you some few-shot examples, here are some examples extracted from the papers I mentioned above:\n\n- RL training on math can get transferable improvement on other non-math benchmarks or even non-reasoning benchmarks (from \"Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning\") \n- Using model-based verifier in RL training can be better than rule-based verifiers, but in some cases, there is reward-hacking (from \"Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning\". \n- You can do RL on general-domain data (WebInstruct) with either model-based verifier or rule-based verifier, ideally on a filtered set of prompts with short answers, and it can get improvement on general benchmarks like MMLU-Pro (literally Takeaway 5.2 from \"Demystifying Long Chain-of-Thought Reasoning in LLMs\"). \n\nIn general, I was hoping for a scientific paper beyond \"we gathered a new training dataset by filtering an existing dataset, and finetuned a new model with existing training algorithms, and it works slightly better on some benchmarks\". Or at least this shouldn't be the narrative of the paper IMHO. Happy to discuss more during rebuttal!\n\nQUESTIONS:\n- What's the SFT data for \"Qwen3-4B-Instruct\" in the second block of Table 2? Is that just one of the official relased checkpoints from Qwen?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGeneral-Reasoner introduces a zero-RL fine-tuning framework that leverages a newly curated, 230 K-question, multi-domain reasoning dataset (“WebInstruct-verified”) and a 1.5 B-param generative verifier to provide reward signals across diverse answer formats; by directly applying policy optimization to base LLMs (Qwen2.5-7B/14B and Qwen3-4B/14B), it achieves consistent ~10 perc gains over baselines on 12 STEM and humanities benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\nThe data collection effort could be valuable to the RL community. It contributes to the endeavor of upscaling verifiable rewards \n\n## Weakness\nThis work is a solid engineering effort that scales verification-based RL across broader question types. But its core contributions are largely in data collation, not in new RL algorithms or understanding model reasoning. This work definitely going to make a solid dataset track paper if it were submitted to that track. \nThey don’t isolate the RL fine-tuning from the dataset or verifier. Without showing how much of the performance is from increased data or from RL dynamics, it's hard to credit RL for true reasoning gains.\nAmong all baselines, the main complaint is that there is no standalone “Distill” comparison: Across all reported results (e.g., Tables 6–8), the paper only evaluates un fine-tuned “Base” models, supervised-fine-tuned “Instruct” models, and zero-RL variants (SimpleRL, General-Reasoner). There is no pure knowledge-distillation / direct SFT (and other no-RL) baseline to isolate the effect of the RL procedure itself.\n\nQUESTIONS:\n1. Did the authors compare the verifier against Gemini generated answers?\n2. What if the verifier is trained only with open LLMs? All curation/verifier training described uses Gemini, no ablation with only open-source models.\n3. Is generative verifier better than scalar RLVR?\n4. Over-reliance on a single verifier model backbone: are there any ablations on alternative verifiers?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes General-Reasoner, a new Zero-RL training paradigm for improving the reasoning capabilities of LLMs across diverse domains. To achieve this, a large-scale, diverse-domain dataset with high-quality reasoning questions from web resources is constructed. Additionally, a generative verifier model is trained for evaluating answers, providing robust and reliable reward signals for RL training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n-\tThis paper tackles an important gap in LLM reasoning that most prior work has primarily focused on mathematical and coding domains due to the abundance of data and the ease of answer verification. \n-\tThe curated dataset, comprising 230K high-quality reasoning questions across various domains (e.g., physics, chemistry, finance), is a significant contribution and will likely benefit future research in general-domain reasoning.\n-\tExperiments are comprehensive, involving a wide range models, such as Qwen2.5 and Qwen3 family, RL models, reasoning models, and multiple evaluation benchmarks (e.g., MMLU-Pro, SuperGPQA). General-Reasoner outperforms baselines on most datasets. Additionally, ablation studies effectively analyze the impact of data abundance and model-based verifier.\n-\tThe paper is well-written and easy to follow.\n\n**Weaknesses**\n\n-\tThe technical novelty is relative limited, but the main contributions lie in the curated dataset, the new Zero-RL training paradigm with a compact model-based verifier, and thorough experiments.\n-\tThe data collection process heavily relies on Gemini models. Specifically, Gemini-1.5-Pro is used to extract questions, Gemini-2.0-Flash is applied to annotate each question, and solutions are generated by Gemini-2.0-flash. The potential issues introduced by these models, such as factual errors or biases, are not addressed. There is a lack of quality control in the collected data. It would benefit from including human evaluation, even on a small subset, to assess data quality.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **General-Reasoner**, a reinforcement learning framework for improving the general reasoning abilities of language models across diverse domains—ranging from math and science to social science and finance. Unlike standard RLHF pipelines, the method performs reinforcement learning **without any supervised fine-tuning**, a setting the authors call “Zero-RL.”\n\nThe paper introduces two main components:\n\n1. **WebInstruct-Verified Dataset**: A large-scale, high-quality, and verifiable multi-domain QA dataset collected and filtered using LLMs (e.g., Gemini) to ensure answerability and consistency.\n2. **Generative Verifier**: A lightweight, CoT-based model trained to semantically evaluate the correctness of answers. It serves as a reward model during RL training via GRPO (Group Relative Policy Optimization).\n\nThe method is tested on 12 benchmarks, including MMLU-Pro, GPQA, SuperGPQA, and various math datasets. It achieves state-of-the-art or highly competitive performance across the board—even with smaller models (e.g., Qwen3-4B)—without using any instruction-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- **Conceptual clarity and practical significance**: The method aims to train a general-purpose reasoning agent without SFT, a meaningful contribution given the costs and biases introduced by supervised pipelines.\n\n- **Innovative dataset and reward design**: The use of LLM-filtered, verifiable multi-domain questions and a generative verifier offers a new approach to constructing diverse and semantically grounded reward signals.\n\n- **Strong empirical performance**: General-Reasoner consistently outperforms base models and strong RL baselines across diverse tasks and domains, even when using small-scale models like 4B.\n\n- **Readable and well-structured**: The paper is clearly written, with logically organized sections, clear diagrams, and helpful qualitative examples.\n\n---\n\n## Weaknesses\n\n- **Related work on Zero-RL is not acknowledged**: Recent work such as **DeepSeek Zero** also demonstrates RL training without SFT. While General-Reasoner offers distinct innovations (e.g., the generative verifier and diverse domain coverage), the paper does not acknowledge or contrast itself with these methods, leaving its novelty under-specified.\n\n- **Unclear why Zero-RL works at small scale**: The paper shows that 4B models trained with Zero-RL achieve strong results, but does not investigate *why*. Are the gains due to data quality, the verifier, the GRPO algorithm, or something else? No ablations or training curve analysis are provided to support this important finding.\n\n- **Potential risk of reward hacking is not addressed**: Model-based reward signals are known to be vulnerable to “hacking” (i.e., models learning to game the verifier without improving actual reasoning), as highlighted in recent work (e.g., arXiv:2505.22203). The paper does not evaluate whether this risk affects its results, nor does it include human or external evaluation to verify reasoning quality.\n\n- **Verifier correctness is assumed but not deeply evaluated**: The paper claims that the generative verifier is more flexible than rule-based ones, which is likely true. However, it does not assess whether the verifier enforces domain-specific norms—e.g., in a chemistry case where it marks “26.32” correct instead of “26.3,” violating significant figure rules. This could lead to inaccurate training feedback.\n\n- **WebInstruct-Verified dataset lacks transparency**: While the data curation pipeline uses multi-answer voting and LLM filters, the thresholds for acceptance, coverage across domains, and error rates are not reported. No human validation is used to ensure that retained samples are factually or pedagogically sound.\n\n- **No statistical significance or variance reporting**: The paper reports average accuracies but omits standard deviations or confidence intervals. Given that generalization is known to be sensitive to seeds and sampling, this limits trust in the results.\n\nQUESTIONS:\n1. Why does Zero-RL work so well at 4B scale in your setting? Could you provide analysis (e.g., with/without verifier, or smaller RL steps) to explain?\n\n2. Have you tested whether the model is overfitting to the generative verifier's patterns? For example, can it fool the verifier with shallow patterns?\n\n3. How robust is the generative verifier to answer formatting, over-precision (e.g., significant figures), or semantic ambiguity?\n\n4. How would you position this work in relation to DeepSeek Zero or other recent Zero-RL efforts?\n\nLIMITATIONS:\nNo limitation section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}