Reinforcing General Reasoning without Verifiers
Authors: name: Xiangxin Zhou; name: Zichen Liu; name: Anya Sims; name: Haonan Wang; name: Tianyu Pang; name: Chongxuan Li; name: Liang Wang; name: Min Lin; name: Chao Du
Published: 2025-05-27 17:56:27

Abstract: The recent paradigm shift towards training large language models (LLMs) using
DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
led to impressive advancements in code and mathematical reasoning. However,
this methodology is limited to tasks where rule-based answer verification is
possible and does not naturally extend to real-world domains such as chemistry,
healthcare, engineering, law, biology, business, and economics. Current
practical workarounds use an additional LLM as a model-based verifier; however,
this introduces issues such as reliance on a strong verifier LLM,
susceptibility to reward hacking, and the practical burden of maintaining the
verifier model in memory during training. To address this and extend
DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
verifier-free method (VeriFree) that bypasses answer verification and instead
uses RL to directly maximize the probability of generating the reference
answer. We compare VeriFree with verifier-based methods and demonstrate that,
in addition to its significant practical benefits and reduced compute
requirements, VeriFree matches and even surpasses verifier-based methods on
extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
benchmarks. Moreover, we provide insights into this method from multiple
perspectives: as an elegant integration of training both the policy and
implicit verifier in a unified model, and as a variational optimization
approach. Code is available at https://github.com/sail-sg/VeriFree.