RL Tango: Reinforcing Generator and Verifier Together for Language
  Reasoning
Authors: name: Kaiwen Zha; name: Zhengqi Gao; name: Maohao Shen; name: Zhang-Wei Hong; name: Duane S. Boning; name: Dina Katabi
Published: 2025-05-21 02:43:15

Abstract: Reinforcement learning (RL) has recently emerged as a compelling approach for
enhancing the reasoning capabilities of large language models (LLMs), where an
LLM generator serves as a policy guided by a verifier (reward model). However,
current RL post-training methods for LLMs typically use verifiers that are
fixed (rule-based or frozen pretrained) or trained discriminatively via
supervised fine-tuning (SFT). Such designs are susceptible to reward hacking
and generalize poorly beyond their training distributions. To overcome these
limitations, we propose Tango, a novel framework that uses RL to concurrently
train both an LLM generator and a verifier in an interleaved manner. A central
innovation of Tango is its generative, process-level LLM verifier, which is
trained via RL and co-evolves with the generator. Importantly, the verifier is
trained solely based on outcome-level verification correctness rewards without
requiring explicit process-level annotations. This generative RL-trained
verifier exhibits improved robustness and superior generalization compared to
deterministic or SFT-trained verifiers, fostering effective mutual
reinforcement with the generator. Extensive experiments demonstrate that both
components of Tango achieve state-of-the-art results among 7B/8B-scale models:
the generator attains best-in-class performance across five competition-level
math benchmarks and four challenging out-of-domain reasoning tasks, while the
verifier leads on the ProcessBench dataset. Remarkably, both components exhibit
particularly substantial improvements on the most difficult mathematical
reasoning problems. Code is at: https://github.com/kaiwenzha/rl-tango.