RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
Authors: Kaiwen Zha
Zhengqi Gao
Maohao Shen
Zhang-Wei Hong
Duane S Boning
Dina Katabi
Venue: NeurIPS 2025 poster ()
Keywords: LLM reasoning
RL post-training
Process reward models
Primary area: deep_learning

Abstract: Reinforcement learning (RL) has recently emerged as a compelling approach for enhancing the reasoning capabilities of large language models (LLMs), where an LLM generator serves as a policy guided by a verifier (reward model). However, current RL post-training methods for LLMs typically use verifiers that are fixed (rule-based or frozen pretrained) or trained discriminatively via supervised fine-tuning (SFT). Such designs are susceptible to reward hacking and generalize poorly beyond their training distributions. To overcome these limitations, we propose Tango, a novel framework that uses RL to concurrently train both an LLM generator and a verifier in an interleaved manner. A central innovation of Tango is its generative, process-level LLM verifier, which is trained via RL and co-evolves with the generator. Importantly, the verifier is trained solely based on outcome-level verification correctness rewards without requiring explicit process-level annotations. This generative RL-trained verifier exhibits improved robustness and superior generalization compared to deterministic or SFT-trained verifiers, fostering effective mutual reinforcement with the generator.  Extensive experiments demonstrate that both components of Tango achieve state-of-the-art results among 7B/8B-scale models: the generator attains best-in-class performance across five competition-level math benchmarks and four challenging out-of-domain reasoning tasks, while the verifier leads on the ProcessBench dataset. Remarkably, both components exhibit particularly substantial improvements on the most difficult mathematical reasoning problems.

RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
Kaiwen Zha 1.
Zhengqi Ga
Maohao Shen'
Zhang-Wi Wei Hong?
Duane S. Boning'
Dina Katabi"
MIT
MIT-IBM Watson Al Lab
Abstract
Reinforcement learning (RL) has recently emerged compelling approach for enhancing the reasoning capabilities of large language models (LLMs), where an LLM generator serves a policy guided by verifier (reward model). However, current RI post-training methods for L.MSs typically use verifiers that are fixed (rule-based or frozen pretrained) trained discriminatively via supervised fine- tuning (SFT). Such designs are susceptible to reward hacking and generalize poorly beyond their training distributions. To overcome these limitations, TANGO, a novel framework that uses RL to concurrently train both an LLM gener- ator and a verifier in interleaved manner. central innovation of TANGO is its generative, process-leve LLM verifier, which is trained via RL and co-evolves with the generator. Importantly, the verifier is trained solely based on netee ver- ification correctness rewards without requiring explicit process-level annotations. This generative RI.-trained verifier exhibits improved robustness and superior gener- alization compared to deterministic or SFT-trained verifiers, fostering effective mu- taal reinforcement with the generator, Extensive experiments demonstrate that both components of TANGO achieve state-of-the-arr results among 7B/8B-scale mod- els: the generator attains best-in-class performance across five competition- math benchmarks and four challenging out-of-domain reasoning tasks, while the verifier leads on the ProcessBench dataset. Remarkably, both components exhibit particularly substantial improvements on the most difficult mathematical reasoning problems. Code at: https:/ https://giiiinn //gittbb....///////// con/ ka wwwwwezaaaaaeeaaaaaaaaaaaaaaaaaa www.weewwwaweeeeeeeeeeeeee rl-tango.
propose
85
Verifier Warmup
Reinforcing Generator & Verifier Together (Tango)
Accuracy or F1 Score (%)
9.1%
75
3.3x efficient
65
Generator Accuracy (GRPO)
Generator Accuracy (GRPO w/ Tango)
Verifier F1 Score (GRPO w/ Tango)
55
50
100
150
200
250
300
Global Steps
Figure 1: Generator and verifier training dynamics of TANGO. The generator and verifier co-evolve through mutual reinforcement. Supported by the verifier, the generator of TANGO achieves significantly better training efficiency and stronger final performance compared vanilla GRPO. The generator accuracy is the pass@ accuracy on the MATH 500 dataset, and the outcome Fl score the verifier is reported on training data.
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) have recently demonstrated remarkable capabilities across broad spectrum of natural language processing (NLP) tasks [22, 5. 16]. Despite their impressive perfor- mance, pretrained L.MSs often struggle with complex reasoning tasks requiring multi-step thinking and planning [23, 19]. To enhance reasoning abilities, LLMS typically undergo post-training via supervised fine-tuning (SFT) [61, 10] or reinforcement learning (RL) [23, 19]. SFT teaches models training lacks interaction and generalizes poorly to unfamiliar reasoning paths [9]. In contrast, RL frames learning as an active exploration process, where models learn from experience and directly optimize for task success through trial and feedback, enabling stronger generalization [9]. Therefore, RL has become a central component of recent LLM post-training, with large-scale industrial deployments such as OpenAl's ol [23] and DeepSeck RI [19] demonstrating its effectiveness in unlocking advanced reasoning capabilities.
to mimic curated solutions, but this imitation-
In LLM post-training with RI., the LLM generator acts as the policy model, where each action corresponds to generating the next token based on the current sequence (the state). A reward model, commonly known as a verifier, assesses the quality of the generated outputs and provides feedback that is used to guide the generator's 's policy updates using RL algorithms [41, 40, 43, I].
However, a critical limitation of current RL post-training approaches [43, 50, 60] is their reliance on a fixed verifier, typically implemented using bases-based metrics or a frozen pre-trained reward model. This fixed verifier limits the potential improvement of the generator and is vulnerable to reward hacking in distribution changes [14]. Ideally, verifiers should be trained jointly with generators [30], enabling mutual improvement. Yet, designing an effective co-evolving system remains challenging. Among recent attempts, PRIME [12] is, to the best of our knowledge, the only approach that trains the generator alongside the verifier. However, PRIME's s verifier still faces critical shortcomings. First, it employs a discriminative logit-based process reward model (PRM) that generates deterministic reward signals, making susceptible to reward hacking [13], despite being trained online. Second, PRM is trained using SFT with outcome-level labels, despite these labels being collected in an online manner. SFT significantly restricts verifier reasoning capabilities and generalization potential [9].
We argue that the effectiveness of co-evolving system critically relies on the capabilities of both the generator and the verifier. If one component significantly weaker and lags behind, it can impede the overall learning dynamics and limit mutual improvement. An effective co-evolutionary framework requires both agents be robust and to continuously enhance each other's performance. To this end, we introduce TANGO, a novel framework that jointly trains an LLM generator and an LLM verifier in an interleaved manner via RL. Unlike existing methods that use frozen, discriminative, or SFT-trained reward models [43, 60, 12], TANGO introduces a process-level, generative LLM verifier that is trained via RL and co-evolves alongside the generator throughout training. Specifically, the generator produces multi-step reasoning trajectories, while the verifier offers natural language feedback comprising both step-level assessments and an overall correctness judgment. The generator leverages gold outcome-level correctness signals combined with detailed step-level rewards from the verifier, improving the efficiency of policy learning [48], and guiding the generator toward more robust reasoning strategies [30]. Importantly, the verifier is trained exclusively using outcome-level verification correctness rewards, without process-level annotations. Through RL, it progressively refines its chain-of of-thhught [55] verification reasoning, gradually aligning its step-level feedback with final correctness outcomes as the generator's 's reasoning trajectories evolve.
TANGO offers a more effective design for the co-evolving generator-verifier system, addressing the limitations of previous approaches. First, by training the verifier using RL rather than SFT, the verifier develops stronger reasoning skills and generalizes better beyond supervised imitation. This mirrors the rationale behind preferring RL over SFT when training generators under outcome-only supervision. Second, the generative and sampling-bies based nature of TANGO's verifier introduces stochasticity into the reward signals, enhancing its robustness against reward hacking. Consequently, through interleaved training, the generator and verifier mutually reinforce each other, enabling improved reasoning strategies and superior generalization performance, as shown in Figure 1.
We conduct extensive experiments to evaluate the effectiveness of TANGO across diverse reasoning tasks and experimental settings. Compared to vanilla RL methods trained only on outcome-level rewards, TANGO achieves an average relative improvement of 25.5% on five competition-level math benchmarks and 7.3% on four challenging out-of-domain reasoning tasks, consistently across three RL algorithms. Remarkably, TANGO with GRPO doubles the accuracy on the most challenging

---

benchmark, AIME 2025, relative to vanilla GRPO. Furthermore, TANGO substantially outperforms baselines, including PRIME. In a comprehensive comparison with prior
ORM- and state-of-the-art L.M reasoning methods, TANGO establishes new state-of-the-ar results among 7B/8B-scale models, delivering the best performance on the most difficult tasks, namely AIME 2025, AIME 2024, and AMC 2023. TANGO verifier also sets new sate--o-theeaie on ProcessBench [64], despite not using process-level annotations. In particular, it achieves the highest step-level verification performance on the most challenging subsets, OlympiadBench and Omni-MATH, significantly surpassing previous methods, including the much larger Qwen2.5-Mathh Baatt-72B-Instruue model even though our verifier is initialized from just Qwen2.5-7B base model. Finally, an in-depth analysis on an algorithmic reasoning task with available gold step-level labels confirms that TANGO effectively bootstraps both the generator and verifier into highly capable states through mutual reinforcement.
2 Related Work
RL for LLM reasoning. RL was initially used to align LLM outputs with human preferences, enhancing response quality, instruction ceatuction-foolowingg following, and style [8. 47, 67, 39, 25]. As LLMs expanded into domains demanding multi-step reasoning and structured problem-solving such as mathematics, coding [24, 2], and web navigation [66, 291, RL has evolved beyond basic alignment toward enhancing LLM reasoning abilities [23, 19, 43, 27, 12, 60, 44, 37, 52, 7, 53.. A pivotal milestone was OpenAl's ol [37, 23], which demonstrated RL's effectiveness scale. Subsequent research has further advanced RL-based LLM reasoning through improved optimization algorithms [41, 40, 43, 1J, curriculum- based training [46, 60], and enriched evaluation benchmarks [11, 64, 49].
Reward modeling. Reward signals critical for guiding LLM post-training toward desirable behaviors and enabling effective inference-tine scaling. Reward models (RMs a.k.a. verifiers) categorized as outcome reward models (ORMs) process reward models (PRMs) based on the granularity of evaluation. ORMs [39, 19, 35] assign a single scalar reward to the final token in response trajectory, resulting in sparse supervision. Typically discriminative, ORMs attach classification head to pretrained LLMs and are trained via SFT on labeled response pairs (preferred vs. rejected) [67, 47, 12, 32]. Recently, generative ORMs have emerged [62, 33], producing explicit rationales before assigning outcome scores.
In contrast, PRMs [12. 7, 31, 53, 32, 54] provide fine-grained, step-level feedback throughout the generation trajectory, facilitating more precise credit assignment [30] and improved training efficiency [48]. This detailed feedback helps models efficiently explore policy spaces and develop stronger reasoning capabilities. However, most existing PRMs remain discriminative, frozen, and deterministic, rendering them brittle to distribution shifts and reward hacking [42], while also requiring expensive step-level annotations. PRIME [12] partially addresses these issues by jointly training PRM via SFT alongside the generator, reducing annotation overhead. Yet, PRIME's logit-based, deterministic rewards still leave it vulnerable to hacking, and its SFT-based training constrains generalization. To address these limitations, we propose a generative, process-level verifier (generative PRM) that outputs stochastic, step-level rewards as textual judgments (e.g.. "Correct" or "Incorrect"). Unlike existing PRMs, both logits-based and generative, that rely exclusively on SFT. ours is the first PRM trained using RL. We note that the idea of generative PRMs traces back to LLM-as-a-judge [65], which uses frozen LLMs for scoring. More recent works [28, 63] have explored SFT-based generative PRMs for inference time scaling, but these concurrent approaches remain orthogonal to our work, as none involve RL-trained, co-evolving generative PRMs.
3 Method
3.1 Preliminaries
We denote the autoregressive LLM generator and verifier as ny and π.. respectively. For notational simplicity, we use mo as a unified symbol when an equation applies to both models. RL aims to optimize policy model by maximizing the expected cumulative discounted reward through interactions with the environment, i.e., by taking actions and transitioning between states. In the context of RL post-training for LLMs, the policy model corresponds to the LLM generator or verifier. The state at step i defined as the combination of the input prompt (i.e., question) x and the partially generated response O while the action corresponds to generating the next token or. To
3

---

Multi-Stop Solution
Gold
RL Update
RL Update
"atep-Dap Claar the denom nator by sipiing dden be
Scheetin
2
Outcome Rewards
Verifoation Mull:
Sap Dap de Se the ame st 1-Vitape
*apeprdapp Contér
lettor
vilagg
LLM Generator
LLM Verifler
seluition L-hiip
Outcome Rewards
Answer
caroecty CORREcT-iaag
estepok tarera noved to the vihou eror cORRECT-
exspoCkeemrii cireiiiction factoring are correct candideee
CORRECT
Question
Step Rewards
Verification
"zhip-The soldice balls equatinn the only va idrool p-7. INC INCORRECTraaap <fral Jerf WNO--RECOOOOOOOO
makes original natter INCOMRECTHibage
Figure 2: Overview of the TANGO framework with a generation and verification example. Given question, the LLM generator produces multi-step solution. which is then evaluated by the LLM verifier. The generator trained using both step-level rewards from verifier and outcome- rewards based its final answer, while the verifier trained only with outcome-level rewards based on the correctness its final judgment and format.
optimize ne, policy gradient methods are employed to estimate the gradient of the expected reward with respect to the policy parameters 0. Built upon it, widely used surrogate objective [41, 40, 43, 1] formulated using importance sampling:
[(0) = E (x oonta (+)x))
min (())), clip (cc 1+0) DxL [mo
|| (
(1)
c0 = To(0i x. o2e)
(o x, 0x
where Taay and The denote the current, old, and reference policy models respectively, |o] the sequence length, A, is an estimator of the advantage step and y is the gold answer used to compute the reward and subsequently the advantage A. The hyperparameter controls the clipping range the importance sampling ratio, while 3 regulates the KL-divergence penalty strength. RL algorithms mainly differ in their methods for estimating A,, such as group- noup-noomllized rewards (GRPO [43]), leave-one-out reward averaging (RLOO [1]), or batch- natch-normalized rewards (REINFORCE++ [21]).
3.2 TANGO
TANGO jointly trains an LLM generator and an LLM verifier via interleaved RL, creating self- reinforcing loop where each agent iteratively strengthens the other. Figure illustrates the over- all TANGO framework. Specifically, we alternate training the generator policy Tg for N steps and the verifier policy Tu for N,, steps, repeating this cycle iteratively. Below, we detail the RL training for each component. Please refer to Appendix C for the detailed algorithm flow of TANGO, and Appendix F for more generation and verification examples.
RL-based LLM generator. Given question- pair (x. y) from the training distribution P. the generator you produces a step-by-step solution 0g ~ Tg.. (. x). Our reward design is as follows:
Rule-based outcome-level rewards: Extract the predicted answer ý from the generated solution Og. and compute an analytical rule-based outcome-level correctness reward:
T ogt (og) = [1, 0. otherwise.
if =
Step-level rewards from the verifier: We prepare a verification prompt using the question x and the generator solution Og. and then sample verification response from the verifier O2 ~ Te(- x. o,). If there are K reasoning steps in the generator response Og, then the response o will also contain K step-wise judgments Vaup. (1,, where k = 1, ..... K, with -1 denoting 'Incorrect" and denoting 'Correct'. Please refer to the right part of Figure for an example (x, y, Og, ). Finally, the step-level rewards computed
(((1)) R Rgy.tep = { " (o,), (..),............... "
I(K) T(K)) 9.stop (o,)
I(k) g (o,))
Mstep. k K
E
(3)
K
K

---

where [k) is the index of the end token in the generator's 's k-th reasoning step (k = ..2.....K We normalize the reward by the number of reasoning steps to remove policy's 's bias toward step length, allowing the generator to adaptively determine an appropriate number of steps based on the problem. Essentially, our approach adopts generative process-level verifier that produces natural language judgments, enabling stochastic sampling-based step-wise evaluations.
We compute advantages separately for outcome-level and step-level rewards, combining them through weighted sum. Using GRPO [43] as an illustrative example (though TANGO is compatible with other RL algorithms [1, 41], as shown in Section 4), the generator policy aau samples group of M responses {o which are evaluated by the verifier to produce verification outputs fo Jo Each data sample (o) o contains K" reasoning and verification steps. Below are the advantages:
Outcome-level advantages: The outcome-level advantage of the i-th response A g.out, is calculated by normalizing the group-level outcome rewards Rg,out frggout [)))
Tg.our (0g) mean(Ry..oo
A g.omt
(4)
std (Rg.
Step-level advantages: For step-level advantages, group-level normalization performed across all step reward elements from all responses within the group, i.c.,
M
R stspp
R UR
U 1=1
I(1) g.sep (o))..... ...., g.step (o²))
I(K+)
(5)
step
i=
To clarify, the set size Ry,stepl calculated as the sum of the normalized rewards of its following steps:
K'. Next, the step advantage each token 4 g.sep, is
[(k] "gstep (o)
mean RR g.step)
A g.step.
(6)
std Mid(Rg.ste)
(M)7()22)
We note that for a given sample (i.e., with fixed index i), the outcome-based advantages A g.out, the same across all tokens indexed by whereas the step-level advantages Ad Agystep. may vary across tokens. The final advantage is derived by blending the outcome advantage Eq. (4) and the step advantage Eq. (6) with hyperparameter E (0, 1):
AS. = (1 A)A..un. aA g.
We highlight two key design choices that are crucial to the success of our generator training:
We apply an exponential decay schedule to a, which essential TANGO's success. Early in training, step-level supervision has stronger influence to encourage exploration of reasoning strategies, As training progresses, we gradually reduce its weight to promote stable convergence and mitigate reward hacking.
Empirically, we find that computing and normalizing the step and outcome advantages separately before combining them yields significantly more stable learning than merging the rewards first and computing a single advantage. This is because advantage normalization depends on the scale and distribution of the underlying rewards. Merging step and outcome rewards before normalization could distort their relative contributions due to scale mismatch, resulting in instability and degraded performance. By normalizing each advantage independently, we preserve their intended effects prior to aggregation.
RL-based LLM verifier. The verifier generates a verification response o,, conditioned on the question x and the generator's solution Og. As the example shown in Figure 2, the verifier's final judgment label Yinal E {0, 1} can be extracted from o,, where yhoal = 1 indicates the verifier considers the generator's 's solution correct, and yhinal 0 indicates incorrect. Given that the correctness of the generator's 's answer is known from Eq. (2), we define the verifier's 's outcome-level reward based on how well its final judgment matches this ground-truth correctness, well its format score:
[1,
if yinal ouu (o))) 00 otherwise.
Tr,our (o..) Fr,coorea (Ov) + + Fo.foraa (O,), To,comeet (o.,)
(8)
5

---

forfom The default value of Te. po.fomat( (o.) is set to 1.0 and is gradually reduced for each unmet formatting criterion, such as the absence of step-wise justifications or discrepancies in step numbering between the verifier output Og and the generator output Og. The hyperparameter E R+ controls the contribution of the format score in the final outcome-level reward. The verifier is trained without any process-level supervision, eliminating the need for step-level annotations. Empirically, we observe that although the verifier is trained solely with outcome-level signals, it progressively learns to produce accurate step-level judgments by refining its chain-of- -ftoughhtttttt verification reasoning through RL training, thereby providing useful guidance to the generator.
A key challenge we observe in training the verifier is class imbalance in early stages of learning. Since the generator initially produces mostly incorrect solutions which leads most r'g owg ( the majority of verifier supervision is biased toward negative labels. If we directly apply the original GRPO advantage calculation (as used for the generator in Eq. (6)), we find that the verifier quickly collapses to always predicting Viiall = O, resulting in a trivial but locally stable solution. This collapse not only harms verifier performance but also provides poor step-level verification reward signals to the generator, degrading overall co-training dynamies. To mitigate this issue, we introduce class-aware reweighting scheme into the verifier's advantage computation. Specifically, we apply based on the correctness of the corresponding generator
sample-specific scaling factor 8+ solution after normalizing the outcome rewards using the group R = (re,ouu (o,,)) ) statistics:
A 8 X Tr,ou mean (Rr,ou))
8+ RR+, if gg (o =
S
(ud(R.,ouu
8- IR+, otherwise.
The coefficients {s+, } are set to be inversely proportional to the square root of the number of samples with correct and incorrect generator outputs, respectively. In practice, we maintain these values per batch using an exponential moving average (EMA) to ensure smooth updates throughout training. To build intuition for Eq. (9), consider the case where most rewards T (o)) = O, which results in 84 > relatively fewer samples with yu (oi) = 1 in the overall objective Eq. (1), while downweighting the influence of the more frequent samples with (g. O.
Under this condition, Eq. (9) effectively amplifies the contribution of the
Remarks. We highlight three key advantages of TANGO. First, our verifier is trained via RL, enjoy- ing stronger reasoning and generalization capabilities without requiring costly step-level annotations. Second, unlike prior logit-based methods, it produces transparent, text-based judgments that reduce step-level noise and introduce sampling stochasticity, mitigating reward hacking. Third, the evolving generator produces increasingly diverse outputs, exposing the verifier to broader reasoning patterns and encouraging it to adapt new verification strategies, which in turn improves the generator.
4 Experiments
Base models. We primarily evaluate our method on mathematical tasks to assess reasoning capabil- ity, and on unseen out-of-domain tasks to measure generalization. The generator uses Qwen2.5-Math 7B [59] for its strong mathematical reasoning. while the verifier uses Qwen2.5-7B [58] due to its larger context window accommodating both questions and generator outputs. Notably, Qwen2.5-7B underperforms on math tasks, making our verifier initially weaker than the generator, unlike prior work relying on stronger verifiers for distillation. Instead, our framework uses mutual reinforcement, enabling both agents to co-evolve from weaker starts, yielding a more scalable and practical solution.
Implementation details. We first perform SFT on the generator using 1 113K math prompts from Eurus-2-SFT-Datt. [12], guiding step-by-step reasoning enclosed in step tags. Responses are produced using Llama-3.1 ----- -I---rtttt [16] with system prompt (see Appendix F). The verifier is initialized directly from the base model without SFT. In the RL stage, we use 455K math question-ansee from Burus-2-RL-DD
pairs [12]. We set N₂ = 3 and N. i.c. the verifier updates once every three generator steps, to compensate for slower generation optimization. The generator is trained for 200 steps by default (300 for Table 2 comparison). To prevent early instability, we warm up the verifier for 40 steps to learn the output formatting and reach a reasonable accuracy before guiding the generator.
Experiments are conducted using veRI. [45], with 5 rollouts per prompt. Both generator and verifier policies are trained using AdamW [34] with a constant learning rate x 10-6, batch size 256, and microbatch size 4. 7 is set to 0.8. The coefficient follows an exponential decay schedule,

---

Table 1: Performance comparison TANGO different vanilla RL algorithms on mathematical and out-of-domain reasoning benchmarks. TANGO consistently yields substantial improvements across all tasks when combined with various RL algorithms. All RL models are trained for 200 generator steps.
Mathematical Reasoning
Out-af-Demmin Reasoning
Model
MATR5500
AIME2124
AIME302
AMC3923
OlympiadBennh
BGQA
CRUXEval
Strategr QA
TableBench
Ang.
TANGO 7B SFT
66.6
3.3
3,3
27.5
28.1
25,8
46.6
64.3
85.9
34,4
52.8
63.1
18.2
57.6
TANGO
98.0
RL,00
74.0
13.3
10.11
$2.5
36.0
57.2
35.0
48.5
87.5
38.6
57.4
RLO0 TANGO
80.8
23.3
16.7
67.5
45.3
46.7
68.9
52.9
98.4
43.0
61.8
REINFORCE++
87.3
39.8
57.0
REINFORCE++ YANGO
81.6
20.0
23.3
63.0
44.6
46.9
61.1
52.0
89.0
44.3
61.6
GRPO
GRPO
ORM
PRIME (PRM)
Tango
MATH S00
A/ME (2024 & 2025)
AMC 2023
OlympiadBench
85
40-
651
45
Accuracy (%)
80
Accuracy (%)
30-
Acouracy (%c
Accuracy (%6
40
20-
55-
75+
35
10
50-
Figure 3: Performance comparison of TANGO with ORM- and PRM-based baselines. TANGO consistently outperforms models guided by ORM or PRM, demonstrating the superiority of our co-evolving framework in boosting generator's performance. For AIME, results for 2024 and 2025 are combined and shown below and above the dashed line respectively. All models are trained for 200 generator steps. We reproduce and evaluate PRIME at 200 steps, achieving better performance than the 240-step results reported in its original paper [12].
starting at 0.1 for GRPO and RLOO, and 0.5 for REINFORCE++, to balance step- and outcome-level advantages with an emphasis on step-level guidance early in training. These relatively high initial values of encourage step rewards guide early exploration more, while gradually decaying to 1x10-1 to reduce reward hacking risks in later training. All baselines share identical generator SFT and RL configurations for fair comparison, with GRPO used as the default RL algorithm unless otherwise specified. Please refer to Appendix D for more implementation details.
Benchmark and evaluation. We primarily evaluate the generator on five competition-lever math benchmarks: AIME 2025 [38], AIME 2024 [3], AMC 2023 [4], MATH-500 [32], and Olympiad- Bench [20]. Following [44], we further assess general reasoning and generalization capabilities using four out-of-domain benchmarks: logical reasoning (BoardgameQA, i.e., BGQA [26]). code reasoning (CRUXEval [17]), commonsense reasoning (Strategy [15]), and tabular reasoning (TableBench [56]). All models, including baselines, are evaluated via greedy decoding, reporting zero-shot pass @1 accuracy, i.e., the percentage of problems correctly solved on the first attempt. Additionally, we evaluate our verifier's ' step-level verification accuracy on ProcessBench [64]. which contains annotated reasoning errors for competition math problems.
4.1 Main Results
Comparison with vanilla RL post-training methods. We first evaluate TANGO on standard RL al- G
gorithms commonly used for L.M post-training- GRPO [43], RLOO [1], and REINFORCE++ [21] comparing each against its vanilla counterpart, which employs rule-based outcome rewards. The generator's performance after SFT is also included for reference. As shown in Table 1, integrat- ing TANGO consistently yields substantial improvements across all benchmarks, particularly on challenging math competitions. For example, TANGO with GRPO achieves relative gains of 50.4% on AIME 2024, 100.0% on AIME 2025, and 30.0% on AMC 2023, averaging a 24.6% improvement across all math tasks. Furthermore, TANGO with GRPO enhances generalization out-of-domain reasoning tasks, delivering an average relative improvement of 1%. Please refer to Appendix E for more results using the Llama base model.
Similar trends oceur with RLOO and REINFORCE++, often surpassing those seen with GRPO: RLOO achieves relative gains averaging 25.5% math and 7.7% out-of-domain tasks, while
7

---

Table 2: System-level performance comparison with prior methods on mathematical and out-of-domain reasoning benchmarks. TANGO achieves state-of-the-art performance among 7B/8B-scale models across both domains. For math reasoning, results are from the original papers or prior work [18, 44), except PRIME [12), which we reproducc and evaluate, finding it outperforms the best 5te-step results reported in the original paper, and for AIME 2025, which we evaluate for methods. For out of-dom reasoning. results are from [44]. Our TANGO-Qwen-7B is trained for 300 steps. Best performance per task among 7B/8B models is bolded.
Mathomatical Reasninng
Oul-of-Dummin Reasmming
Medel
MATHS00
AIME3024
ALME2025 AMCMG2
OlympladBench Arg.
BGQA
CRUNEval
StrategrQ4
Tablellench Avg.
Frouteer L2Ms
GPT 4o [22]
76.6
47.
43.3
Clande3.5 Sonnet [5]
78,5
16.0
o1-peeviow
85.5
46.6
900
(23)
90.0
95.0
Qgeessaroed mocseningg (large)
L.amn-3 1-708-Innicet [16]
68.0
13.3
42.5
20.4
58.3
59
85.8
34.2
OpenMath2 (aaelllllnnn11111 1-70B [S1]
71.8
13.3
45.0
30.1
68.7
35.1
95.6
46.8
NuminaMath 72B
Qvee2 2oou2.5-Mat-722--meee
[59)
QWQ-33BBBB Provicu [50]
90.6
77.5
61.2
62.5
71.1
85.2
$1.5
09.0
Upen sourced reasentin LLMs (peatl)
I Jamma- I-S8-- [16]
$1.9
3.3
3.3
32.5
15.1
19.3
90.3
18.5
92.2
32.4
$1.4
OpenMale-Hlamee. 1-58 (51)
78.8
Ower2.5 Inseret
52.5
36.0
53.0
Qwen2 5-Math-788-- In-mnnmmet (89)
8366
187
62.5
41.6
429
51.3
28.0
85.3
35.2
50.2
1Ste-Math-28 (15)
78.4
26.7
47.5
47.1
Darss 2 TB PRIME
80.4
26.7
13.3
00.0
43.7
44.8
Ohrr
TANGO-Qum-78
49.5
62.3
obtains gains of 26.4% and 8. 8.1, respectively. These results highlight TANGO's robustness and broad applicability across diverse RI. algorithms and tasks.
REINFORCE+
We further visualize TANGO's 's training dynamics in Figure 1. Notably, our method matches the accuracy of vanilla GRPO after 200 generator steps in only 60 steps, 3.3x improvement in training efficiency (the figure plots global steps to account for both the generator and verifier). At 200 generator steps, TANGO also achieves a 9.1% higher relative accuracy, underscoring significant gains in both training efficiency and reasoning quality.
Comparison with different RM baselines. We compare TANGO with ORM and PRM baselines in Figure 3. For PRM, we select PRIME [12] as it similarly does not require step-level supervision, making it directly comparable to our method. Note that our method uses the same SFT and RL data as PRIME, as well as the same base model. Integrated with GRPO, TANGO substantially outperforms both ORM and PRIME across all benchmarks. We attribute these gains to our co-evolving design, where the generator and verifier mutually reinforce each other through interleaved RL training. Unlike ORM, which provides only sparse, outcome-leve feedback, rewards, guiding the generator toward better reasoning. Compared to PRIME, our RL-trained verifier offers more accurate and robust reasoning. Its generative, sampling- sampling-based verification introduces stochasticity and enables longer chains of thought, resulting in rewards that are more resistant to hacking and better aligned with true correctness, providing stronger and more informative supervision.
verifier delivers detailed, step-level
System-level comparison with prior methods. We further validate TANGO through a compre- hensive system-level comparison against previous methods on mathematical and out-of-domain reasoning benchmarks, as shown in Table 2. Among 7B/SB-scale reasoning LLMs, TANGO achieves state-of-the -hec----thhertttt art performance, averaging 49.5% accuracy math tasks and 62.8% -ut-of-domain tasks. The improvements are especially significant on challenging math competitions, with scores of 26.7% on AIME 2024, 23.3% on AIME 2025, and 70.0% on AMC 2023, surpassing all prior models at similar scales. These gains highlight the effectiveness of our co-evolving training framework, where the generator and verifier mutually reinforce each other through progressive refinement of feedback, enabling deeper exploration and improved reasoning capabilities on complex problems.
4.2 Verifier Results of TANGO
In the previous section, we demonstrated that TANGO delivers a strong generator through co-evolving. interleaved RI. training. Here, we show that the verifier also significantly benefits from this co- evolution, stcadily improves throughout training and ultimately becomes highly effective.
8

---

Table Evaluation results on ProcessBench. The verifier TANGO achieves -lat--ffthe-arr -he--rr performance among 7B/8B-scale models without using any process labels. The metric reported is the Fl score of the respective accuracies on erroncous and correct samples. Best performance per dataset among 7B/SB models is bolded.
Model
GSMBK MATH OlympiadBeuch Omai-MATH
wown-sowredd language models, prompicdt as critic madels
Qwen2,5-3225 5-322--nnstna [58]
65.6
53.1
40.0
38,3
49.3
Liama 1.1- Iostrrct [16]
48.2
46.7
41.0
Qwen2,5-Maan 5-Math--2B---- Insttrdd [59]
65.8
52.1
32.5
31.7
45.5
Llamu-3, 1-8--Innttad [16]
288
1.6
Qwen2 M-t.----M------hh Innssett- [19)
26.8
25.7
14.2
12.7
19.9
Quen2.5-7B-lnn Be702.5-7--mssmmct [58]
29.7
Open-sourced process reward models (PRMs)
Mall--Shuphore Seals-hhhhhhe-PMM--- [54]
47.9
29:5
24.8
23.8
31.5
RI HFloo-PRM PR-n---RMMMMRS Mistral--R [57)
$0.4
33.4
13.8
28.4
RLHFlow-PRRE PRM-
Met----RM---ppsooooo Deepsook- 8B [57]
38.8
33.8
16.9
16.9
26.6
Man7sPRM-7R [12]
43.0
35.1
Skywork-PRM.. PMM-- [36]
70.8
53.6
22.9
21.0
42.1
verifeer
TANGO-Qwen-T2 (vecilier)
53.1
18.2
37.8
36,3
43.9
We first visualize the verifier's final verification Fl score over training steps in Figure 1, observing consistent improvement. Although the absence of gold step-level labels in math training dataset prevents direct tracking of step-wise accuracy, we provide such analysis using a well-designed algorithmic reasoning task with step-level annotations in Section 4.3. There, confirm that RL training enhances both step-level and final verification performance throughout the training.
We further evaluate the step-level verification accuracy of our final verifier on ProcessBench [64], a benchmark featuring competition-leva math problems annotated with step-wise reasoning errors. As shown in Table 3, TANGO's 's verifier achieves state-of-ther of-the-art results among 7B/8B-scale models, despite training without any step-level supervision. It notably excels on the most challenging subsets, OlympiadBench and Omni-MATH, surpassing previous models significantly, even outperforming the much larger Qwen2.5-Math I2s-rutt despite being initiated only from a Qwen2.5-7B base.
These results confirm that our verifier progressively improves both its outcome-level (Figure 1) and step-level verification accuracy (Section 4.3) over the course of co-evolving RL training with the generator. Ultimately, it delivers highly accurate step-level verification even on the most challenging mathematical problems (Table 3).
4.3 Ablation Analysis with Gold Step-Level Information
In this section, we design an algorithmic reasoning task with gold step-level labels to enable a detailed analysis of TANGO and better illustrate the co-evolution dynamics between the generator and the verifier. Specifically, we adopt the last letter concatenation problem introduced in [55]. The prompt is constructed to elicit step-by-step reasoning from the generator, where the n-th step involves extracting the last letter of the nith word (see Appendix F for examples). This setup allows us to automatically generate gold step-level outputs without any additional annotation effort when constructing the training and evaluation datasets, and also enables evaluation of the verifier's step- level judgments. We use Qwen2.5- 1.5B [58] as the base model for both the generator and the verifier. We compare TANGO against three baselines: (i) the vanilla GRPO method without a verifier, (ii) GRPO with TANGO while keeping the generator fixed, and (iii) GRPO with TANGO while keeping the verifier fixed. More detailed experiment setups be found in Appendix D.
TANGO (ours). As shown in Figure 4, when both the generator and verifier are are jointly updated under the TANGO framework, observe consistent and strong improvements. The generator achieves the best accuracy (left), while the verifier steadily improves on both step-level and outcome-level Fl scores (middle and right). This result confirms that although the verifier trained only with outcome-level rewards, it gradually improves its step-level verification accuracy as RL enhances its chain-of- of-thought reasoning. It also demonstrates that the generator and verifier mutually reinforce each other, leading to stronger reasoning capabilities and more accurate verification.
Fixing generator. In this setting, only the verifier is updated. Initially, it learns from the fixed generator's 's output distribution and improves its Fl score. However, since the generator remains static,

---

45
keys
begetiel Tirge peraeer
Lago 2-4)) Tange permaoo
Targe penacaoot
Outcome Score (95)
75
40
UMDO laoe
Score
Accurecy
60)
30
55
25
45
Steps
Stege
Steps
Figure 4: Ablation of generator and verifier training dynamics in the algorithmic reasoning task. Left: generator accuracy generator training steps; Middle: verifier step Fl score verifier training steps; Right: verifier outcome score v.s. verifier training steps. curves are evaluated on unseen test data.
the verifier's progress quickly plateaus, shown in the middle and right panels. This underscores the importance of continuously improving the generator to provide richer and more diverse reasoning traces that better support verifier training.
Fixing verifier. Although the verifier is frozen, its scores (middle and right) shift slightly generator training alters its input distribution during evaluation. On the generator side (left), performance remains flat for the first 20 steps due to inaccurate step-level feedback from the fixed verifier, which misguides learning. As the a schedule gradually shifts focus from misleading step- level to reliable gold outcome-level rewards, the generator starts to improve. However, its final accuracy still lags behind the baseline, highlighting how static and inaccurate verifier feedback can hinder learning, especially early on, when step-level signals are most critical for strategy exploration.
5 Conclusions
present TANGO, a novel unified RL-based framework that jointly trains an LLM generator and generative, process-level verifier using RL in interleaved manner. Unlike existing approaches that rely on frozen or SFT-trained reward models, TANGO is the first to train the verifier via RL and co-evolve it with the generator without requiring any process-level annotations. Extensive experiments show that both the generator and verifier of TANGO, through mutual reinforcement, achieve ttat-oof-the-art performance across multiple challenging reasoning benchmarks.
10

---

References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce-style optimization for learning from human feedback in 1lms. arXiv:2402. 2eti::2402.11444444 14740, 2024.
[2] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges, arXiv::4402.00157,.000 2024.
[3] AI-MO. Aime 2024.
aimo-validation
2024.
[4] AI-MO. Ame 2023.
httes://uugiinffe..c/aattttttt/
aimo vano-vase ietoo-valdatioon amc, 2024.
[5] Anthropic. Claude 3.5 sonnet.
claude-3- sonnett, 2024.
[6] Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 72b cot. https: 72B- CoT, 2024.
/ //huggingface co www..///././.i........... STL-MM///mmiaaaaat..
[7] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. In EMNLP, 2024.
[8] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurlPS, 2017.
[9] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sit memorizes, rl generalizes: comparative study of foundation model post-training. arXivv2501.17161 2025.
[10] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language models. In JMLR, 2024.
[11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.1416 2021.
[12] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv:250..01456 2025.
[13] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Help- ing or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv:2312.09244 2023.
[14] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML, 2023.
[15] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. In Trans. Assoc. Comput. Linguist. MIT Press, 2021.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024.
[17] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: benchmark for code reasoning, understanding and execution. arXiv:2401.33065, 2024.

---

[18] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun. Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv:2501.04519 2025.
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv:2501.222 /2948, 2025.
[20] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. In ACL, 2024.
[21] Jian Hu. +:inforee+:: A simple and efficient approach for aligning large language models. arXiv:2501.0322 2025.
[22] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv:4410211276, 2024.
[23] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. 2.22:2442.166720, 2024.
[24] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv:2310006770, 2023.
[25] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv:2312.14925 2023.
[26] Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. Boardgameqa: A dataset for natural language reasoning with contradic- tory information. In NeurlPS, 2023.
[27] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv:2410.01679, 2024.
[28] Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv:2504. 16828, 2025.
[29] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv:2401.13649 2024.
[30] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv: 181 1.07871, 2018
[31] Yansi Li, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Qiuzhi Liu, Rui Wang. Zhuosheng Zhang, Zhaopeng Tu, Haitao Mi, et al. Dancing with critiques: Enhancing 11m reasoning with stepwise natural language self-critique. arXiv:2503. 2025.
[32] Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2023.
[33] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. SXXi:25504.02495, 2025.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
12

---

[35] Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang. Shuaibin Li, Qian Zhao, Haian Huang. Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathematical reasoning. WwXiv:2502.0678.. 2025.
[36] Skywork ol Team. Skywork-ol open series. htttss://hugginnfaaeee co skywkrkk,, November 2024.
[37] OpenAl. Learning to reason with Ilms, September 2024. Accessed: 2025-05-05.
[38] OpenCompass. Aime 2025. http:////ggggggaacaaaaaaaaaaaaaa AIME2025, 2025.
[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[40] Rafael Rafailov, Archit Sharma, Yiding Jiang. Ludwig Schmidt, and Stefano Ermon. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, 2023.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347 2017.
[42] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv:2410.08144 2024.
[43] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 2024.
[44] Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-hanaa Wei Hong. Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain- of-action-thought enhances IIm reasoning via autoregressive search. 25Xi::5502..2508, 2025.
[45] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint arXiv:2409. 19256, 2024.
[46] Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuming via adaptive curriculum learning. S.Xiv:2504.05222 2025.
[47] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In NeurlPS, 2020.
[48] Richard S. Sutton, Andrew G. Barto, et al. Reinforcement Learning: An Introduction. MIT Press, 1998.
[49] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Chal- lenging big-bench tasks and whether chain-of- offin-of-thought can solve them. 10.09122. 2022.
[50] Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. https://qwen.n. github. wtiiio//logg qwq- 1:2b-preview. November 2024.
[51] Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinssructt-2 Accelerating ai for math with massive open-source instruction data. arXiv:2410.01560, 2024.
[52] Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reason- ing with reinforced fine-tuning. In ACL, 2024.
13

---

[53] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv:2211 1422:2211.14275,, 2022.
[54] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verity and reinforce Ilms heep-by-step without human annotations. In ACL, 2024.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[56] Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. In AAAI, 2025.
[57] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. implementation of generative prm. https: / //t/t/tttbt/aat/ com/RLHE 1ow RLHE Reward Modeling, 2024.
[58] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, ct al. Qwen2.5 technical report. a1X1v:2412.15115. 2024.
[59] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self- self-imprrveement arXiv:2409.12122, 2024.
[60] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv:2503. /4476, 2025.
[61] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In NeurlPS, 2022.
[62] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction. In ICLR, 2025.
[63] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. 05X0022504.00891,, 2025.
[64] Chujie Zheng, Zhennu Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv:2412.000 22.55595,, 2024.
[65] Lianmin Zheng, Wi Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging alm-as-a-judge with mt-bench and chatbot arena. In NeurlPS, 2023.
[66] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In ICLR, 2024.
[67] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv: 1et:::999009999 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, supported by numerical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations have been discussed in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Section and Appendix D for full implementation details to reproduce our experiment results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: We are committed to reprodueibility and will open spen-source the code, and data upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: ///:://nii htt///nisssss cc/
des publ c/ ges CodeSulom ss onPol cyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (ht tnttpp: "nips.cc/public.. httpppublllgggl/ele
/ /nips. cc/
Codee Submi ss sswonnoooo conPoliee olicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all details of experiment setups. Please refer to Section and Appendix D for further information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As described in the Benchmark and Evaluation of Section 4, the results are evaluated via greedy decoding, which deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the information on the computing resources in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///eerrppp.eeeee http://scccccl.ietettcccooellell
Answer: [Yes]
Justification:
We strictly conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: have discussed broader impacts in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We propose RL post-training framework using the standard academic open-sourced datasets and models. This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the relevant papers and provided the licenses of the assets in Appendix G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers rswi thoode. com / da sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /neurrii i ps cc/ Conferences// 2025/ LLM) for what should should not be described.
21

---

A Broader Impacts
We introduce novel framework that jointly trains a generator and verifier via interleaved RL, with the goal of enhancing the reasoning capabilities of language models through mutual reinforcement. While we believe this approach can meaningfully improve the reliability, interpretability, and general- ization of LLM reasoning, several potential societal risks associated with such advancements warrant careful consideration.
First, as models become increasingly proficient at producing fluent and logically structured multi-step outputs, there is risk that such capabilities may be applied in settings where correctness is difficult to ascertain, or where persuasive yet flawed reasoning can lead to harmful outcomes, for instance, in domains like education, policymaking, scientific communication. Second, although the verifier provides step-level natural language feedback that is more transparent and less noisy than scalar reward signals, such responses may still perceived as authoritative even when incorrect, particularly by non-expert users. Third, the use of RL in open-ended generation remains imperfectly understood, and may introduce unintended behaviors through reward misalignment or policy oereootimizziinn.
That said, we note that these concerns are not unique to our approach, but reflect broader challenges in RL for LLM reasoning. Our design choices, such employing generative, text-based verification and sampling-based reward signals, may help mitigate some known issues related to rigidity and reward hacking. Moreover, by training the verifier through RL rather than supervised imitation, our framework encourages the verifier to develop more accurate and adaptive verification strategies, which may help reduce the likelihood of producing misleading or unjustified assessments. Nonetheless, we encourage responsible and cautious use of this framework, particularly in domains where reasoning quality directly impacts users or high-stakes decisions.
B
Limitations
While TANGO demonstrates strong performance across range of reasoning tasks, several aspects merit further discussion. First, the co-evolving RL training in TANGO offers a higher performance ceiling compared to single-agent RL approaches, but realizing this potential requires careful tuning, such as generator-vent veriffer update schedule and advantage mixing strategy. While the framework remains effective with minimal tuning, achieving optimal performance relies well-coordinated training dynamics. Second, jointly training the generator and verifier in TANGO introduces additional computation compared to optimizing the generator alone, but the overhead is modest in practice. Much like verifying a mathematical proof easier than constructing one, the verifier's 's task is more sample-efficient and easier optimize than generation, allowing less frequent updates. We empirically find that updating the verifier once every three generator steps strikes a good balance between performance and efficiency, keeping the overall cost comparable single-agent setups. Lastly, while our evaluation spans both mathematical and several structured out-of-domain reasoning tasks, including logic, code, commonsense, and tabular reasoning, it does not yet cover more open-ended domains such as long-contess reasoning, multi-hop fact verification, or dialogue- based reasoning. We expect TANGO can generalize such settings, which we reserve for future work.
22

---

C
TANGO Algorithm Flow
The algorithm flow of TANGO is detailed in Algorithm 1.
Algorithm 1: Interleaved RL Training of Generator and Verifier in TANGO
Input:
Training data distribution P, generator policy Tg. verifier policy Te. mixing weight a, rollout Size M, generator update steps Ng. verifier update steps No
Output: Trained generator To and verifier while not converged do for n = to Ng do
Sample batch B {or
For each (x,y) E B, generate M rollouts multi-step step solutions M ~ [g(: x) and query the verifier to generate corresponding verifications: g
og (-(- x, o)), 1.
Extract predicted answer ÿ* from of and step-level judgments (Visp, & &=== from of
Compute generator advantages {A J.M via Eq. (7). Perform policy gradient update generator Fg using At 9.2-
end
for to N2 do
Sample batch B ~ P. For each (x.y) B, generate a multi-step solution Og ~ T Tg x) and query the verifier generate M verification rollouts: LM (- x, 0,)),
1, Extract final judgment Vinua from o Compute verifier advantages 11 via Eq. (9).
M. ,,2.
Perform policy gradient update on verifier n. using A
Update EMA reweighting coefficients 94 and s..
end end
D Additional Experiment Details
D.1 Main Experiments
In addition to the experimental setup described in Section 4, we provide further details below.
For the SFT stage, first generate training data by prompting Llama-3.1 --strr--ntt (system prompt shown in Section F.1) with decoding temperature generated responses then used perform SFT on the generator base model Qwen2.5-Math-7BB. We conduct full-parameter SFT using a learning rate of 5 x 10-6 with the Adam W optimizer, a cosine annealing learning schedule, and warmup ratio of 0.1. The model is trained for 800 steps with a batch size of 64.
0.1 and a top-p value 0.5. The
For the RL stage, both the generator and verifier generate rollouts using sampling temperature of 1.0 and top-p value of 1.0. We set the KL loss coefficient 3 to 0.001. The EMA decay factor for tracking correct and incorrect samples from the generator is set to 0.8.
D.2 Algorithmic Reasoning Experiment
For the algorithmic reasoning task, specifically, the last-letter concatenation experiment presented in Section 4.3, we first construct SFT datasets by randomly generating 2 to words, each containing characters. These datasets are used to train both the generator and verifier for several dozen steps. primarily to ensure that the Owen-2.5-1.51 base models learns to follow the specified instructions and produce outputs that conform the required format. For RL training dataset, we similarly generate input sequences consisting of 2 to 10 words, with each word containing 3 to 10 characters. The test dataset is constructed in the same manner but includes slightly longer sequences, to 12 words with generalization ability.
3 to 12 characters, to cover the evaluation of the model's 's out-of-distributie Most training hyperparameters follow those used in the main experiments detailed in Section 4 and Appendix D. except that use a batch size of 64 and an exponential learning rate decay schedule for the generator. For the three baseline settings, vanilla GRPO, fixing the generator, and fixing the verifier, we adopt the same configurations to ensure fair comparison.
23

---

E Additional Results Using the Llama Base Model
To further demonstrate the generalizability of TANGO, we include results from Llama base model in addition to the Qwen series of models. Specifically, we conducted experiments using Alama-3.1- 8B-Instruct [16] as the base model for both the generator and verifier, following the same training and evaluation protocol as in Table 1. The results are presented in the table below.
Table 4: TANGO performance using Llama-3.1 Gaama-3.188B-Ima Instrutt as the base model on mathematical benchmarks. TANGO still achieves substantial improvements across datasets Llama base models.
Model
MATH500
AIME2024
AIME2025
AMC2023
OlympiadBench
Avg.
Llama3. 1-8B-Instruct (Basc)
46.4
3,3
3.3
22.5
13.6
17.9
TANGo-Llamms Ja..aa..m..----..... 1-88-SFT
49.4
3.3
3.3
25.0
15.0
19.2
GRPO
56.2
10.0
3.3
20.9
25.1
GRPO
These results show that even when using Llama as the base model, TANGO continues to deliver significant improvements, demonstrating strong generalization across different model families. This further confirms that TANGO's 's effectiveness stems from our interleaved RL co-evolving framework, rather than being specific to any particular model family.
F
Additional Generation and Verification Examples
F1
Mathematical Reasoning
Generator's Prompt in Math Reasoning
helpful Assistent solves mathemat-caa. problens step by step.
Your task is to provide detailed solution process within specific tags.
You MUST follow this exact format:
1. Start with a <thinks section containing your step-by-stop reasoning, Traide <thinko, each dist inet logioal step MUST enal in its <step> </step> Lags.
ethinks, provide final answer within </answer> tags, using bboxed format.
Here example of the required format:
Calculate
Assistant: <thinks <stephrirst, calculete expression inside parentheses, multiplied by stepa
<step>3 2 equals 6.</step>
<step>Wext, subtract the result from the original number, which is 15 minus 6.</step> <step>15 6 equa a 9.</step> </think>
You MUST strictly adhere this Format.
Output ONLY the content within <think), <step>, and <answer> tags.
NoT include any
characters before the opening <think> tag after clasing
</anawer> tag.
Ensure every part your step by step reascning inside <step> </step> Laga within <thinks </thinks section.
the ansver inside <answer> (noner>>boxadd(<<<nneeeee //</anwee>>. final ansver extracted autematically by the \boxed(] tag.
how many integers does the expression
Tog(n²) (logn)²
log n 3
represent real number, where log denotes base 10 logarithm?
Assistant:
24

---

Generator's Response Math Reasoning
<thinks
<stepasirat, simplify the expression ingide the square root: log(x²") [lugn)?<</atep) <<stop>> <stepaUsing the logerithm property log(n") n, the expression becomes log (logn) c// step>
<stephšo the expression inside the square root is 2log 2logm=((ogn) </step>
logn-3
<step>ractor the numerator: 2log (logn)² = logn(2 log </stepp
<step>The expression now is los [oomn12-oggn) <//toop>
Ing m-
<step>ror the expression to be real number, the fraction lox loxn(a log n) must be non-negative
lng
and log & 3.</stpp>
<stepsConsidar the critical points where the numerator or the denominator is zaro: logn 0, logn cstep=logn =0 implies n == =.</sttp>
log = 3/s/sppp
<step>logn impltes calep=logn = implies
100.</atep>
11000000/ee </ateg/
<step>Nos, analyze the intervels determined by these critical points: (0, 1), (1,2), (2,3), (3, <//stt/
<stepster I): 0 log < implias Ing - logn) > 0, and logn - 3 < O, so the fraction is positive.</step>
<step>Por (1,2): 0 logn <22 and logn -ogn-<< -1, so the numerator and deneminator must be positive or regative.</step>
negative.
log 3 3 -1, fraction is always
<steph:'r (2,3): 1 < log < implies log lo > 0, and logn 3 -1, so the fraction is Degativa.</stepp
<stepoTor (3, logn > 0 and logn - -on-3>100 I, so the numeracor and cenominator are both fraction positive.</ster
negative, making
<step=Ihus, the expression is defined for 0 <
and 100 n 1000.</stap
<step>The integera
intervals are
101, 10.............................. 999, which gives us
999 i/tegers.</step> </think>
1000 /anawer>
25

---

Verifier's Prompt in Math Reasoning
You are verification assistent specialized in mathematica". reasoning. Your taak is carefully evalvate the provi ded solution step by step, checking for mathemation correctness and logical echerence, You will be given the original problem and the Assistant' s sclutict, which contains specific aumber of steps within <step> tags. You MUST verify EACH <atep> block found in the Ansistant's solution and provide your judgment using the exact *ormat spec' fled in the natructionss You MUST output ONLY the content wichin the specified verification tags nothing
Here the problem you need to verify, and the Assiatant's solutions
Problem: For how many integers n does the expressien
log(n") (logn)
log n 3
represent real number, whare log danotes the base 10 logorithm?
Assistant's Solution:
<think>
<steporirat, simplify the expression inside the square root: log(n") (log " <//ttep> <stepxUaing the logarithm property log(n") llogn, the expresatio becomes log ()og step>
<step>50 the expression inside the aquare roat is 2 n (log (log </step>>
log n
<step>lactor the numerator: 2log (logn]² = log ((2 log </step>
<step>The expression
V flog (10m(12)))))) </step>>
los
<stepeTor the expresaion to real number, the fraction log lngn(22lopm) muat be non-negative log o 3/sttep>
lug
<atep>Consider the critical points where the numeratos or the denominator is zero: log 0, logn
log = 3.</spp>>
<sLep=logn =0 implies n 1.</step>
<step>logn== = implies n =00.</step> <step>lugn = implies
1/00e</step>
<step>Nos, analyze the intervals is determined by these critical pointa: (0, 1), (1,2), (2,3), (3, </stee)
<step>Por (U,1): 0 log <1 implies log logn) > 0, and logn - 0, so the fraction is <ositivs.</step>
<step>Por (1,2): 0 log < and logn 3 -1, the rumarator and denominator must be either beth positive or both negative. since < log 3 3 -1, the fraction is always sogativa.</step.
<stepoTor (2,3): 1 < log 2 Implies log logn) > 0, and logn - 3 -1, so the fraccton is negative.</steg>
<step>:or (3) log > 0 and logn-3>2 1, so the numerazor and denominator are beth negativa, making
fraction positivo.</step
<step>Thus, the expresaion is detined for 0 < < and 100 a 1000.</steps
<step>The integera in these intervals are n and n 101, 102. ...,.,,, 999, which gives us integers.</atap>
</think>
<answer> 1000 </answer>
Aastatant's solution containa steps within <step> tags.
<step> blocks in Assistant' a Scluticn, you BUET provide ONE corresperding verification analysis within <atep> tag inside the rstep_verificatiee judgment in the ffinaliverification> tag.
verify this soluticn step by step. each
section. Afcer verifying ell steps, provide final overall
MUST follow this format:
<step>step 1 Analysis: your detailed verification reasoning goes here. concluda with only one judgement: (CRREC() or \boxed (NCORRETI)))) </step- <step>Step 2 Analysis: your detailed verification reasening goes here. Conclude with only one judgement: Maved[[ooReet) or \ [ccoRRCTT<//atapp ... (CONTINUE </step.verificationn
<steps blocks
Assiatant's Solution]
Seinal_verifee www.tiin (CORRECT) or \boxed (anessscccc//fmmaaeeeffffic
Hore exampla:
Problem: What Assietant's 's Solution:
<thinks
rste miltiply 5 by 3. <5.t/step>
26

---

<step=Iher, sdd to the result, 15 18.e/step> </think> <answar>\boxed (16)</anawer>>
Your Verification:
<atep_
<step>Step Analyeis: The aultiplication 5 is correctly calculated as 15, This step is mathematicslly sound. (Cosstt))cc))
<step>štap 2 Analysia: Adding the previous result (15) givea 16, which correct.
Tala atep tollows logically and 18 nathematioalll accurate. (CRRe))) c/stop_verification <*inal ft catoaatatooooooooo www.oodddoooooooo (afaaact//ffnaaleerrrt
IMPORTANT INSTRUCTIONS (Read Carefully):
1. The Assistant's solutien hes 17 steps. You MUS: anslyze and provide a verification for EACH EVERY of these stepa. The number of section MUST be exactly 17.
within your satep_verl fictti
analyze step provide original solution text.
copy the Each verification <step> must end with EXACTLY ONE judgement: either (COREEC: \
verification reasening
4. Your final verification within cfimal_vorification must judge whather the overall correct.
rinal.
5. You MUST output OND? the content within the velep_veriiiaationn and ve:
tags. no NOT output anything else.
Your Verification:
27

---

Verifier's Response Math Reasoning
estop_varificattion
the expreasion the square root 1a necessary before proceeding further, The is correctly Ldeatilying the need to simplity the expression. CORRECT </step>
<step25teep Analysta: The simpli
<step>Step 2 Analyais: The logarithnic property log(n*) = 2 la applied correctly, transforming the expression into 2 (logn)". This step mathenatically sound. CORRECT </step>>
<step>Stap 3 Analysis: The expressien inside the square reat is corractly rewricten after applying the logarithmie property from Step 2. CORRECT </step>
<stepeatep 4 Analyals: Factoring out logn from the numerator results in log n22 - logn), which is valid algeoraic operation. CORRECT </step2
<step>Step 5 Analysis: The expression is correctly simplified to log (o22-ogg)) This step los n-
follows logically from the previous steps. CORRECT </step>
<step>step 6 Analysis: for the square roct to represent real number, the argument must be denominator connot be zero, meaning log This
son-negative. Additionally,
condition is correctly identified. CORRECT k/step>
<aLepoSLep Analysis: Identifying critical points where the sumerater denominstor equals zero is crucial for determining the sign of the fraction. The critical points are correctly identified log = logn
logn =3 3. CORRECT </step>
<step>Step 8 Analysis: When logn = 0, it implies n 10" =1. 1. This step is correct.
CORRECT </step>
<step>Step 9 Analysis: When logn = 2, it implies = 10 = 100. This step is slso correct.
CORRECT </step>
<sLep=SLep 10 Analysis: When logn = it implies n 10 = 1000. Tais step is correct.
CORRECT </step>
<step>stt.cp Analysis: The intervels determined by the critical points are cerrectly intager, the
identifled (0, (1,2), (2,3), (3, mo). However, since
interval (0, 1) ahould not be conaldered. INCORRECT </atep>
<step2sttep 12 Anal ysis: For the interval (0, I), the analysia is incorrect bacause n cannot
less than if it is en integer. The correct interval to consider would be 1), which
empty. INCORRECT </step>
<step>Step Anslysis: Fer interval (1,2), analysis incorrect. should that the fraction is nagative because the numerator is positive and the danominator is
negativa. INCORRECT </step>
<aLep>BLep 14 Analysis:
Por the interval (2, the analysis is incorrect. It should state denominator
that the negative. INCORRECT </step>
is positive
numerator is negative
Per the interval (3, the analysis incorrect. should state that the fraction is nagstive because the numerator is negutive and the danominator is
<step>Step Anslysis:
positive. INCORRECT </stepp>
<step>Step Analysis: Based on the analysis of the intervals, the expression defined for 100 1000. The conclusion about the intervals is partielly incorrect due to the
misintespzetation of the first interval. INCORRECT </step>
<step>Step 17 Anslysis: The count of integers in the intervals is incorrecs based on the corrected intervais. There are integers (0, and correct count 101 S n $
1s 999 101 899. Including n incorrectly adda to the contusion. INCORRECT </step> </step_veriflcations
cinaa_vvriifications
INCORRECT </tina //tina__verr_ ver! fication>
28

---

F22
Algorithmic Reasoning
Generator's Prompt in Last Letter Concatenation
cenversation between solves it step by step.
Assistant.
question,
The Asslatant thinks through the problem in cleas, logical steps, with each step enclosed in <step> </step> tags within the overell <thinks </think> section. Each <atep> </step> tag pair should enclose ALL Assistant provides the final enswer within <answer> </answer> tags.
for that step. After completing the thinking process, the
The Assistant should strictly lol the fornat below:
<think>
<step>Your deteiled reasoning. [Statusl: Letters </step> <stephYour datailed reasoning. [Statual: lettera </step>
</think>
<anawer>\hoxed YYur final lettera)</answer>
User: You concatenate them to form a new word.
several words. Your task extract the last letter of each word and
This is the problem: prepurpose, overflow,, lovesick, burghalpenny, toran, outworn, coadjacence, forgotten, imparilment, spidictical
let us think step by step and please follow the format: <stepo Your detailed reasoning. Statusl: Letters </step>. the a-th step, extract the last letters of the first words and replace Letters with tha concatenaticn of these letters. Finally, return your result usiing following
senswer>\boxed/y
final <
must output exactly steps for this problem. em. not skip or any extra steps.
Assistant:
Generator's Response Last Letter Concatenation
<thinks
<stepoPxt the lettar from preporpose, get Statusl: es/step>
<alepeSxtzact the last letter from everflows, get 6, [SLatusl: ed</stepo
<step>Extzact the lest letter from lovesick, gez k. [Status edk</step>
<stephextract the lest letter from Eurghalpenny, <stepa"xt the 1ast lettar from toran, get n. |Statusl: adkyns/stap> <stephextract <step>ext ract the lest lettar from coadjacence, get [Statua]: edkynnes/step>
y. |Status|: edkys/step>
cutworn, get [Status]: edkyan</step>
<stepeExt the last letter from forgotten, get 6, |Statual: edkynneds/scep> <step>Extract the last letter Erom imperilment, get [Status]: edkynzednc/step
<stephextract the lest letter from epidictical, get 1. [Status|: edkymsednl</step>
</think>
<answer> edkynnedml </answer>
29

---

Verifier's Prompt in Last Letter Concatenation
You are verification assistent specializing in olgorithmic readoning, Your taak is carefully evalvate each step in the provided solution problem The problen lavolves concatenating the last lettera of list of words to Fozm a new string. You must verify BACH numbered step in the solutien and provide youz judgment using the exact format specified the instructions.
last letter concatenation
Here problem you
verify,
Assistant's solution:
Problem:
prepurpose, overflown, lovesick, burghalpenny, toran, outworn, coadjacence, forgotten, imperilment, epidictical
Assistant's Solution:
<thinks
<stepsExtract
prepurpose, get Statusl: e</step>
<step>=xt the last letter from evarflow,, get 6. Istatual: ed</step>
caleptExizact the last letter from lovesick, get k, IStatus edkc/step>
<step>Extzact the last letter from burghalpenny, <stepssxtract the lest letter from toran, get n. |Statusl: ackyn</stsp> <step25xt <atep=Sxtract the last letter from coad'acence, get e, (Saausl:: edkymae</atep> <step>Extractt
y. |Status]: edky</step>
get |Statual: edkynn</steps
forgotten, get é. |Statual: edkynnadk/ssopp <stepe5xt the last letter from imperilment, get 1Status|: edkynneem</step>
<stepeExtract the lest letter from epidictical, get 1. 18tatus/: edkyneednl/sttep </think>
<anawar> edkynnedm </answer>
The Assistant's solution conteins 10 steps within <step> tags.
Please verify this solution atep by For each of the 10 <ateps blocks In the Asalstant' s Selutien, you MUST provide ONE corresperding verification analyeis within a <atep> tag inside the setep_veriffletiion judgment in the veinal_verificaaiion Lag.
section, After veritying all stepa, provide final overall
You MUST follow this exact format:
vetep_verification
<step>Step 1 Analysis: Your detailed verification reasening goes here. Cenclude with only judgement: (ZORECC))
<step>Step 2 Analysis: Your detailed verification reasoning goes here. Conclude with only judgement: Vboxed
\boxed (ICURREC)))
[CONTINUE for ALL 10 <atep> blacks in the Assistant's Salution] </step_vers www.iiiaiiiiioon ciinal_vesiflcattonn (boRed
\boxed] /
IMPORTANT INSTRUCTIONS (Road carefully):
1. The Assistant's solutien has 10 steps. You MUST anslyze and provide a verification for EACH and EVERY
thase steps. number <step> tags within your katep_verifcaat sectioon MUST be exectly 10.
You MUST analyze the step and provide YOUR OWN verification reasoning DC NOT copy the original solution text.
Each verification <step> must end with EXACTLY ONE judgement: either Vboxed (CORKECT) or (TCCORFFCCRFCCT).
4. Your final verification within <fimel_ must judge whether the overall ensver correct.
solution
You MUST output ONLY the content within the vstep_veriticationn and frisstionctioootiioo fication> tags. Do HOT output anything else.
Your Veriiinttin:
30

---

Verifier's Response Last Letter Concatenation
<step_ sntepp_varificationn
<stepp3teep Analysta: The last letter preperpase e; Assistant gets e response.
CORRECT k/step>
<step2Step Analysla: last letter overflown is ny Assistant gets in 15s reaponse,
INCORRECT </step>
<step>Step Analysis: The Last letter lovesick Assistant in its response.
CORRECT </step>
<step>Step 1 Analysis: The last letter of burghalpenay is yi Assistant gets in its
response.
CORRECT
</step>
<st.ppsst.pp 5 Aralysia: The last letter of coran ia ny Assistant gets n in its response.
CORRECT
</step>
<st.ppStt.pp Analysia:
outworn my Assistant geta n its response.
CORRECT </step>
<step>>tt.p 1 Analysia: last letter coadjacence Assistent in its
response. CORRECT </step>
<sLep>SLep 8 Analysis: The last letter of torgotten is Assistant gets in iis reapense.
INCORRECT </step>
<sLep=BLep 9 Analyais: The last letter of imperilment La by Assistant gets in its
response. INCORRECT </step>
<step>step Anslysis: The lest letter of epidictical is 1, Assistant gets in its
response. CORRECT </step>
c/step_verificatiinn
<final verilll_eiiiitiion CORRECT </final_werificattt
G Licenses for Assets
All models, datasets, and codebases used in experiments open-sourced and adopted under permissive research licenses, as detailed in Table 5.
Table 5: Licenses for external assets used in experiments.
Asset
Type
License Terms
Owen2.5-Math-77 [59]
Model
Apache 2.0
Qwen2.5-7B [58]
Model
Apache 2.0
Llama-3.1 1...-.11---Isstttti [16]
Model
Llama 3.1 Community License
Qwen2.5-115B [58]
Model
Apache 2.0
Burus-2-FFT-Data [12]
Dataset
MIT
Burus-2-RL-Data [12]
Dataset
MIT
MATH 500 [32]
Dataset
MIT
AIME 2024 [3]
Dataset
Apache
AIME 2025 [38]
Dataset
MIT
AMC 2023 [4]
Dataset
Apache
OlynpiadBench [20]
Dataset
Apache 2.0
Buandgame [26]
Dataset
CC-BY 4.0
CRUXEval [17]
Dataset
MIT
StrategyQA [15]
Dataset
MIT
TableBench [56]
Dataset
Apache
ProcessBench [64]
Dataset
Apache 2.0
veRI.
Codehase
Apache 2.0
31


Reviews:
[{"review_id": "zg2AvWpebJ", "reviewer": "Reviewer_DEZv", "summary": "The paper proposes to jointly train a \"generator\" policy model along with a process reward model \"verifier\", using RL from the outcome-level rewards only. This leads to strong empirical gains in math and reasoning tasks at the 7/8B model scale.", "questions": "Q1. The claim the generative verification mitigates reward hacking is intriguing. Aside from the overall strong performance of the method, is there any more specific evidence that mitigating reward hacking is one of the mechanisms by which performance improves?", "limitations": "Yes, although some of these math datasets are quite small so it would be good to have confidence intervals or some other measure of finite-sample variance", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "S1. The primary strength of the paper is empirical: the trained process verifier achieves substantial performance improvements, apparently over strong baselines.\nS2. The proposed approach also yields text-based judgments, which the authors claim mitigates reward hacking.\nS3. The experiments seem solid and rigorous, although these reasoning tasks are not my area of expertise. \n\nW1. While the approach is well-motivated at a high level, there is relatively little conceptual novelty and a considerable amount of heuristic reward/advantage shaping (sec 3.2). However, I think that's okay for this type of paper so I am still recommending accept.", "was_revised": true, "final_justification": "This paper proposes generative reward models as a more robust approach for process-level rewards. The empirical results are strong and the approach is conceptually well-motivated. I recommend accept.", "rebuttal": "Dear Reviewer DEZv,\n\nThank you very much for your constructive comments and positive feedback. It is heartening to note that you found the proposed approach well-motivated, the experiments solid and rigorous, and the empirical results strong. Below, we address your questions one by one and provide additional clarifications.\n\n> ***W1. Clarification on the novelty of our paper.***\n\nWe thank the reviewer for acknowledging and recognizing the contributions of our work. We believe that novelty in deep learning research, especially in the LLM era, can manifest in various ways. Often, the most impactful advances arise not from introducing entirely new components, but from developing approaches that are particularly **well-suited to a specific context**, thereby enabling a major performance boost. Similarly, the novelty of our paper stems from recognizing that jointly training the verifier and generator is essential, and that such training *cannot* be effectively achieved with a verifier trained via SFT. Instead, RL is required for both components. (Our novelty is also acknowledged by Reviewer esdQ.) This design, while conceptually **simple**, is nontrivial to realize and proves **highly effective** in practice, leading to a substantial boost in performance. \n\n> ***Q1. ... is there any more specific evidence that mitigating reward hacking is one of the mechanisms by which performance improves.***\n\nThanks for the question. As the reviewer noted, the strong empirical results support this claim. Below, we provide both an intuitive explanation and concrete evidence for why we believe TANGO’s generative, text-based verifier mitigates reward hacking and contributes to improved performance:\n\nCompared to a logit-based process verifier, a generative, text-based verifier is inherently more robust against exploitation. Consider how a generator might attempt to maximize process rewards in each case. With a logit-based verifier, the generator can game the reward by producing sequences that merely increase token-level log probabilities under the verifier model, regardless of whether the underlying reasoning is actually correct. In contrast, a text-based verifier evaluates reasoning by generating natural language judgments (e.g., stating whether a step is valid and why), which requires the generator to produce semantically coherent and logically sound reasoning in order to obtain a high reward. This makes shallow optimization strategies far less effective and substantially reduces the risk of reward hacking. \n\nThe ablation results shown in Figure 3 of the main paper further support this intuition: compared to PRIME, which uses a logit-based PRM, TANGO, with its generative process-level verifier, achieves better performance, even though both approaches involve co-training a generator and a verifier.\n\nWe will add the above discussion in the revised paper. \n\n> ***It would be good to have confidence intervals or some other measure of finite-sample variance.***\n\nThank you for the valuable suggestion. We agree that reporting error bars from multiple independent runs is especially important for smaller math datasets such as AIME24, AIME25, and AMC23. Due to limited time and computational resources during the rebuttal period, and the high cost of RL with 7B-scale models, we report here the mean and standard deviation of accuracies for GRPO and GRPO w/ TANGO (corresponding to the second and third rows of Table 1 in the main paper) based on three independent runs across the five math datasets, as shown in the table below. These results demonstrate a relatively **stable** trend across runs. We will include error bars for the remaining results in the revised paper.\n\n| Model            | MATH500 | AIME2024 | AIME2025 | AMC2023 | OlympiadBench |\n|:-----------------|:-------:|:--------:|:--------:|:-------:|:--------------:|\n| GRPO             |  74.4 ± 0.3  |   11.1 ± 1.6   |   8.9 ± 1.6   |  50.8 ± 1.2   |     37.1 ± 0.3       |\n| **GRPO w/ TANGO**| **81.3 ± 0.2**| **18.9 ± 1.6** | **18.9 ± 4.2** | **62.5 ± 2.0**|   **43.5 ± 1.1**     |\n\n---\n\nWe hope our response has addressed all of your concerns and can lead to a favorable increase of the score. Please feel free to let us know if you have other questions or suggestions."}, {"review_id": "z1H83Bzol4", "reviewer": "Reviewer_A2Yu", "summary": "In this paper, the authors address the limitations of vanilla fixed verifiers which are prone to reward hacking and exhibit poor generalization. To overcome these issues, they propose a framework called TANGO, which employs reinforcement learning to jointly train an LLM generator and a verifier in an interleaved manner. Notably, TANGO dynamically trains a process-level LLM verifier using only outcome-level rewards, thereby improving training efficiency through more fine-grained reward signals.", "questions": "Please see the weakness.", "limitations": "Please see the weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The authors train a verifier that combines the advantages of both outcome-level and process-level supervision, enriching the reward signals available during training.\n2. The proposed interleaved RL training framework enables the generator and verifier to be jointly optimized, allowing them to continuously improve in a co-evolutionary manner.\n\nWeaknesses:\n1. The experiments are conducted solely on the Qwen series of models, which is insufficient to demonstrate the generalizability of the proposed method. Including results on additional base models would make the findings more convincing.\n2. The efficiency is a concern. Since TANGO requires interleaved training between the generator and verifier, it introduces higher computational costs. For instance, as shown in Figure 1, the verifier undergoes a 40-step warm-up and must be periodically updated during training. The authors should ensure that vanilla RL baselines are granted equivalent training steps or report the actual wall-clock time to reflect computational efficiency.\n3. The ablation study is insufficient. A dedicated analysis of the process-level reward is necessary to validate the effectiveness of the implicit step-wise supervision.\n4. It could be better to outline the contributions in the Introduction section to improve the overall readability and clarity of the paper. The code has not been released, which limits reproducibility and reduces the potential impact of this work on the broader research community.", "was_revised": true, "final_justification": "The response has solved my concerns. \n\n- The generalizability of the proposed method has been verified on other LLM backbones. \n- The model is more efficient with smaller training steps.\n- The ablation study is sufficient.\n\nThus I raise the score.", "rebuttal": "Dear Reviewer A2Yu,\n\nThanks for your insightful and detailed feedback. We appreciate your recognition of TANGO’s strengths and contributions, particularly its enriched reward signals and co-evolutionary training framework. Below we address your concerns one by one.\n\n> ***Including results on additional base models would make the findings more convincing.***\n\nThank you for the valuable suggestion. We agree with the reviewer that including results on additional base models would strengthen the validity of our findings. Following the reviewer’s suggestion, we conducted experiments **using Llama-3.1-8B-Instruct as the base model** for both the generator and verifier, following the same training and evaluation protocol as in Table 1. The results are presented in the table below.\n\n**Base Model: Llama-3.1-8B-Instruct**\n| Model            | MATH500 | AIME2024 | AIME2025 | AMC2023 | OlympiadBench | Avg. |\n|:-----------------|:-------:|:--------:|:--------:|:-------:|:--------------:|:----:|\n| TANGO-Llama3.1-8B-SFT     |  49.4   |   3.3    |   3.3    |  25.0   |     15.0       | 19.2 |\n| GRPO             |  56.2   |  10.0    |  3.3    |  35.0   |     20.9       | 25.1 |\n| **GRPO w/ TANGO**| **60.5**| **13.3** | **6.7** | **40.0**|   **23.6**     |**28.8**|\n\nThese results show that even when using Llama as the base model, TANGO continues to deliver significant improvements, demonstrating strong generalization across different model families. This further confirms that TANGO’s effectiveness **stems from** our interleaved RL co-evolving framework, rather than being specific to any particular model family. We will include the above results and discussion in the revised paper. \n\n> ***The efficiency is a concern … The authors should ensure that vanilla RL baselines are granted equivalent training steps ... to reflect computational efficiency.***\n\nThank you for the question. We would like to clarify that for Figures 1 and 2, as well as Table 1, we followed the same evaluation protocol as the PRIME [1] paper, comparing our method to vanilla RL baselines based on the number of generator update steps. However, we agree with the reviewer that from an efficiency standpoint, it is important to ensure that vanilla RL baselines are granted equivalent total training steps. In response to the reviewer’s suggestion, we extended the experiments in Figure 3 to include detailed step counts and additional comparisons with all baselines (vanilla RL, ORM, PRIME). The results are presented in the table below.\n| Model            | G training steps | V training steps | Total training steps | MATH500 | AIME2024 | AIME2025 | AMC2023 | OlympiadBench | Avg. |\n|:-----------------|:-------:|:-------:|:----------------:|:-------:|:--------:|:--------:|:-------:|:--------------:|:----:|\n| GRPO (Original)             |   200     |   0     |        200         |  74.6   |   13.3   |   10.0   |  50.0   |     36.9       | 37.0 |\n| GRPO  (Extended)           |   400     |   0     |        400         |  76.8   |   13.3   |   13.3   |  55.0   |     38.8       | 39.4 |\n| GRPO + ORM       |   200     |   200 (pretrained)     |        400         |  75.8   |   13.3   |   16.7   |  55.0   |     39.1       | 40.0 |\n| PRIME (GRPO + PRM)       |   200     |   200 (alternating)     |        400         |  79.4   |   16.7   |   13.3   |  60.0   |     41.9       | 42.3 |\n| **GRPO w/ TANGO**|   200     |   40 (warmup) + 66 (alternating)     |        **306**         | **81.4**| **20.0** | **20.0** | **65.0**|   **43.9**     |**46.1**|\n\nSpecifically, we added an experiment that extends vanilla GRPO training from 200 to 400 steps. As shown in the table, doubling the training steps yields limited performance gains, particularly on the most challenging benchmarks such as AIME2024 and AIME2025, where performance nearly plateaus after 200 steps. For the stronger baselines, GRPO + ORM and PRIME, both require training a verifier (reward model). In GRPO + ORM, the ORM is pretrained for 200 steps. In PRIME, the PRM is updated alternately with the generator in a 1:1 ratio, also resulting in 200 update steps. In our method, GRPO w/ TANGO, the verifier is first warmed up for 40 steps and then updated alternately with the generator at a 3(G):1(V) ratio, resulting in 66 additional verifier updates, yielding a total of 106 verifier update steps.\n\nTaken together, these results show that our method achieves the best performance while **using fewer training steps**. This highlights that TANGO’s improvements stem from our proposed training framework rather than from increased training steps. We will update the above results and discussion in the revised paper. \n\n> ***The ablation study is insufficient. A dedicated analysis of the process-level reward is necessary to validate the effectiveness of the implicit step-wise supervision.***\n\nWe thank the reviewer for the suggestion. First, we would like to clarify that the results presented in Figure 3 of the main paper (which we have augmented and summarized in the table above) already ablate the effectiveness of our RL-trained step-wise supervision, in comparison to: (1) SFT-trained logit-based token-level supervision (PRIME), (2) outcome reward from a pretrained ORM (GRPO + ORM), and (3) rule-based outcome reward (vanilla GRPO).\n\nTo further validate the effectiveness of our step-wise reward design, we conducted two additional ablations:\n\n1. **Removing step normalization in Eq. (3)**: We change the reward from the normalized form $r_{g,\\text{step}}^{I(k)}(o_g)= \\frac{y_{\\text{step},k}}{K}$ to the unnormalized form $r_{g,\\text{step}}^{I(k)}(o_g)= y_{\\text{step},k}$; \n2. **Removing intermediate step-level supervision**: Instead of assigning step-wise rewards at each reasoning step, we aggregate the step signals into a single scalar $\\frac{\\sum_{k=1}^K |{y}_{\\text{step},k}|}{K}$ and add it to the final outcome reward. This setting reduces step supervision to a coarse auxiliary signal at the final step. \n\nWe summarized all the results in the table below.\n\n| Model            | Outcome Reward | Process Reward | Avg. accuracy on five math benchmarks |\n|:----------------- |:----------------:|:----------------:|:----:|\n| GRPO  (400 steps)           |   rule-based, hard        |  no process reward | 39.4 |\n| GRPO + ORM       |   soft from ORM  |  no process reward  | 40.0 |\n| GRPO w/ TANGO (removing step-level supervision) |  rule-based, hard + RL-trained soft step accuracy | no process reward | 42.0 |\n| PRIME (GRPO + PRM)       |   rule-based, hard | SFT-trained, token-level        |  42.3 |\n| GRPO w/ TANGO (removing step norm) |  rule-based, hard | RL-trained, step-level (no step norm) | 43.4 |\n| **GRPO w/ TANGO** |  rule-based, hard | RL-trained, step-level (w/ step norm) | **46.1** |\n\nThe results above reveal several key findings: \n\n- Even when step-wise signals are only used as an auxiliary term in the final reward (without intermediate supervision), the method still outperforms the ORM baseline, validating the effectiveness of the learned step-level supervision;\n- However, assigning supervision explicitly at intermediate steps as process rewards is significantly more effective, as it enables more accurate credit assignment and facilitates early-stage exploration;\n- Normalizing step-wise rewards by the number of reasoning steps, as done in our method, is critical. It removes the policy's bias toward longer or shorter step lengths, allowing the generator to flexibly determine the appropriate number of steps for each problem.\n\nWe will add the above results and discussion in the revised paper. \n\n > ***It could be better to outline the contributions in the Introduction section to improve the overall readability and clarity of the paper.***\n\nThanks for the constructive suggestion. We will include a clear list of contributions in the Introduction section in the revised paper.\n\n > ***The code has not been released.***\n\nIn fact, we have already released the code to ensure reproducibility. However, to preserve the integrity of the double-blind review process, we omitted the code link in the current submission. The code link will be included in the camera-ready version upon acceptance.\n\n---\n\n[1] Process Reinforcement through Implicit Rewards. Cui et al. Arxiv 2025.\n\n---\n\nWe hope our response has thoroughly addressed your concerns, and would really appreciate it if you could consider raising your score accordingly. If you have any further questions or suggestions, please do not hesitate to share them. We are eager to engage in further discussions with you."}, {"review_id": "R7KXQaC80n", "reviewer": "Reviewer_5Vry", "summary": "This paper introduces Tango, a reinforcement learning framework that jointly trains an LLM generator and verifier through interleaved RL updates. Unlike existing approaches that use fixed or discriminatively-trained verifiers, Tango employs a generative process-level verifier that is trained via RL alongside the generator. The key innovation is that the generative verifier learns to provide step-level feedback using only outcome-level supervision, eliminating the need for expensive step-level annotations. The framework creates a co-evolutionary dynamic where the generator produces increasingly diverse reasoning trajectories while the verifier learns to provide more accurate step-level judgments, leading to mutual reinforcement. Experiments on mathematical reasoning benchmarks demonstrate that both components achieve state-of-the-art performance among 7B/8B-scale models, with particularly strong improvements on challenging competition-level problems.", "questions": "Is there any reason to derive the final advantage as in (6)? Or is there a principle way to combine the advantage of different granularity?", "limitations": "Though the evaluation includes out-of-domain tests, the training is still Qwen on math, which has been proven to be noisy as we can not disentangle whether the effectiveness comes from the base model or the algorithm itself [1,2].\n\n[1] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025. \n\n[2] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- It may not be intentional but it reads like this paper is the first to propose “a novel framework that uses RL to concurrently train both an LLM generator and a verifier in an interleaved manner”, which is incorrect and an overclaim if it did not explicitly say “generative verifier”. As mentioned in the introduction, PRIME also uses RL to concurrently train both an LLM generator and a verifier in an interleaved manner, though it uses a deterministic verifier, which, in my opinion, is the biggest distinction between Tango and PRIME.\n- The method is straightforward, and the contribution mainly comes from engineering rather than novelty (I don’t mean it’s negative). As stated above, the biggest difference between Tangle and PRIME, the major baseline in this paper, is the adoption of generative verifier. However, the generative reward model is not new and has gained much attention to enhance verifier capability [1, 2]. Therefore, the method to replace the determinative verifier in PRIME’s framework with a generative one is straightforward. Nevertheless, making the recipe work still requires dedicated efforts, and the conclusion that generative verifiers work in training may still inspire future work along with improving supervision in RL, so this paper is meaningful in this sense.\n- The results show solid improvements over baselines and generalize outside math, and the ablation study validates (1) how the supervision signal improves along the co-evolution with the policy model, and (2) how the policy model benefits from mitigating reward hacking. Such results are meaningful in provoking a second thought on recent fever over RL with verifiable rewards.\n\n[1] Generative Reward Models. Mahan et al. 2024.\n\n[2] Process Reward Models That Think. Khalifa et al. 2025.", "was_revised": true, "final_justification": "The authors' clarification solves my concerns.", "rebuttal": "Dear Reviewer 5Vry,\n\nThank you for acknowledging the contributions of our work, particularly the significance of incorporating a generative verifier within a co-evolutionary training framework to advance RL supervision, the dedicated efforts required to make this recipe work in practice, the strong empirical performance and generalization beyond math, and the detailed and meaningful ablation studies. In the following, we address your concerns in detail.\n\n> ***It may not be intentional but it reads like this paper is the first to propose “a novel framework that uses RL to concurrently train both an LLM generator and a verifier in an interleaved manner” ... PRIME also uses RL to concurrently train both an LLM generator and a verifier in an interleaved manner, though it uses a deterministic verifier...***\n\n> ***... the major baseline in this paper, is the adoption of generative verifier. However, the generative reward model is not new and has gained much attention to enhance verifier capability [1, 2]...***\n\nThank you for pointing this out. We would like to clarify that the verifier in PRIME is a logit-based model trained via **supervised fine-tuning (SFT)** with a cross-entropy loss, rather than RL. In contrast, our framework uses RL to concurrently train both the generator and the verifier in an interleaved manner. The generative verifiers (reward models) in [1, 2] are also trained via **SFT**. Therefore, we believe it is accurate to state that our work is the first to propose a framework that uses RL to concurrently train both an LLM generator and a (generative) verifier in an interleaved manner. We will also make this point clearer in the revised paper.\n\nWe would also like to emphasize that this concurrent RL training of both components is not merely a swap-in replacement, but **central** to the success of our co-evolving framework. The effectiveness of such a system depends on both the generator and verifier being sufficiently capable and improving in tandem. If one component significantly lags behind (for example, if the verifier is trained only with SFT, which is known to generalize poorly in complex reasoning tasks [3]), it can disrupt the learning dynamics and hinder mutual progress. By contrast, RL enables both components to continually adapt to each other's evolving behaviors, which is essential for robust generalization and effective reasoning. That’s also the key reason why our framework outperforms PRIME.\n\nThat said, we are happy to revise “verifier” to “generative verifier” for clarity and precision, and we remain open to further rewording to avoid any potential ambiguity or overstatement, should any concerns persist.\n\n> ***The method is straightforward, and the contribution mainly comes from engineering rather than novelty (I don’t mean it’s negative) ... Nevertheless, making the recipe work still requires dedicated efforts…***\n\nWe are very delighted and grateful to see that the reviewer recognized and appreciated our contributions. However, we believe that novelty in deep learning research, especially in the LLM era, can manifest in various ways. Often, the most impactful advances arise not from introducing entirely new components, but from developing approaches that are particularly **well-suited to a specific context**, thereby enabling a major performance boost. Similarly, the novelty of our paper stems from recognizing that jointly training the verifier and generator is essential, and that such training *cannot* be effectively achieved with a verifier trained via SFT. Instead, RL is required for both components. (Our novelty is also acknowledged by Reviewer esdQ.) This design, while conceptually **simple**, is nontrivial to realize and proves **highly effective** in practice, leading to a substantial boost in performance. \n\n> ***Is there any reason to derive the final advantage as in (6)? Or is there a principle way to combine the advantage of different granularity?***\n\nThank you for the insightful question. Intuitively, Eq. (6) computes the advantage at the at the $t$-th token as the sum of normalized step rewards whose end-token indices (denoted by $I(k)$) are no smaller than $t$. This design aligns with the the process-level advantage computation used in GRPO by DeepSeek [4] (see Section 4.1.3).\n\nOur approach to combining advantages at different granularities, step-level and outcome-level, is defined in Eq. (7). We believe this provides a **principled** way to integrate multi-granularity advantage signals and has the potential for broader applicability.\n\nSpecifically, we first compute step-level and outcome-level advantages independently and then combine them using Eq. (7) with a scheduling coefficient $\\alpha$. We highlight two key design choices that are crucial to the success of our method:\n\n- We apply an exponential decay schedule to $\\alpha$, which is essential to TANGO’s success. Early in training, step-level supervision has a stronger influence to encourage exploration of reasoning strategies. As training progresses, we gradually reduce its weight to promote stable convergence and mitigate reward hacking.\n- Empirically, we find that computing and normalizing the step and outcome advantages separately before combining them yields significantly more stable learning than merging the rewards first and computing a single advantage. This is because advantage normalization depends on the scale and distribution of the underlying rewards. Merging step and outcome rewards before normalization could distort their relative contributions due to scale mismatch, resulting in instability and degraded performance. By normalizing each advantage independently, we preserve their intended effects prior to aggregation.\n\nWe will expand on the discussion above and include it in the revised paper.\n\n> ***the training is still Qwen on math, which has been proven to be noisy as we can not disentangle whether the effectiveness comes from the base model or the algorithm itself.***\n\nWe thank the reviewer for raising this important point. To better address the reviewer’s concern, we conducted additional experiments **using Llama-3.1-8B-Instruct as the base model**, following the same training/evaluation protocol as in Table 1. The results are presented in the table below.\n\n**Base Model: Llama-3.1-8B-Instruct**\n| Model            | MATH500 | AIME2024 | AIME2025 | AMC2023 | OlympiadBench | Avg. |\n|:-----------------|:-------:|:--------:|:--------:|:-------:|:--------------:|:----:|\n| TANGO-Llama3.1-8B-SFT     |  49.4   |   3.3    |   3.3    |  25.0   |     15.0       | 19.2 |\n| GRPO             |  56.2   |  10.0    |  3.3    |  35.0   |     20.9       | 25.1 |\n| **GRPO w/ TANGO**| **60.5**| **13.3** | **6.7** | **40.0**|   **23.6**     |**28.8**|\n\nThey show that even when Llama is used as the base model, TANGO continues to deliver significant improvements, demonstrating strong generalization across different model families. This further confirms that TANGO’s effectiveness **stems from** our interleaved RL co-evolving framework, rather than being specific to any particular model architecture. We will include the above results and discussion in the revised paper.\n\n---\n\n[1] Generative Reward Models. Mahan et al. Arxiv 2024.\n\n[2] Process Reward Models That Think. Khalifa et al. Arxiv 2025.\n\n[3] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. Chu et al. ICML 2025.\n\n[4] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Shao et al. Arxiv 2024.\n\n---\n\nThank you again for your time and feedback. We hope that our response has adequately answered your questions, and would lead to a favorable increase of the score. We are happy to discuss more if you have any further questions."}, {"review_id": "aLY5sSEesp", "reviewer": "Reviewer_esdQ", "summary": "Training language models with reinforcement learning on tasks with verifiable rewards (such as math and code) has emerged as a strong approach for improving reasoning in language models. Prior work typically relies on either a fixed verifier (e.g. rule based checks, trained reward model). The paper proposes an alternate paradigm where the verifier is a language models which is trained along with the generator. Specifically, the verifier and the generator are both trained with RL - the generator using feedback from the verifier and the verifier is trained using gold labels. Notably, even though the verifier provides step level feedback to the generator, it is trained only using the final label. Training the verifier and generator jointly can potentially mitigate reward hacking issues with fixed trained reward models and provide more fine-grained feedback than rule-based rewards. Empirical results show significant performance gains for the generator in the domain of math as well as generalization to some non-math domains. Additionally, even without process-level annotations the verifier demonstrate strong performance on ProcessBench. The authors also present some ablations to demonstrate the importance of joint training.", "questions": "* While reading the paper, I was reminded of two frameworks which I would be curious to hear the authors thoughts about:\n    - The setup seems quite similar to the work on prover-verifier games (e.g. [1]) where you have a similar setup. This has also been instantiated in the language model setting [2]. To me the major difference seems the generative nature of the verifier training with RL which can provide process level feedback and using natural language reasoning.\n    - Work on EM style training (e.g. [3]) where the \"generative model\" (i.e. M step) can be viewed as a verifier for the policy training (E-step). The difference from Tango is that the verifier in the EM case only provide the likelihood as the signal. \n* Currently, the verifier and the generator both use different models. I am curious if you have tried setting them to be the same, or for example having both models be of different capabilities to understand what role that plays?\n* For this setup, I believe an async training setup (e.g. [4,5]) could be quite useful to speed up training (even though authors say the computational overhead is not too high)\n* The text explanation from the verifier is not used at all. Maybe it can be used for inference time search?\n\n\n[1] Learning to Give Checkable Answers with Prover-Verifier Games. Anil et al. arXiv:2108.12099.  \n\n[2] Prover-Verifier Games improve legibility of LLM outputs. Kirchner et al. arXiv:2407.13692.\n\n[3] Amortizing intractable inference in large language models. Hu et al. arXiv:2310.04363.\n\n[4] Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. Noukhovitch et al. arXiv:2410.18252. \n\n[5] Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training. Bartoldson et al. arXiv:2503.18929.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Generative verifiers have been shown to be quite helpful for improving RL training of LLMs. However, as noted in the paper reward hacking, fine-grained credit assignment and generalizability are challenges with static fixed verifiers. So overall I think the idea of jointly training the verifier and generator is quite interesting and promising. Joint training can conceptually mitigate some of these issues by updating the verifier to be in step with the generator.\n* The paper also does a good job of clearly explaining the idea, and all of the details. \n* To the best of my knowledge, the specific instantiation of the idea is also novel within the context of RL with language models (but I acknowledge that this is a fast moving area so I could be missing something).\n* The method also seems to provide significant improvements empirically. The fact that the verifier can have such strong performance on ProcessBench without explicit process-level annotations is quite impressive. (Though as I discuss below the experiments can be improved)\n\nWeaknesses:\n* One major concern I have with the results presented in the paper is that they are limited to a Qwen2.5 base model. This model has some peculiarities [1] which can distort the evaluations. So it would be useful to have experiment on at least one model family (e.g. Gemma, Llama, OLMo, etc) to better understand the applicability of the method.\n* (minor) While I think the results are impressive, I find the choice of models in Table 2 and Table 3 a bit arbitrary. The models all have different base models which can have a significant impact on the results, so I think these tables should be discussed with a bit more nuance. \n* Finally, there no guarantees for the convergence of the joint training procedure. The authors mention that in their experiments it was not too sensitive. But I think a principle study could make the method even stronger.\n\nAside from the question of applicability with other base models I quite like the paper and lean towards accepting it. \n\n[1] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. arXiv:2506.10947.", "was_revised": true, "final_justification": "The authors added new experiments with different base models and addressed a lot of my concerns and comments. The paper is a strong contribution in the broad LLM + RL space.", "rebuttal": "Dear Reviewer esdQ,\n\nWe sincerely thank the reviewer for the thoughtful and detailed feedback. We are very delighted to see that the reviewer quite likes this paper and finds our idea novel and promising, our presentation clear, and our empirical performance strong. Below, we address the reviewer’s concerns in detail and provide additional clarifications where needed.\n\n> ***One major concern I have with the results presented in the paper is that they are limited to a Qwen2.5 base model … So it would be useful to have experiment on at least one model family...***\n\nWe thank the reviewer for raising this important point. To better address the reviewer’s concern, we conducted additional experiments **using Llama-3.1-8B-Instruct as the base model**, following the same training/evaluation protocol as in Table 1. The results are presented in the table below.\n\n**Base Model: Llama-3.1-8B-Instruct**\n| Model            | MATH500 | AIME2024 | AIME2025 | AMC2023 | OlympiadBench | Avg. |\n|:-----------------|:-------:|:--------:|:--------:|:-------:|:--------------:|:----:|\n| TANGO-Llama3.1-8B-SFT     |  49.4   |   3.3    |   3.3    |  25.0   |     15.0       | 19.2 |\n| GRPO             |  56.2   |  10.0    |  3.3    |  35.0   |     20.9       | 25.1 |\n| **GRPO w/ TANGO**| **60.5**| **13.3** | **6.7** | **40.0**|   **23.6**     |**28.8**|\n\nThey show that even when Llama is used as the base model, TANGO continues to deliver significant improvements, demonstrating strong generalization across different model families. This further confirms that TANGO’s effectiveness **stems from** our interleaved RL co-evolving framework, rather than being specific to any particular model architecture. We will include the above results and discussion in the revised paper.\n\n> ***I find the choice of models in Table 2 and Table 3 a bit arbitrary. The models all have different base models which can have a significant impact on the results, so I think these tables should be discussed with a bit more nuance.***\n\nThank you for your suggestion. The purpose of Tables 2 and 3 is to position our results in the broader context of existing works and to demonstrate that TANGO-trained generators and verifiers remain highly competitive **regardless of base models**. For this reason, these tables include results from a variety of model families. However, we agree with the reviewer that more nuanced discussions would help readers better understand the performance of both our method and the baselines. In the revised paper, we will indicate the base model used for each method in Tables 2 and 3, and include additional discussions in the main text.\n\nAdditionally, we note that in the paper we already include controlled experiments where the base model is fixed (Table 1 and Figure 3). Furthermore, as discussed above, we have conducted additional experiments using models from the Llama family, further confirming that TANGO’s effectiveness is not tied to any specific model architecture. We will make these points more explicit in the revised paper.\n\n> ***there are no guarantees for the convergence of the joint training procedure…I think a principle study could make the method even stronger.***\n\nThank you for raising this insightful point. Theoretical convergence guarantees for co-evolving agents, such as our generator and verifier, remain an open challenge, especially in large-scale RL with LLMs.\n\nThat said, we observe stable and consistent training across diverse settings. As shown in Figure 1 and Section 4.1, TANGO improves both training efficiency and final performance. Our ablation in Section 4.3 further shows that co-evolving both components yields significantly better results than fixing either one, supporting the stability of our interleaved optimization.\n\nOur setup resembles multi-agent RL (MARL) and two-timescale optimization, where both agents use policy gradients and mutually influence each other’s reward landscape. Theoretical tools from [6] may serve as a foundation to analyze TANGO’s convergence with suitable extensions for text-based feedback. While a complete convergence analysis remains beyond the scope of this work, we leave this as a promising direction for future work.\n\n> ***Comparison with Prover-Verifier Game [1,2] and EM-style training [3].***\n\nWe thank the reviewer for the inspiring question. Compared to the **Prover-Verifier Game** [1,2], our approach differs in **two key aspects** among a few others:\n\n- Their verifier is trained using a supervised classification loss based on sneaky and helpful examples, whereas our verifier is optimized through RL; \n- Their verifier outputs a single scalar score that contributes to the outcome-level reward used in PPO training of the prover. In contrast, our verifier generates natural language step-level judgments, enabling fine-grained process-level rewards that more effectively guide the generator’s training.\n\nRegarding the second question, it is intriguing to draw connections between our proposed TANGO framework and **EM-style training** as explored in [3]. In that work, the LLM reasoning problem is formulated as sampling from the posterior $p(Z \\mid X, Y)$, where $Z$ denotes the reasoning chain, $X$ the prompt, and $Y$ the outcome. Because this posterior is intractable, it is approximated via a GFlowNet in the E-step [3]. At a high level, we acknowledge the conceptual similarity noted by the reviewer. However, there are **two fundamental technical differences** that distinguish the two approaches:\n\n- While both involve an LLM-based generator, [3] introduces a GFlowNet specifically to approximate $p(Z \\mid X, Y)$, a unique modeling choice. In contrast, TANGO incorporates a text-based generative verifier trained to evaluate the generator’s outputs, rather than to sample reasoning chains $Z$, thus serving a fundamentally different purpose;\n- The framework in [3] is grounded in the EM algorithm, optimizing expected log-likelihood. TANGO, on the other hand, is based on RL, directly optimizing expected return under both outcome-level and process-level rewards.\n\nWe thank the reviewer for pointing out these connections, and we will cite and discuss these relevant works in the revised paper.\n\n> ***Currently, the verifier and the generator both use different models. I am curious if you have tried setting them to be the same, or for example having both models be of different capabilities to understand what role that plays?***\n\nThank you for the constructive feedback. In our main math experiments, we use Qwen2.5-Math-7B as the generator and Qwen2.5-7B as the verifier, primarily because the verifier requires a larger context window to accommodate both the input question and the generator’s full output. In contrast, for the algorithmic reasoning experiments and the Llama experiments discussed above, we use Qwen2.5-1.5B or Llama-3.1-8B-Instruct for both the generator and verifier, thereby exploring the scenario where both components share the same base model. This design allows us to evaluate TANGO’s effectiveness under both settings.\n\nGenerally speaking, there is no strict requirement on whether the generator and verifier should share the same base model or not. However, several considerations can guide the choice: the generator should possess reasonably good reasoning capabilities to start from, while the verifier should have a sufficiently large context window to process the full generator response for accurate step-level evaluation. Additionally, to ensure a fair comparison, we typically choose the verifier such that its reasoning ability is *not stronger* than that of the generator, in order to prevent potential “distillation” effects from the verifier to the generator. We will also make this point clearer in the revised paper.\n\n> ***For this setup, I believe an async training setup (e.g. [4,5]) could be quite useful to speed up training.***\n\nThank you for your valuable suggestion. We agree with the reviewer that asynchronous training is a promising direction for improving training efficiency. We plan to explore this in future work to further speed up the training process of TANGO. In the revised paper, we will also explicitly discuss the potential of using asyn training and cite [4,5] for context.\n\n> ***The text explanation from the verifier is not used at all. Maybe it can be used for inference time search?***\n\nThank you for your thoughtful suggestion. Yes, the generative process-level verifier (PRMs) trained by TANGO is indeed **well-suited** for inference-time scaling. Specifically, it can assign process-level scores to generator outputs, enabling inference-time search strategies such as Best-of-N selection and tree-based search. Moreover, unlike discriminative reward models, the generative verifier uniquely supports simultaneous scaling of both generator and verifier compute. By sampling multiple verification trajectories, we can improve verification accuracy, thereby providing more reliable supervision to the generator. That said, as this work primarily focuses on RL post-training, we reserve this exploration as future work.\n\n---\n\n[1] Learning to Give Checkable Answers with Prover-Verifier Games. Anil et al. Arxiv 2021.\n\n[2] Prover-Verifier Games improve legibility of LLM outputs. Kirchner et al. Arxiv 2024.\n\n[3] Amortizing intractable inference in large language models. Hu et al. Arxiv 2023.\n\n[4] Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. Noukhovitch et al. Arxiv 2024.\n\n[5] Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training. Bartoldson et al. Arxiv 2025.\n\n[6] Multi-agent reinforcement learning: A selective overview of theories and algorithms. Zhang et al. Handbook of reinforcement learning and control (2021).\n\n---\n\nWe hope that our response has addressed all of your concerns, and that you may consider a favorable increase of the score. Please do not hesitate to discuss with us if you have any other comments."}]

Author rebuttal:

Meta-review: This paper studies joint RL training of generator and verifier LLMs in reasoning tasks, i.e., ones where a latent token sequence intervenes in the generation of the answer given the query. The generator receives trajectory-level rewards for the correctness of its answer and step-level rewards from the verifier. The verifier, which also uses latent reasoning, receives a reward for correctly classifying the generator's reasoning as correct or incorrect. Various techniques are proposed to enable the joint optimisation of the two models. Evaluations on a variety of tasks (mainly maths benchmarks) show a consistent improvement over baselines.

The main reasons to accept, all mentioned by multiple reviewers, are (1) the substantial performance improvements given by the proposed algorithm, (2) the interesting effect of the verifier's latent reasoning promoting explainability and mitigating reward hacking, (3) the diversity of tasks considered and the demonstration of generalisation abilities outside of maths tasks.

The authors' questions were almost entirely addressed in the rebuttal. Several reviewers note that the main contributions are empirical and the conceptual novelty is not very high. However, the reviewers unanimously give a rating of Accept, and I follow their recommendation. The authors are encouraged to take account of the reviewers' suggestions when preparing the final version.
Decision: Accept (poster)