Generative Verifiers: Reward Modeling as Next-Token Prediction

Authors: Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal
Primary area: foundation or frontier models, including LLMs
Keywords: LLM reasoning, reward models, verifiers

Abstract:
Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional test-time compute via majority voting for better verification. We demonstrate that GenRM outperforms discriminative, DPO verifiers, and LLM-as-a-Judge, resulting in large performance gains with Best-of-N, namely 5% → 45.3% on algorithmic tasks, 73% → 93.4% on GSM8K, and 28% →44.6% on easy-to-hard generalization on MATH. Furthermore, we find that training GenRM with synthetic verification rationales is sufficient to pick out subtle errors on math problems. Finally, we demonstrate that generative verifiers scale favorably with model size and inference-time compute.

Decision: accept (poster)
Reviewer scores: 5, 8, 3

Reviews and discussion:
<thread>
creation date:
1729969988105

thread_type:
official_review

<thread_object>
creator: reviewer_odxm

<content>
summary:
The paper proposes a way to automatically verify whether a solution generated by a LLM is correct or not. Instead of replying on a third-party verifier which ranks multiple solutions (as the best-of-N method), it suggests the verifier should use a CoT step, followed by a majority voting, to determine the correctness probability of the solution.

strengths:
An easy to understand method that works in the few experiments of the paper.

weaknesses:
I have several concerns about this paper. First, I don't think such a process can be called a "verifier", as there is no rigor in the entire process. Especially, we do not have any guarantee on the final probability value. It completely relies on the quality of the other LLM to evaluate the solution, and as mentioned at the beginning of the paper, " ... often confidently make logical and factual mistakes ". I understand this is what the community is doing, but on the other hand, this paper does not make any progress on this aspect. 

Second, the novelty, i.e., utilising the autoregressive ability rather than ranking, is not significant. The contribution is very incremental. 

Third, I suppose the utilisation of another LLM will not always lead to positive impact. The study of its potential negative impact (e.g., the propagation of the vulnerabilities of multiple LLMs) may be more interesting than what presents in the paper. 

Finally, there is no comparison (and no discussion) with the methods on uncertainty estimation, which generates a confidence when producing a solution. I don't see this "verifier" methodology offers significantly more than an uncertainty estimator. 

In summary, I found the method proposed in the paper is incremental, without tackling the major issues (guarantees, negative impacts, etc) of this problem.

questions:
see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643783115

thread_type:
official_review

<thread_object>
creator: reviewer_24yj

<content>
summary:
The authors investigate training LLMs to act as verifiers using a generative objective (training the LLM to verify if a solution is correct by directly predicting the yes/no token). Notably, they investigate the implications of this modification for scaling inference compute and for jointly optimizing solution generation and verification.

strengths:
The authors examine a very relevant and important problem of learning good verifiers for LLM generations. 

They present a new method that is well-motivated (increasing verification compute and framing verification to match the original LLM objective).

The authors conduct a lot of key experiments exploring these dimensions, the paper is explained clearly and is very easy to understand.
- In particular, it’s great to see experiments measuring the generation performance change as well as experiments investigating the scaling properties of more verification inference compute.

weaknesses:
In many of the plots (figure 1, 4, 5, 6), the y-axis scaling changes from plot-to-plot and is often very restricted (ie. sometimes spanning only 4%). This is misleading when comparing results, and it would be great to standardize it more.

GenRM does improve over the baselines (it seems like more on harder tasks which is worth highlighting more!) but a lot of times the improvement is relatively small (ex. 1% for gsm8k over discriminative).

"In Figure 8, we show that generative verifiers, especially GenRM-CoT, exhibit
better scaling behavior than discriminative RMs,"
- Unclear from figure 8 that generative verifiers scale better -> the boosts are very similar for the discriminative RM.

Although the authors explore a lot of baselines which is great, there are some key verification methods that are missing. Specifically process reward models are generally better than the ORMs studied, which is important given that the performance of ORMs in some settings are close to GenRM.

Verification is only done on a max of 32 generated solutions. Although this is done in many other past works, given that the best-of-n performance scales to thousands of samples for some datasets it would be great to see the scaling properties along this dimension.

I am willing to raise my score if some of these concerns are addressed! In particular, the presentation of results and/or adding stronger baselines.

Nits:
- Figure 4, the color is wrong for GenRM

questions:
In all evaluations are the actual generations are the same, the only difference is the verifier in each method? I want to confirm that you aren’t using the fine tuned generator for GenRM (it would be great to make this clear in the text).

Did you try any PRMs? How do they compare?

Did you try verifying sample collections larger than 32?

Couldn’t reference guided rational training introduce a train/test mismatch? Ie at training the verifier objective is conditioned on a correct answer, but it isn’t at test time?

Are the CoTs faithful? Ie. is the reasoning for yes/no accurate to the actual problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693559444

thread_type:
official_review

<thread_object>
creator: reviewer_8gsi

<content>
summary:
This paper proposes Generative Verifiers (GenRM), a novel framework for verification in large language models (LLMs), which reframes reward modeling as a generative task. Specifically, the authors introduce GenRM and GenRM-CoT (Chain-of-Thought), where GenRM-CoT incorporates additional reasoning steps. They aim to improve verification by using the model (Gemini 1.0 Pro) for both solution generation and generating synthetic verification rationales and then train Gemma open source 2B, 7B, 9B models. Experiments are conducted on GSM8K, MATH and algorithmic tasks to demonstrate the effectiveness of this approach over discriminative reward models.

strengths:
1. Innovative Approach: Reframing verification as a generative task, specifically with GenRM-CoT, is novel and shows promise for complex reasoning tasks.
2. Synthetic Rationale Generation: The use of the same model to generate both solutions and synthetic rationales offers a more streamlined and potentially scalable verification process.
3. Improved Performance: Results indicate that GenRM-CoT improves upon discriminative reward models, especially when using chain-of-thought (CoT)/ScratchPad [1] reasoning and majority voting.

weaknesses:
Scientific Reservations  
1. Limited Mathematical Task Scope: The reliance on GSM8K and limited algorithmic tasks raises concerns about generalizability. These datasets represent only basic levels of math reasoning (grade school and high school). Including results from more rigorous benchmarks, such as the IMO portion of OlympiadBench [2] or math subsets of MMLU, would strengthen the claims.
2. Over-Reliance on Proprietary Model (Gemini 1.0 Pro): By using Gemini 1.0 Pro to generate solutions and rationales for training smaller Gemma models, the paper introduces a dependency on proprietary resources, which might limit reproducibility. Showing results on more accessible, open-source models would be essential to add credibility.
3. Toy Nature of Algorithmic Tasks: The algorithmic tasks feel limited and not representative of real-world complexity. Including a more robust task, or additional toy tasks for variety, would better support the general claims.

Writing Reservations  
1. Inconsistency in Figures and Text: Figure 4 uses inconsistent colors (e.g., GenRM in blue but lines are cyan/green), which makes interpretation challenging. Additionally, the reported improvements (e.g., 73%-93.4% in the introduction vs. 16%-40% in the abstract) should be unified to avoid reader confusion.
2. Notation and Explanation Gaps: Section 3.1’s notation (e.g., inconsistent usage of x, y) creates confusion and requires more clarity. Specific variables, like I, need explicit definitions or cross-references to earlier sections to ensure readability.
3. Incomplete Background: Key concepts, such as "reference-guided grading" and "LLM-as-a-Judge," are insufficiently explained, causing unnecessary interpretive burden. Adding a background section for these terms, or moving some non-essential related work to the appendix, could improve clarity.
4. Confusing Terminology: The paper should clarify that “CoT Verifiers” refers to CoT reasoning in the verification process, not the solutions themselves, which also contain CoT. Renaming these methods would reduce ambiguity.
5. Inference and Training Separation: The distinction between training and inference (lines 211-241) is blurred. Separating these sections would make the methodology clearer.
6. Inconsistent Use of Majority Voting: The term "majority voting" implies selecting the most frequent result, yet the paper uses an averaging approach. Clarifying this terminology would prevent misunderstanding.

questions:
Suggested Improvements  
1. Broader Mathematical Validation: To strengthen the scientific claims, I suggest including results from more advanced math reasoning benchmarks, such as IMO tasks from OlympiadBench or relevant subsets of MMLU. Results from these additional benchmarks could significantly boost the paper’s credibility. BigBench has an induction and Identify Math Theorem https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/identify_math_theorems 
2. Justification for Larger Model Use: The reliance on a larger model (Gemini 1.0 Pro) to generate training data for smaller models needs a sound explanation, as it impacts reproducibility. Without this, the setup may seem biased.
3. CoT-Only Baseline: To isolate the effect of CoT reasoning in verification, a baseline experiment using CoT reasoning alone without verification reasoning steps could help confirm the added value of GenRM-CoT.
4. Consider Extending to Other Models: Testing GenRM-CoT on open-source models would help show that the approach generalizes beyond the proprietary Gemini/Gemma series.
5. Length Generalization: Generalizing to shorter problem lengths is not particularly noteworthy, as longer problem lengths often include shorter steps. Showing robustness across various task lengths would be more convincing.

I'm willing to revise if points addressed well.

citations
[1] Scratch pad: https://arxiv.org/abs/2112.00114 (cite too, they created CoT concurrently)
[2] OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems https://arxiv.org/abs/2402.14008
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. The method appears to be novel and experimental results outperforms DPO verifiers, and LLM-as-a-Judge. 

Strength: A novel approach to build LLM-based verifiers and experimental results had some good improvements over existing methods.
Weakness: 1. Some of the experiments could be more comprehensive to include more tasks. 2. Models are based on private models and could be difficult to replicate.