Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Authors: Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, Semih Yavuz
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Verifier, Math Reasoning, LLMs

Abstract:
Despite significant advancements in the general capability of large language models (LLMs), they continue to struggle with consistent and accurate reasoning, especially in complex tasks such as mathematical and code reasoning. One key limitation is that LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors, which hampers their ability to reliably verify and rank outputs. To address this, we scale up the inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness. To facilitate this, we introduce a comprehensive dataset consisting of correct and incorrect solutions for math and code tasks, generated by multiple LLMs. This diverse set of solutions enables verifiers to more effectively distinguish and rank correct answers from erroneous outputs. The training methods for building verifiers were selected based on an extensive comparison of existing approaches. Moreover, to leverage the unique strengths of different reasoning strategies, we propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification. CoT provides a clear, step-by-step reasoning process that enhances interpretability, while PoT, being executable, offers a precise and error-sensitive validation mechanism. By taking both of their strengths, our approach significantly improves the accuracy and reliability of reasoning verification. Our verifiers, Math-Rev and Code-Rev, demonstrate substantial performance gains to existing LLMs, achieving state-of-the-art results on benchmarks such as GSM8k and MATH and even outperforming GPT-4o with Qwen-72B-Instruct as the reasoner.

Decision: reject
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730366221366

thread_type:
official_review

<thread_object>
creator: reviewer_ajnz

<content>
summary:
This paper presents a new method for verifying LLM reasoning by assessing the consistency between CoT and PoT solutions.The proposed approach filters each CoT solution, ensuring alignment with its corresponding PoT version. Additionally, a trained verifier is employed to evaluate and select from the filtered CoT solutions. Experimental results demonstrate that this method achieves state-of-the-art performance on both GSM8k and MATH benchmarks.

strengths:
1. The proposed verification framework, which combines CoT and PoT, is very intuitive and promising. 
2. Some empirical results are resource-intensive in terms of time and GPU usage, but could be valuable to the research community. 
3. The performance of the proposed method appears very good and achieves state-of-the-art performance on GSM8K and MATH.
4. The authors have committed to releasing both the code and data, which will be beneficial for reproducibility and further research.

weaknesses:
1. The technical contributions of this paper are somewhat limited. For example,  the introduction of a verifier has already been highlighted in [1]; Comparing the CoT reasoning with its formal version have also been explored in [2, 3]. 
2. It is challenging to assess the actual effectiveness improvements resulting from the proposed method. The improvement of collaborative verfication is relatively marginal compared with the SimPO (see Table 1). However, as to the data generation for the verifier, six models (Mistral, Phi3, InternLM2-Math, MAmmoTH2-plus, LLaMA-3-8B, and GPT-4o) are involved. Simply merging these models' knowledge, even without verification, could also enhance the reasoning performance of the LLMs.
3. There is a notable performance imbalance between CoT reasoning and PoT reasoning.This discrepancy may hinder the application of the proposed method in more difficult benchmarks, e.g., AI Mathematical Olympiad. Particularly, CoT exhibits significantly lower accuracy compared to PoT, which could undermine the effectiveness of collaborative verification.

[1] Let’s verify step by step. ICLR 2024.

[2] Don’t trust: Verify–grounding llm quantitative reasoning with autoformalization. ICLR 2024.

[3] Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. ICLR 2024.

questions:
Please also refer to the weakness section.

1. What is the performance improvement achieved by fine-tuning the base model on the generated dataset?
1. How does the performance curve change with varying sizes of sampled solutions?
1. It seems that CoTnPoT sacrifices some recall, which may degrade the upper bound of LLM reasoning performance. Are there any potential solutions to address this issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730500004006

thread_type:
official_review

<thread_object>
creator: reviewer_r5wh

<content>
summary:
This paper proposes two ways to improve LLM reasoning performance via verification. The first way is to train reward models / verifiers on correct and incorrect solutions. The reward models can either be trained with simple outcome-based or preference based. The second way is to filter solutions when the COT solution and PoT solution answer do not disagree with each other. The paper shows that by using these verification methods, the performance can be improved considerably on math and coding datasets.

strengths:
The paper tackles an important topic, improving LLM reasoning, through verification based approach. The motivation makes sense and the authors study different options for training the verifiers. The filtering of comparing if CoT and PoT agree is also reasonable to do.

weaknesses:
There are a few concerns I have for the proposed approach and settings.

1. For code reasoning, many existing work assumes we have access to some if not all test cases for the problem. Because of this, pass@k is typically used to evaluate model performance. I wonder if the verification setting for code reasoning makes sense. Have the authors tried to compare their approach with verification based on say 50% of the test cases?

2. For code reasoning, what is the point of doing CoTnPoT if the problem we are solving is already a coding problem? Also I generally have a hard time understanding equation (2) and the text around it. The LHS is S_Des while the text mentions "we concatenate the description and the code as an integrated input for the verifier..."

3. While CoTnPoT is nice to help and always seems to help, it essentially doubles the amount of inference time compute which makes verification much heavier now (see Equation 1). For a fair comparison, CoTnPoT should be compared against the baseline of sampling twice the number of solutions as before. Even so, it seems that SimPO is already very strong (see Table 1) and CoTnPoT may not be necessary.

questions:
Please see Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682515693

thread_type:
official_review

<thread_object>
creator: reviewer_fhq1

<content>
summary:
The paper presents a number of contributions to improve LLM reasoning:
- First, they create a dataset of math and code reasoning tasks used to train verifiers
- Next, they train various verifiers and compare their performance
- Finally, they use these verifiers to improve performance on math tasks.

strengths:
- The method is simple and effective, while providing improvements above the baseline model.
- There is good analysis on the success of the verifier (Table 4), which provides evidence for why the verifier 
- The paper provides relevant ablations (Section 3.4) that highlight the effectiveness of various components of their approach.

weaknesses:
- The approach seems to be applicable generally for reasoning domains, but the authors only test on math reasoning domains. The paper would benefit from an assessment on another domain as well. 
- While there are improvements above the baselines, they vary dramatically between models
- The paper could benefit from more analysis on what is learned by the verifiers, for example if there are certain types of solutions where the verifiers perform better?

questions:
- To what extent can the technique be applied to other domains? 
- Are there certain types of programs/solutions where the verifier is better at judging?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730839062709

thread_type:
official_review

<thread_object>
creator: reviewer_dnmd

<content>
summary:
The paper provides a dataset of correct and incorrect responses for mathematical and coding problems, which are generated by multiple large language models (LLMs). This dataset is used to train verifiers to judge the correctness of candidate solutions from models at test-time. The verification is done on both chain-of-thought (CoT) and program-of-though (PoT) candidate solutions for a problem. The trained verifiers for math and coding tasks show improvements on GSM8K, MATH500 and MBPP.

strengths:
The compilation of a large dataset containing both correct and incorrect candidate solutions from multiple LLMs, covering math and coding tasks, offers a valuable resource for training solution verifiers.

The CoTnPoT verification strategy improves accuracy by leveraging the complementary strengths and synergies of both CoT and PoT formats, incorporating insights that each provides.

The paper effectively explores a range of verifier training methods, including outcome-reward models (ORM) classifiers and various preference optimization techniques. The experiments, conducted on several models, provide valuable insights into the effectiveness of these approaches and their impact on verification performance.

The authors perform rigorous verification experiments with several generators, including weighted majority voting and thorough hyper-parameter search.

weaknesses:
The paper lacks significant novelty; the primary differences lie in the volume of data, the variety of sources, and minor variations in training (and verification with both CoT and PoT candidates). The tone in the abstract and introduction suggests that the approach is a bit more groundbreaking than it actually is, particularly regarding verification and inference-time compute scaling. For instance, the paper states "To address this, we scale up the inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness." However, this approach is already an established direction in the literature. It would be more appropriate to position this as a current method for scaling inference-time compute, rather than framing it as a novel solution.

Another instance of this is “We develop inference-time verifiers to enhance the reasoning capabilities of LLMs”. This item may not be suitable for inclusion in the contribution list, as similar work has already been completed in the past.

The CoTnPoT method relies on PoT generation from an external LLM, but such code-based solutions may not always be feasible or reliable. Additionally, if the proposed method generates K more (code) solutions per query, it would be appropriate/fair to compare other methods with a best-of-2K budget to ensure a fair comparison. Similarly, for code problems, there are extra description tokens generated by the language model. For a fair comparison, other methods should have access to a similar budget for generation of candidate solutions.

The distinction between the CoTnPoT approach for math tasks and coding tasks could be clarified further in the paper. For example, Figure 3 explains the method only in the context of math reasoning problems, but it is unclear how this approach adapts to coding tasks, especially given that there is no final answer in coding problems. 

There are some grammatical or dictation errors in the paper. For instance, on line 219, I believe the authors meant to write preference-tuning instead of reference-tuning. 

The phrase "and surpassing the performance of existing verifiers" in line 128 seems grammatically awkward and could be omitted without losing meaning.

questions:
Here are some of my questions and comments other than the ones mentioned above: 

If i understand correctly "recall rate" in Figure 1 is basically Pass@64. If that’s the case, it would be clearer to simply use the more widely recognized Pass@64 term.

Similarly, the "sample-then-rank" approach could be replaced with the more widely used "best-of-N" sampling strategy.

In section 3.2 it is mentioned that "We upgraded the backbone model of our verifier in math reasoning
from Mistral-7B to MAmmoTH-7B-plus to enhance performance.". Why not use MAmmoTH-7B-plus for the previous sections of the paper?

Figure 2 is misleading, one should compare Maj@k of models (without verifiers) with verification-based approaches (models with verifiers). I don't believe this is mentioned in the paper.

Line 256 states "Because using the same LLMs for both code and description generation reduces over-reliance on external LLMs". However, for math reasoning problems, the proposed method is in fact using and relying on external LLMs for generation of $S_{PoT}$. Doesn't this seem contradictory?

It would have been better if the number of candidate solutions $k$ in experiments for performance of different verifiers was consistent. Figure 4 shows some models have been tested with 100 and some with 64 candidate solutions.

Line 82 states "In other words, these verifiers are not generally applicable across different tasks and backbone reasoners". However, the proposed method and trained verifiers are also based on a few backbones. How is the proposed verifier more general?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731520167685

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We sincerely thank each reviewer for the thoughtful and constructive feedback, as well as the time and effort invested in evaluating our paper. Your insights have been invaluable in helping us refine and improve our work. Through your questions and comments, we gained a deeper understanding of areas for enhancement, which has allowed us to make meaningful revisions that we believe have strengthened our paper considerably.

We respectfully invite you to review our itemized responses and would be most grateful if you could share any further questions or concerns in the comments. We are fully committed to addressing each point and provide additional clarifications if needed. If our responses have satisfactorily resolved your initial concerns, we would greatly appreciate it if you might consider an update to the evaluation score.

Thank you once again for your valuable feedback and guidance, which have been instrumental in enhancing the quality of our work.

Best Regards,
Authors of ICLR 5271
</content>
</thread_object>

</thread>


Metareview: