Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solver

Authors: Zhenting Qi, Mingyuan MA, Jiahang Xu, Li Lyna Zhang, Fan Yang, Mao Yang
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Reasoning

Abstract:
This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51\% to 63.91\% for LLaMA2-7B, from 36.46\% to 81.88\% for Mistral-7B, from 74.53\% to 91.13\% for LLaMA3-8B-Instruct. Code is available at https://github.com/zhentingqi/rStar.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730181679785

thread_type:
official_review

<thread_object>
creator: reviewer_ldgj

<content>
summary:
This paper tackles the challenge of improving small language models' reasoning abilities without fine-tuning or larger model supervision. The key innovation is a two-phase approach called rStar: first, a generator phase uses Monte Carlo Tree Search with an expanded set of reasoning operations (like decomposing problems, rephrasing questions, and proposing intermediate steps) to create potential solution paths. Then, a discriminator phase uses a second small language model to verify these solutions through "mutual consistency" (checking if the model can arrive at the same conclusion given partial steps). The approach is notable because it improves performance through better inference-time decision making rather than parameter updates or knowledge distillation from larger models. The empirical results demonstrate substantial improvements across multiple reasoning benchmarks and model sizes, suggesting that smaller language models have more latent reasoning capability than previously thought, but need better mechanisms to access it. The authors validate their approach through extensive ablation studies showing the importance of both the expanded MCTS action space and the mutual consistency verification.

strengths:
The paper's primary originality lies in its novel two-phase approach to improving small language model reasoning. Rather than relying on conventional methods, it creatively combines an enriched Monte Carlo Tree Search with a mutual consistency verification phase using a second SLM. Particularly innovative is the expansion of the MCTS action space to include human-like reasoning operations such as problem decomposition, question rephrasing, and stepwise thinking. The proposed "mutual consistency" verification approach, while used before, is an interesting application that seems very effective.

The authors conduct a comprehensive evauation across a diverse set of benchmarks and models. Additionally, they perform convincing ablation studies that validate their key design choices, particularly the importance of both the expanded action space and mutual consistency verification. The provided baselines are also good.

The majority of the paper is well written and clearly expressed.

The significance of this work is particularly noteworthy. It demonstrates that SLMs possess stronger latent reasoning capabilities than previously believed and provides a practical method for improving SLM reasoning without requiring expensive fine-tuning or supervision from larger models.

weaknesses:
The description of rStar in the introduction, starting on line 71, is hard to follow. Perhaps breaking the algorithm down into bullet points would help make the process more explicit and easier to digest than the wall of text. Moving figure 3 to the beginning would also be effective.

On line 255 you say "based on whether it reaches the correct answer" (which would imply you're using the ground truth answer during your search), but it seems like it is simply based on the likelihood of self-consistency majority voting mentioned on line 259.

It seems one potential limitation of mutual reasoning consistency is if an early action makes an incorrect statement that dramatically simplifies the problem. In this case it is likely that $SLM_2$ matches $SLM_1$. Given that this method works so well, this clearly isn't a critial issue, but certainly worth addressing/exploring more (at the very least mention this potential limitation).

Comparisons to other MCTS methods would be nice to have. A quick google search found (on top of the couple cited in the paper) "Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning" and "Improve Mathematical Reasoning in Language Models by Automated Process Supervision". How do these approaches performance compare to yours?

questions:
- When does performance improvement drop off with increased roll outs? The paper stops at 32, but performance seems to still be improving linearly.
- How does this approach compare to other MCTS methods recently proposed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592298705

thread_type:
official_review

<thread_object>
creator: reviewer_hxm4

<content>
summary:
The paper introduces rStar, a self-play mutual reasoning approach designed to improve the reasoning capabilities of small language models (SLMs). This method enhances SLMs with prompting engineering. The key mechanism involves a generation-discrimination process where the target SLM creates reasoning trajectories using Monte Carlo Tree Search (MCTS) enriched with human-like reasoning actions. Another SLM, with similar capabilities, acts as a discriminator to verify the generated trajectories, ensuring they are mutually consistent, which increases their likelihood of correctness. Experiments show that rStar effectively boosts performance on challenging reasoning benchmarks.

strengths:
The idea of introducing human-like reasoning actions and using mutual consistency between SLMs to verify the results during inference time is intriguing. The writing is well-structured and clear. Empirical results with details are given.

weaknesses:
There are still some design and implementation details related to manual action selection, self-reflection, token cost, scalability to larger models that need further clarification.

questions:
1) The authors introduce a set of five human-like reasoning actions as the action types for MCTS. This design requires manual selection and experimental validation, which may not be optimal. Could the authors provide any experiments or analysis comparing their manually selected actions to other potential action sets? Did the authors consider automating the action type design process?


2) A very interesting aspect of OpenAI's o1 model is its self-reflective behavior. Did the authors consider integrating self-reflection as an action type in their framework? What potential benefits or challenges do the authors expect with such an addition?


3) The authors mention that temporal order constraints exist for different action types. Could the authors provide pseudocode or a more detailed explanation of how the temporal order constraints are implemented in their MCTS algorithm?


4) In Section A.3, the authors mention the high token cost associated with this method. For instance, the average number of generated tokens per question on GSM8k is 367.1k, which could limit the method's practical applicability. Have the authors considered optimization strategies to address this issue? While distributed inference can reduce processing time, it does not reduce the overall computational cost.


5) The authors emphasize that this method is designed for SLMs. Have the authors conducted experiments or analysis comparing rStar's performance on SLMs versus on LLMs? What’s the expectation of the method's effectiveness to change with model size?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654015441

thread_type:
official_review

<thread_object>
creator: reviewer_uyvd

<content>
summary:
The paper describes a method to improve "reasoning capabilities" of small language models. This is done by generating trees of prompt using MCTS and by using a second small language model that "critiques" MCTS rollouts. After introducing the method the paper than show in its experimental section that on a suite of benchmarks the new method, dubbed rStar, outperforms existing methods.

strengths:
- The paper is relatively well written making it easy to follow..
-  the methodology is clearly explained and differences as well as similarities to competing methods are made explicit.
- In the experimental evaluation it is clearly shown that the proposed method out-competes state-of-the-art approaches. This is substantiated by a couple of informative ablation studies.

weaknesses:
The paper claims to improve reasoning capabilities of small language models. However, by training/augmenting a SML with MCTS on a specific dataset we now end up with a model that is informed by the statistics of the dataset. In the end there is no reasoning happening but the proposed method allows the SML to better exploit statistical patterns in the data-query pairs that it was trained on.

In the end the proposed methods does not allow SMLs to reason but is a method to perform prompt-engineering in an automated fashion using the statistics of the benchmark in question.


One of the problems I have is the anthropomorphism present in the paper. Specifically, with using "rich set of human like reasoning actions" already in the abstract and continuing it throughout the paper.


While I do not see how the paper enables SMLs to reason better, I can see that the introduces techniques have clear experimental advantages over competing methods. This leads me to tend towards arguing towards accepting the paper ever so slightly. .

questions:
Reasoning capabilities are often studied in terms of generalizability How would you study generalization capabilities of your method? Could you transfer between tasks/benachmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755425857

thread_type:
official_review

<thread_object>
creator: reviewer_ain4

<content>
summary:
The paper puts forward an approach to improve reasoning capabilities of LLMs. The approach is based on MCTS with a discriminator model selecting the most promising answers. Experiments appear to demonstrate superior reasoning capabilities of the approach compared to fine-tuning and other approaches.

Post-rebuttal: My expertise in the area is limited to the one of an interested observer. I am reassured by the responses of the authors to my questions and having read through the related links and explanations, I am happy to revise my score up and recommend acceptance. I would encourage the authors to add explanations to points such as the ones I have highlighted to increase the reach of the paper to people outside the LLM area.

strengths:
The experimental results point to much improved performance of the model under reasoning.

weaknesses:
This is not my area, hence low confidence, however as someone outside the LLM area I struggled to see the arguments put forward in the paper. I recommend the author to put more emphasis into making the key parts of the technical material more accessible by providing more explanations. Here are the key problems I had.

1 I did not understand how the MCTS approach works here both in terms of generation of the next steps. What simulation is performed, how this is tuned and where the MCTS itself results from (is it the model itself). Equally I found no explanation as to why this would/should lead to increased performance in the step generations.

2 I did not understand why a discriminator would have better performance than the original model and why it is the case that these capabilities cannot or should not be employed directly by the reasoning model. 

I think that if I had more explanations on the points above I would have been to understand the technical contribution more.

questions:
Answering the questions 1, 2 above would help.

Edit - post rebuttal: questions 1 and 2 were answered by the authors to my satisfaction.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary:
The paper introduces rStar, a novel self-play mutual reasoning approach that improves small language models' (SLMs) reasoning capabilities without requiring fine-tuning or supervision from larger models. The key technical contributions include:
1. A generation-discrimination process where the target SLM uses MCTS augmented with human-like reasoning actions to generate solutions
2. A verification mechanism using another SLM as a discriminator to ensure mutual consistency
3. Extensive empirical validation showing significant improvements across multiple reasoning benchmarks (GSM8K, GSM-Hard, MATH, SVAMP, StrategyQA)

(b) Strengths:
1. Novel Technical Approach: The combination of MCTS with diverse reasoning actions and mutual consistency verification is innovative and well-justified
2. Strong Empirical Results: Comprehensive evaluation across multiple benchmarks and model sizes, with significant performance improvements
3. No Fine-tuning Required: Method works with pretrained models, making it widely accessible
4. Thorough Ablation Studies: Clear demonstration of the importance of each component
5. Good Generalization: Shows effectiveness across different reasoning tasks and on unseen problems 

(c) Weaknesses:
1. High Computational Cost: The method requires substantial token generation 
2. Manual Action Selection: The five reasoning actions are manually designed, raising questions about optimality
3. Limited Model Size Exploration: Primary focus on 7B-8B parameter models due to computational constraints
4. Potential Edge Cases: In rare cases, early incorrect statements might lead to consistent but wrong answers

(d) Reasons for Acceptance:
1. Significant Technical Innovation: The paper introduces a novel approach combining MCTS, diverse reasoning actions, and mutual consistency verification
2. Strong Empirical Results: Demonstrates substantial improvements across multiple benchmarks
3. Practical Impact: Method works with pretrained models and doesn't require fine-tuning or larger model supervision
4. Thorough Evaluation: Comprehensive ablation studies and analysis validate the approach
5. Clear Presentation: Well-written with clear methodology explanation