Title: SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models (LLMs)

Monte Carlo Tree Search (MCTS)

Data Augmentation

Large Language Models (LLMs)

Sibling Guidance
Abstract: Enhancing large language models by simply scaling up datasets has begun to yield diminishing returns, shifting the spotlight to data quality. Monte Carlo Tree Search (MCTS) has emerged as a powerful technique for generating high-quality chain-of-thought data, yet conventional approaches typically retain only the top-scoring trajectory from the search tree, discarding sibling nodes that often contain valuable partial insights, recurrent error patterns, and alternative reasoning strategies. This unconditional rejection of non-optimal reasoning branches may waste vast amounts of informative data in the whole search tree. We propose SIGMA (Sibling Guided Monte Carlo Augmentation), a novel framework that reintegrates these discarded sibling nodes to refine LLM reasoning. SIGMA forges semantic links among sibling nodes along each search path and applies a two-stage refinement: a critique model identifies overlooked strengths and weaknesses across the sibling set, and a revision model conducts text-based backpropagation to refine the top-scoring trajectory in light of this comparative feedback. By recovering and amplifying the underutilized but valuable signals from non-optimal reasoning branches, SIGMA substantially improves reasoning trajectories. On the challenging MATH benchmark, our SIGMA-tuned 7B model achieves 54.92\% accuracy using only 30K samples, outperforming state-of-the-art models trained on 590K samples. This result highlights that our sibling-guided optimization not only significantly reduces data usage but also significantly boosts LLM reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SIGMA, a refined version of MCTS that uses an oracle LLM prompted with sibiling node reasoning traces to generate feedback to refine the selected reasoning path. SIGMA is used to generate low sample reasoning datasets for fine-tuning which is empirically validated with three model variants. The authors find that SIGMA outperforms other methods in terms of fine-tuning sample efficiency and is competitive with much larger datasets such as DART-math.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The proposed method is intuitive and well motivated. \n* It appears the use of sibling guidance in MCTS is novel in the studied setting. \n* A diverse set of models and math datasets are empirically studied. \n* The result is significant and represents a valuable contribution to an important area of study. Improving sample efficiency is a long-standing challenge of the field. \n* The proposed method is automated and scalable, potentially enabling the curation of large, high quality datasets. \n\n## Weaknesses\n* My primary concerns regarding this work are related to its motivation, some absent analysis, and presentation.\n* The synthesis cost of SIGMA is not discussed. While I am convinced that SIGMA can be used to generate high-quality training samples, it’s unclear what the cost is to generate the dataset. This information would help motivate the selection of using SIGMA to generate a bespoke dataset versus using a larger dataset such as DART-math. \n* The total tokens in the SIGMA dataset is unclear. Since we backpropagate across all completion tokens during fine-tuning, an important but absent discussion is the total number of tokens across SIGMA and baseline datasets. \n* Only GPT-4o-mini is explored as $\\mathcal{C}\\_{LLM}$ and $\\mathcal{R}_{LLM}$. \n* Qwen-2.5-Math-7B is used as the candidate generator to construct search trees. One particularly interesting analysis, which was not included, is to fine-tune the same model on the SIGMA dataset to observe if SIGMA can be effectively leveraged in the self-distillation setting. \n* Presentation concerns:\n\t* Authors should cite sections of the appendix in the main body where relevant. For example, when discussing training hyperparameters that are not explicitly stated in the main body. \n\t* The 60k SIGMA results presented in Appendix A are the most significant, I suggest moving these to the main body in lieu of Table 1. Highlighting that at 60k SIGMA enables higher quality models than even DART-Math or models trained on the full MathFusionQA dataset would improve the impact of the work. \n\t* Overall I find the formalization of Textual Gradient Descent to be more confusing than illustrative. Equation 5 can simply be re-written as $\\tilde{T}\\_p = \\mathcal{R}\\_{LLM}(T_p, \\mathcal{C}_{LLM}(T_p | T_s \\in \\mathcal{S}_p) )$ without requiring us to stretch the definitions of terms such as gradient, loss, and optimizer to contexts beyond which they normally apply in ML literature. I understand that the authors are following the lead of [43] here; however, I believe the method could be more plainly stated without losing meaning and would improve the overall clarity.\n\nQUESTIONS:\n### Key questions to improve my score:\n* What is the synthesis cost of SIGMA vs. other baselines such as DART-Math or MathFusion? Is SIGMA more optimal in terms of number of samples vs. cost to generate than the studied baselines? \n* What is the total number of completion tokens in SIGMA vs. the other baselines? When we analyze the main results in terms of accuracy vs. completion tokens, do the main findings hold? Ultimately, our fine-tuning fine-tuning cost will scale with total tokens, not samples. \n* How does Qwen-2.5-Math-7B perform when fine-tuned on the SIGMA dataset? Does SIGMA enable self-distillation? \n* Some outputs from $\\mathcal{C}\\_{LLM}$ do not actually appear to contain a critique. Eg., from the geometry question: “Evaluation: Step 0-1-2-1-0 calculate the square; step 0-1-2-1-1 calculate the area\nof the triangle; step 0-1-2-1-2 compute the answer.”. $\\mathcal{C}\\_{LLM}$ appears to simply be describing the sibling node approaches but doesn’t offer any specific critiques. Could the authors speculate on whether the more robust critiques could be extracted from $\\mathcal{C}_{LLM}$  and if so, how?\n\n### Minor issues, suggestions, typos\n* Exploring additional oracle models, candidate generation models, and tasks would make the work more impactful. Given that SIGMA produces natural language feedback, it may be good fit for tasks where traditional verifiers are not available (eg., creative writing). \n* Consider my comments regarding the overall presentation above. I believe the work would benefit from improvements in this area.\n* Typos:\n\t* L145: “To”\n\t* L148: “Subscript” -> “Superscript”\n\t* L159: “operats”\n\nLIMITATIONS:\nMore discussion on the synthesis cost and total number of tokens in SIGMA datasets is required to fully understand the impact and limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework that reuses the discarded alternative generations (sibling nodes) produced during LLM test-time search via Monte Carlo Tree Search (MCTS). These sibling nodes were discarded in prior work [1]. The framework instead leverages a critique LLM (GPT-4) to provide feedback on these sibling nodes, which is then used by a revision LLM (also GPT-4) to rewrite the generation step into a chain of thought, thereby improving the quality of supervised training data.\nThis process enables the authors to generate smaller but higher-quality training data compared to using MCTS alone. The approach is evaluated on three 7–8B parameter models across several math reasoning datasets and benchmarks, including Vanilla MCTS, Random-Sibling MCTS, Self-Refine, Tree-of-Thoughts-20, direct CoT SFT, and MathFusion-60K.\n\n[1] Chen, G., Liao, M., Li, C., & Fan, K. (2024). AlphaMath Almost Zero: Process Supervision without Process (arXiv:2405.03553). arXiv. https://doi.org/10.48550/arXiv.2405.03553\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\nTest-time search in LLMs and self-refining reasoning LLMs are highly relevant topics, and this paper is well motivated and well written.\n\nWhile I find the methodological contribution narrow as it mainly relies on using all generations and as such modest, the repurposing of the otherwise discarded siblings is carried out methodologically sound. The evaluation on different baselines (Vanilla MCTS, Random-Sibling MCTS, Self-Refine, Tree-of-Thoughts-20, direct CoT SFT, MathFusion-60 K) is comprehensive.\n\nThe incorporation of critique and refiner models is a simple approach to improve certain siblings and can be easily applied.\n\nWeaknesses:\n\nAs Mentioned already in the strengths, the novelty lies in a narrow domain, using all generations for learning as an addition to previous work using MCTS. There is work in that direction I find not adressed in this submission, such as [2], that in a similar manner use generations for each token in a prompt for supervised training. While the authors add a trick with the critique- and refinement LLMs, this shows some overlap. As such it further limits novelty in my view.\n\nWhile evaluation across benchmarks is comprehensive, I find evidence for the generalizability across models and domains could be more robust:\n1. The approach is limited to the math domain, so I am unsure how well it generalizes to more open domains such as commonsense reasoning.\n2. While it is evaluated across model families, the evaluation is limited to similar sizes and doesn't use specialized models, as [1] does.\n\nQUESTIONS:\nHow would SIGMA’s performance degrade when you change the critique and revision LLMs to weaker open-source models?\nHave you measured how the number of sibling nodes affects SIGMA's performance?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduced a framework for refining the reasoning capabilities of large language models (LLMs) by leveraging sibling nodes in Monte Carlo Tree Search (MCTS), called SIGMA. SIGMA reintegrates discarded sibling nodes to enhance reasoning paths through a two-stage refinement process: a critique model discovers strengths and weaknesses across sibling sets, and a revision model refines the top-scoring trajectory based on the findings from the first step. The method improves reasoning trajectories, achieving state-of-the-art performance on the MATH benchmark with only 30K samples.\n\nSTRENGTHS AND WEAKNESSES:\n1.The core idea of leveraging sibling information for refinement is not entirely novel. Similar concepts have been explored in other domains, such as symbolic supervision via LLM-generated feedback [43]. The paper does not provide a clear distinction or significant advancement over these existing methods.\n2.The paper lacks a formal theoretical analysis of why the proposed method works and its potential limitations. A deeper understanding of the underlying mechanisms and potential failure modes would strengthen the paper\n3.The performance of SIGMA is highly dependent on the quality of the sibling nodes generated during MCTS. If the sibling nodes are not well-formed or fail to capture robust reasoning behaviors, the transferred feedback may not provide significant benefits. \n4. While SIGMA avoids full MCTS fine-tuning on large models, it still requires access to a reasonably strong small expert model trained with MCTS, which can be computationally expensive to obtain. The paper does not provide a detailed analysis of the trade-offs between the computational cost of training the small expert models and the benefits gained from using SIGMA.\n\nQUESTIONS:\nThe authors need to answer all the weak points listed above.\n\nLIMITATIONS:\nPlease check the weak points listed above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SIGMA (SIbling-Guided Monte Carlo Augmentation), a novel framework designed to enhance the reasoning capabilities of large language models (LLMs) by refining Monte Carlo Tree Search (MCTS)-generated reasoning paths. Traditional MCTS-based approaches select a single high-reward reasoning path and discard alternative (sibling) paths, potentially losing valuable information. SIGMA addresses this by:\n\n1. Reusing sibling nodes from the MCTS tree to generate natural language critiques of the selected reasoning steps.\n\n2. Using these critiques as symbolic gradients to iteratively refine the reasoning path via a Textual Gradient Descent (TGD) process.\n\n3. Employing a critique model and a revision model (both LLMs) to perform this refinement without requiring additional rollouts or external supervision.\n\nThe authors demonstrate that SIGMA significantly improves the quality of reasoning traces and downstream performance on mathematical reasoning benchmarks, outperforming larger datasets and state-of-the-art baselines with fewer training examples.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n**Quality**: The method is well-motivated and addresses a clear inefficiency in current MCTS-based reasoning pipelines. Empirical results show consistent and significant improvements across multiple benchmarks and model backbones (DeepSeekMath-7B, LLaMA3-8B, Mistral-7B).\n\n**Clarity**: The paper is clearly written, with intuitive figures (e.g., Figures 2–4) that illustrate the SIGMA pipeline. The methodology is well-structured, with detailed explanations of each component (MCTS, critique generation, revision).\n\n**Significance**: SIGMA offers a data-efficient alternative to large-scale dataset construction, which is valuable given the high cost of data generation. The framework is model-agnostic and could be extended to other domains beyond math reasoning.\n\n**Originality**: The idea of treating sibling nodes as sources of symbolic feedback and using them for gradient-like refinement is novel to the best of my knowledge.\n\n### Weaknesses:\n\n1. The key point about refining the selected step $T_p$ based on the unselected siblings $T_s$ feels a little counterintuitive. If the siblings were not selected because they were expected to yield inferior results by the MCTS process then why would refining the content of $T_p$ to be closer to the content of $T_s$ be beneficial?\n\n2. The experiments are restricted to mathematical reasoning tasks. While the authors claim generalizability, this is not empirically validated.\n\n3. Given the extra effort involved in preparing the data using SIGMA and the fact that MathFusion-60K gives better results for two (LLaMA3 & Mistral) out of the three models considered, it isn't clear if we should go with SIGMA or just use a larger dataset.\n\nQUESTIONS:\n1. What is $n$ & $k$ as mentioned in Section 4.1 in the context of the algorithm given in Section 3? There seems to be some notation abuse here since in Section 3 $n$ is used to denote node and in Section 4.1 it is used to denote candidate completions.\n\n2. In line 280 you say, \"Despite sharing the same generator model...\". How is the generator model the same for black-box CoT generation and fine-tuning using SIGMA when the former uses GPT-4o-mini (line 279) and the latter uses Qwen2.5-Math-7B?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SIGMA, an enhanced MCTS framework that refines model-generated reasoning traces by leveraging sibling information in the search tree. Given a standard MCTS that explores the space of multi-step CoT paths and selects an optimal path, the SIGMA framework additionally refines the CoT on every node of the selected path, by considering the discrepancies between the selected node and discarded sibling node. For every refinement, a critique language model is used to suggest how to revise and a revise language model takes that suggestion and generates an improved version.\n\nThe author applied this framework to the math domain, with a setup that prompts and final answers are given, and the goal is to refine the reasoning traces to construct better training data. The training set is based on prompts from MATH and GSM8K datasets, and Qwen2.5-Math-7B is used to generate the original reasoning paths. Then, GPT-4o-mini is used to conduct the refinement. The resulting datasets are compared to existing math datasets of similar size or larger size, demonstrating better performance in terms of training math models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper addresses a timely topic of enhancing the MCTS framework for generating more effective CoT reasoning paths. The proposed improvement incorporates more information from sibling nodes looks meaningful and promising.\n- I found the formulation of the critique model and refine model under the textual gradient descent very intriguing and inspiring. This may be another good attempt of the [textgrad](https://arxiv.org/abs/2406.07496) framework.\n- The experiment results demonstrated better performance of the resulting math datasets.\n\nWeaknesses:\n- My major complaint is about the evaluation setup. The resulting datasets in this paper is based on CoT reasoning generated with Qwen2.5-Math model and then refined by GPT-4o-mini. The other existing datasets for comparison were generated with different models (e.g., MathFusion was generated using GPT-4o-mini only?). I believe that the base model makes a considerable difference here, and it's not clear to me how the proposed refinement contributed to the final improvement.\n\nQUESTIONS:\n- What is the reasoning behind choosing Qwen2.5-math for generating the original CoT and GPT-4o-mini for the refinement?\n- For the training experiments, why not finetune Qwen2.5 model using the resulting data? It might be intriguing to see the model's self-improvement using this framework.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}