Title: Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Modal Large Language Model

Monte Carlo Tree Search

Reasoning
Abstract: In this work, we aim to develop an MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. To this end, we propose Collective Monte Carlo Tree Search (CoMCTS), a new learning-to-reason method for MLLMs, which introduces the concept of collective learning into ``tree search'' for effective and efficient reasoning-path searching and learning. The core idea of CoMCTS is to leverage collective knowledge from multiple models to collaboratively conjecture, search and identify effective reasoning paths toward correct answers via four iterative operations including Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Using CoMCTS, we construct Mulberry-260k, a multimodal dataset with a tree of rich, explicit and well-defined reasoning nodes for each question. With Mulberry-260k, we perform collective SFT to train our model, Mulberry, a series of MLLMs with o1-like step-by-step Reasoning and Reflection capabilities. Extensive experiments demonstrate the superiority of our proposed methods on various benchmarks. Code is available at https://github.com/HJYao00/Mulberry.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Mulberry, a multimodal large language model (MLLM) enhanced with step-by-step reasoning and reflection capabilities, achieved through the Collective Monte Carlo Tree Search (CoMCTS) method. CoMCTS leverages collective knowledge from multiple models to collaboratively search, evaluate, and refine reasoning paths, addressing challenges like search effectiveness and efficiency in traditional MCTS. This paper also constructs a dataset with rich reasoning trees and uses it to train Mulberry, which outperforms existing models on benchmarks requiring complex reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper introduces Collective Monte Carlo Tree Search (CoMCTS), a novel approach that leverages collective knowledge from multiple models to enhance reasoning-path searching and learning in MLLMs.\n\n2. The construction of Mulberry-260k, a multimodal dataset with rich, explicit reasoning nodes, provides a valuable resource for advancing research in step-by-step reasoning and reflection.\n\n3. The paper demonstrates strong performance of the Mulberry models across various benchmarks, outperforming other models.\n\n## Weaknesses\n\n1. The CoMCTS algorithm merely integrates model ensemble techniques with the MCTS algorithm, lacking significant novelty. Previous work, like LE-MCTS [1], has already investigated this combination in NLP tasks. Furthermore, CoMCTS does not exhibit a specialized design for multimodal models. For example, *expansion, simulation and error positioning, backpropagation, and selection* can also be applied to standard LLMs, and some related research on these approaches already exists.\n\n2. The performance improvement from MCTS alone appears marginal. For instance, in Table 2, GPT-4o direct achieves 58.2, while GPT-4o + CoMCTS reaches only 63.8—a modest gain given the increased computational overhead. The more significant improvement seems to stem from ensembling the Qwen2-VL-72B model rather than MCTS itself.\n\n3. The effectiveness of CoMCTS hinges on the diversity and quality of the models in the collective learning group. Additionally, the reliance on multiple models for collective learning and iterative tree search increases the computational overhead.\n\n[1] Park, Sungjin et al. “Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning.” North American Chapter of the Association for Computational Linguistics (2024).\n\nQUESTIONS:\n1. What are the innovative aspects of the CoMCTS tree search method designed for MLLMs, and how does it differ from text-based methods?\n\n2. How can the overhead introduced by CoMCTS during the search phase be measured? If feasible, could you estimate the total token cost, GPU hours, or API price for data generation?\n\n3. Could you provide statistics on the contribution of each model in the *Collective* process, such as the similarity in sampling between different models or the proportion of each model in generating correct paths?\n\n4. Is it possible to achieve good performance without introducing smaller models, for example, by only using GPT-4o and Qwen2-VL-72B? For Qwen2-VL-72B and Qwen2-VL-7B, their behaviors might be similar due to shared training data—does integrating both models still provide an advantage?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces Collective Monte Carlo Tree Search (CoMCTS) to equip Multimodal Large Language Models (MLLMs) with step-by-step reasoning and reflective capabilities. Unlike traditional single-node expansion, CoMCTS performs collective expansion, where each MLLM in the group simultaneously generates potential reasoning paths from the current leaf node. This collaborative expansion mechanism aims to (1) generate diverse and complementary candidate reasoning nodes, (2) prevent the search process from getting stuck in low-quality reasoning spaces of individual models, and (3) significantly improve search efficiency by exploring multiple paths in parallel. Furthermore, this study generated a high-quality tree-structured multimodal reasoning dataset (Mulberry-260k), demonstrating that training on this dataset enhances the reasoning abilities of the Mulberry model and improves performance on complex multimodal benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n+ CoMCTS enables multiple models to collaboratively expand reasoning paths and collectively identify and prune erroneous steps, thereby significantly enhancing the effectiveness and efficiency of MLLM search.\n+ Training MLLMs on data generated through CoMCTS, data that includes explicit step-by-step reasoning and reflection paths, leads to improved reasoning abilities and allows the models to self-correct during complex tasks.\n+ High-quality tree-structured multimodal reasoning datasets like Mulberry-260k are valuable for advancing MLLM intelligence beyond direct prediction, toward more robust, explainable, and human-like reasoning processes.\n\nWeakness\n\n+ Although CoMCTS itself is not specific to MLLMs, the evaluation is conducted solely on MLLMs. From the outset of Section 1, the research assumes an MLLM-based setup, but it would be necessary either to include evaluations on tasks like mathematics or to provide a justification for not addressing such tasks.\n+ Table 2 demonstrates the effect of using collective intelligence from different MLLMs. However, the most significant performance gain is observed when Qwen2-VL-72B is added. Therefore, it would be interesting to see the experimental results of applying CoMCTS using only Qwen2-VL-72B. In other words, it is important to distinguish whether the improvement stems from collective intelligence or from the capabilities of the powerful MLLM, Qwen2-VL-72B.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nI could not find any dedicated section of paragraph explaining the potential limitations of this work. I'd recommend adding the explanation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper develops Mulberry, a Multimodal reasoning model based on Collective Monte Carlo Tree Search. Specifically, the authors introduce collective learning into “tree search” for reasoning-path searching and learning, construct Mulberry-260k dataset, and then finetune a series of multimodal models, aiming to empower the models with reasoning ability. Experiments demonstrate the effectiveness of the models.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well organized and easy to follow.
2. This work introduces the Mulberry-260K dataset and Mulberry models, which will facilitate future research in multimodal reasoning and related areas.
3. Research findings indicate that multimodal models, even with small sizes (2B parameters), can benefit from the reasoning and reflective abilities.



Weakness:
While this work provides a solid incremental contribution, the novelty and insights appear somewhat limited.

QUESTIONS:
1. Table 1 has too many missing values.

2. For Table 2's comparison, what results might emerge if all four models were strong foundation models (e.g., GPT-4o)?

3. Would the authors consider open-sourcing the training data and models?

4. Some writing problems: L68 has an extra space. It is recommended to use \citet{} on L123-125.

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Collective Monte Carlo Tree Search (CoMCTS), an ensemble-style extension of MCTS that aggregates proposals from multiple MLLMs. Aside from the usage of multiple models for generating and evaluating the steps, another innovation is to use some wrong nodes in the chain to simulate self-reflection via transitioning to the right step. CoMCTS improves the search success rate and reduces the number of iterations over standard MCTS. A large scale dataset is created for finetuning MLLMs, which leads to substantial performance improvements in several MLLMS on multiple standard benchmarks.\n\nOverall, this paper presents a significant advancement in search based multimodal reasoning by combining large model ensembles with MCTS and reflective data generation. The proposed approach is innovative, practical, and well-validated empirically, and the dataset in itself is a valuable contribution. Therefore, I recommend acceptance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n 1) A new large-scale multimodal dataset is constructed, annotated with structured reasoning trees, error paths, and reflection nodes. This provides rich training signals for developing compositional reasoning abilities in MLLMs. \n\n2) Mulberry demonstrates strong results across multiple multimodal reasoning benchmarks, outperforming open-source models and rivaling some closed-source systems. \n\n3) Ablations clearly demonstrate the utility of each CoMCTS component. The use of erroneous nodes encourages models to self-correct and engage in more human-like, iterative thinking, and combining steps from multiple models is more effective than using each of those individual models.\n\n\nWeaknesses:\n\n1) Running multiple MLLMs in parallel and pruning through simulations remains expensive. A comparison of necessary token usage for creating LLAVA CoT-100k and this dataset would be useful in understanding this aspect thoroughly.\n\n2) The organization of the experimental results could be better. For example, the contents in Appendix C.1 and C.3 are very interesting and it would possibly be better if they are included in the main paper. Otherwise Table 1 gives the impression that Mulberry is only compared to the base models which is not true. Highlighting its effectiveness in comparison to other ways of finetuning the MLLMs would strengthen the paper.\n\nQUESTIONS:\nI do not have any major questions, please address the concerns raised in the 'Weaknesses' section.\n\nLIMITATIONS:\nAddressed adequately in Appendix K.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}