Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search
Authors: Huanjin Yao
Jiaxing Huang
Wenhao Wu
Jingyi Zhang
Yibo Wang
Shunyu Liu
Yingjie Wang
YuXin Song
Haocheng Feng
Li Shen
Dacheng Tao
Venue: NeurIPS 2025 spotlight ()
Keywords: Multi-Modal Large Language Model
Monte Carlo Tree Search
Reasoning
Primary area: applications

Abstract: In this work, we aim to develop an MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. To this end, we propose Collective Monte Carlo Tree Search (CoMCTS), a new learning-to-reason method for MLLMs, which introduces the concept of collective learning into ``tree search'' for effective and efficient reasoning-path searching and learning. The core idea of CoMCTS is to leverage collective knowledge from multiple models to collaboratively conjecture, search and identify effective reasoning paths toward correct answers via four iterative operations including Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Using CoMCTS, we construct Mulberry-260k, a multimodal dataset with a tree of rich, explicit and well-defined reasoning nodes for each question. With Mulberry-260k, we perform collective SFT to train our model, Mulberry, a series of MLLMs with o1-like step-by-step Reasoning and Reflection capabilities. Extensive experiments demonstrate the superiority of our proposed methods on various benchmarks. Code is available at https://github.com/HJYao00/Mulberry.

Mulberry: Empowering MILM with ol-like Reasoning and Reflection via Collective Monte Carlo Tree Search
Huanjin Huang Wenhao Wur, Jingyi Zhang', Yibo Wang?, Shunyu Liu', Yingjie Wang', Yuxin Song?, Haocheng Feng', Li Shen', Dacheng Tao 1s Nanyang Technological University Tsinghua University Baidu Inc. Sun Yat-sen University
Equal Contribution
S Corresponding Author
Abstract
In this work, we aim to develop MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. To this end, we propose Collective Monte Carlo Tree Search (CoMCTS), new learning-to-rest to-reason method for MLLMs, which introduces the concept of collec- tive learning into "tree search" for effective and efficient reasoning-pate searching and learning. The core idea CoMCTS is to leverage collective knowledge from multiple models collaboratively conjecture, search and identify effective reason- ing paths toward correct answers via four iterative operations including Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Using CoM- CTS, we construct Mulberry-2660 and well-defined reasoning nodes for each question. With Mulberry-2601 we perform collective SFT to train our model, Mulberry, a series of MLLMs with ol-like step-by-step Reasoning and Reflection capabilities. Extensive experiments demonstrate the superiority of our proposed methods on various benchmarks. Code is available https:://// :/gg thub. com/HHYaaoO0//ubbbrrry
multimodal dataset with a tree of rich, explicit
Introduction
"What I cannot create, I do not understand."
RRichard Feynman
Multimodal large language models (MLLMs) embody the essence of this dictum, which understand the world by learning to create expected responses multimodal inputs such as images and text. While MLLMs have recently shown significant progress in straightforward tasks [12), they often
experience obviously increased failures on complex tasks requiring in-depth reasoning 3 Feynman's dictum might be the perfect metaphor of such failures of MILMM, we should only be able to work something out if we can create and have a firm understanding of each step of the reasoning involved. However, current MILMS predominantly operate in a simple "direct prediction" mode [4], ie.. generating brief, final answers to questions with little explicit and well-defined intermediate reasoning steps.
In this work, aim develop MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. Recent advances in NLP, such as OpenAI o1 [5], have shown great potential in enabling LLM to learn to reason and tackle complex language tasks [ The core design of these advances lies in AlphaGo-lia "tree search": they employ tree search methods, like MCTS to bootstrap an L.LM itself to build a tree
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
10 Comparioon Search Buoceso Rotes and Average Search Itoraerro
Accuracy Mathvtrts
B Searn Race
Average Seace beatcc
Our Todeo
Restt Search Ioneationn (lmme
Search Succes thee (1)
MUS
Acusscy PM
+0
Miller
34
LOOGA
prod
MCTS
ReSTAICTS
Ortgga-MCTS
COMCTT
Figure I: (a) Our CoMCTS shows great superiority in search effectiveness and efficiency against (b) Our Mulberry, trained CoMCTS -earchd data, outperforms most open-sourced MILMs and achieves competitive results against closed-source ones, showing outstanding abilities in step-by-step reasoning and reflection.
other tree search methods.
of intermediate thoughts, explore effective reasoning paths, and leverage these paths to teach model to reason step-by-step.
An intuitive idea is to directly apply these "tree search" methods to search effective reasoning paths for MILMs, which, however, does not work well. As shown in Figure I we believe this largely attributed to several search challenges for MLLMs. (1) Search Effectiveness: Traditional MCTS methods 7 [Q generally work by bootstrappiin while current MILMs are typically trained with little explicit and well-defined intermediate reasoning steps, making these search methods often trapped in homogeneous low-quality nodes within the reasoning space of a single MILM, ultimately leading to low search success rates. (2) Search Efficiency: Traditional MCTS methods typically expand and explore only one reasoning node per search iteration, which advance single step each time and demand massive iterations, making them inefficient for computation-intensit MILMs.
To tackle these challenges, we propose Collective Monte Carlo Tree Search (CoMCTS), new Searningtto-raase method for MLLMs, which introduces the concept of collective learning into "tree search" for effective and efficient reasoning path searching and learning. The core idea of CoMCTS is to leverage collective knowledge to collaboratively conjecture, search and identify effective reasoning paths toward correct answers. Specifically, CoMCTS searches effective reasoning paths iteratively, and in each iteration, it leverages collective knowledge from multiple MLLMS to jointly (a) expand diverse and complementary candidate subsequent reasoning nodes till the end from a given start node, (b) simulate reasoning outcomes, position error candidate nodes and prune them along with their child nodes, (c) backpropagate to update the score and visit count of each reasoning node in a bottom-up manner, and (d) select the leaf reasoning node with the highest Upper Confidence Bound value as next start node.
In this way, our CoMCTS achieves effective and efficient reasoning search. (1) The joint expansion mechanism enables CoMCTS to concatenate reasoning trajectories from multiple MLLMs via iterative search, ultimately constructing an unified reasoning tree comprising diverse and complementary reasoning nodes. Thus, allows reasoning-pathh search not only within the reasoning space of a given MLLM itself but also among those of others, benefiting from the synergy of multiple MLLMs while avoiding being trapped in homogeneous low-quality nodes within the reasoning space of a single MLLM itself.
(2) The joint simulation and error positioning mechanism enables CoMCTS to, in each search iteration, skip multiple intermediate steps and select the last correct step as the next start node, largely reducing search time while maintaining search effectiveness. Here, collective knowledge is also crucial as it is often challenging for a model to recognize and position errors made by itself while relatively easy by using other models.
Furthermore, we extend our CoMCTS for reflective reasoning-pan search. Based on the unified reasoning tree constructed by CoMCTS, which provides both positive and negative reasoning nodes. we identify and integrate negative sibling nodes into effective reasoning paths to build the reflective reasoning path that includes a transition from a negative reasoning node to positive one. By learning from reflective reasoning paths, MLLMS can perform appropriate step-wise reflection, dynamically calibrating their reasoning trajectory from an erroneous node toward a correct one during long-chain reasoning. Here, collective knowledge facilitates reflective reasoning- caashnnn--pahh search by providing a rich set of diverse positive and negative reasoning nodes.

---

Using our CoMCTS, we search effective and reflective reasoning paths for set of multimodal inputs. and construct Mulberry-2665 a Multimodal learning-to Beeeiingtt-eeooon--e seeen-an--Roe mef-Reflect dataset with tree of rich, explicit and well-defined reasoning nodes for each question. With Mulberry-260k, we perform collective supervised fine-tuning to train our model, Mulberry, a series of Multimodal LLMSs with ol-like step-by-step Reasoning and Reflection capabilities.
The main contributions of this work are fourfold. First, we introduce the concept of collective learning into MCTS, and propose CoMCTS which leverages collective knowledge to collaboratively conjecture, search and identify effective and reflective reasoning paths for MLLMs, significantly improving search effectiveness and efficiency. To the best of our knowledge, this is the first work that explores collective learning with MCTS for MLLMs. Second, we construct Mulberry-200 that provides a valuable resource for advancing research in step-by-stey reasoning and reflection in MI.LMs. Third, we develop Mulberry, a series of MI.LMs with outstanding capabilities in step- by-step reasoning and reflection. Fourth, extensive experiments demonstrate the superiority of our proposed methods on various benchmarks.
Related Work
2.1 Multimodal Large Language Model
have made notable advancements in general vision-language understanding, enabling them interpret visual semanties across various domains. Recent studies [17] 3 explore MLLM reasoning and reveal that directly employing CoT prompt to derive the final answer may result in limited gains or even degradation. In addition, some studies [88 19] introduce plan- based CoT prompting to guide models to generate intermediate information for predicting final answers. Recent advances attempt structured reasoning with planed flow of certain pre-defined stages, enhancing the CoT capabilities [15] of MILMs. Differently, this paper, for the first time, introduces the concept of "tree search" into MILMM reasoning and proposes a novel CoMCTS technique to search effective and reflective reasoning paths to train our Mulherry, a series of MILMMs with outstanding capabilities in atep-by-step reasoning and reflection.
MLLMs 2 12 3
14
15
16
2.2 Large Language Model Reasoning
LLM reasoning methods can be broadly categorized into three types, i.e., prompt-based, plan-based and learning-base based reasoning. Prompt-based methods, like Chain-of-Thought (CoT) (20), mimic human reasoning by providing few hand-crafted, step-by-step solutions as references. Plan-based methods, such as teee/Grrah-fftthouge graph manner and take consistent units of thought for thoughtful decision- makinn.. Learning-basee reasoning methods, represented by GPTol, Star (23). Iter-MCTS 66) and ReST-MCTS [24], first employ tree search approaches (25), like MCTS, to bootstrap LLM itself to build tree of intermediate thoughts, explore effective reasoning paths, and leverage these paths to train model to reason step-by-step.
[2I] 22], predict multiple reasoning paths a tree or
2.3 Monte-Carlo Tree Search
Monte-Carlo Tree Search (MCTS) is a powerful search paradigm for complex decision mak-
ing problems and has been extensively explored across diverse fields, including games 266 27]
robotics 28 29), theorem proving 300 matrices multiplication [31], etc. For instance, AlphaGo 26
introduces deep learning into MCTS, achieving superhuman results in board and video games 26 27
Besides, 332 33] explore MCTS for path finding and train timetabling problems, while (34] integrates
MCTS into thysics-informed planning networks for robot control. In this work, we propose CoMCTS
that enables effective and reflective reasoning-pate searching and learning on MLLMs.
2.4 Collective Learning
Collective learning, also known as Co-training, aims harness collective intelligence of multiple individuals to improve learning outcomes. This concept originates in early pioneering studies 35 36
37 which utilize collective knowledge to address data insufficiency issues in classification learning. Recent advances introduce collective learning into deep neural networks for efficient and effective
3

---

deep learning. For example, [8] 39] employ collective knowledge from multiple classifiers to predict
more accurate pseudo-labels for -uuieeuuee classification; 40 utilizes collective knowledge from multiple discriminators to enhance image discrimination and generation; and [1 leverages the synergy of multiple models for reinforcement learning.
3 Method
We first present CoMCTS that introduces collective learning into "tree search" for effective and efficient reasoning-pali searching and learning. We then illustrate the extension of CoMCTS for reflective reasoning-pand search, and describe data construction and model training using CoMCTS.
3.1 CoMCTS for effective reasoning
The core idea of CoMCTS is to leverage collective knowledge to collaboratively conjecture, search and identify effective reasoning nodes in an iterative manner, aiming to find effective reasoning paths leading to correct answers.
We denote a policy model as π, initialized by a pre-trained MLLM. We leverage collective knowledge from group of MLLMs { :2...., ..., TK) to jointly search and learn effective reasoning paths. Given a multimodal input question Q (e.g., a text instruction with an image, Q = {text, image }), each model can generate sequence of intermediate reasoning states toward the final answer (#1, $2, $3, SMr) ~ (-(-) via autoregressive next token prediction. We define the intermediate reasoning state at step m as 8m and the state generated by model gıK step m as sk Each reasoning step consists of one or few sentences.
CoMCTS algorithm begins at the root node, i.e., either the start of response or an incomplete response, and performs reasoning-path search via a certain number of iterations, where each it- eration comprises four key operations: (a) Expansion, (b) Simulation and Error Positioning, (c)
Backpropagation, and (d) Selection, as elaborated below.
(a) Expansion. The goal of this operation in CoMCTS is to expand the current leaf reasoning node (if it is not a terminal node) to integrate new subsequent candidate reasoning nodes. Given the current leaf node st, (i.e., the node selected by Operation (d) Selection or the root node), CoMCTS utilizes collective knowledge from a group of MLLMs, (m1, T2, TK to jointly expand a set of diverse and complementary candidate reasoning paths Seandidate = U St candidate in parallel till terminal node:
S Sundiidate (-(:1Q. Parent( (s,, )
where Parent(s (t returns all parent nodes of st and (Parent (xtt) sth denotes the current reasoning path from the root node to d St model n starting from s
{ stands for a potential reasoning path generated by
candidate
(b) Simulation and Error Positioning. In this operation, CoMCTS utilizes collective knowledge from [T1, ......, Tx }o to jointly estimate the potential value of child nodes si E Seundidate (added in Operation (a)), and considers low-score nodes as erroneous reasoning nodes, and positions and filters out them along with their child nodes:
R(x))
1 K
Σ Ti( (-Ipprompteval Q, Parent(s/)) (s)),
(2)
Sandidatt = {s} E Senndidate [R(s?) >=
where R(s{) denotes reasoning node evaluation function that uses the prompt, promptova request a group of MLLMs, { T2, T }, to jointly evaluate the candidate reasoning node s! t is a threshold and discontinued reasoning nodes in Sandidate automatically removed following the error node removal in (3).
(c) Backpropagation. Given the new reasoning tree expanded and simulated using collective knowledge in Operations (a)-(b), CoMCTS performs a bottom-up update from the leaf nodes back to the root node. Each node along the newly expanded path in the reasoning tree updates its statistics,

---

O
Temporary
000
O
Repeat X timen
found connect neassmiing part
Eipansion
Smolltion Erior Postionnn
Backprosagetion
Selection
Path Scarch with CaMCTS
Updatel Policy
Searched Reasoning Pachs
Quendionn
CoSFT with GaMCTS Data
perimeterof of the trfangle
Policy Medell
Updated oj
10
Polies Model
Updated Medel
Modelig
Updaied Medel as
Model Training with CuMCTS
[aj COSTT
CIGFT Retactia Bastoring
Figure 2: Overview. Our CoMCTS trains Mulberry with two alternating phases. In top part, CoMCTS searches reasoning paths iteratively, and each iteration, utilizes collective knowledge from multiple MLLMs to jointly (a) expand diverse and complementary candidate subsequent reasoning nodes till the end from a given start node, (b) simulate reasoning outcomes, position error candidate nodes and prune them along with their child nodes, (c) backpropagate to update the score and visit count of each reasoning node a bottom-up manner, and (d) select the leaf reasoning node with the highest UCB value next start node. In bottom part, train the model to learn from the reasoning trees constructed by CoMCTS.
including visit count N and node value V:
V(s) + ((s) V(s) + E,jechiaass) R(si)
(4)
((s) + CountChild (S Candidane S)
(()) + (x)) + CountChild (aantChild(Ssaniiiee Sandidate 8),
(5)
where Child(s) returns all the child nodes of s, and DountChild(SSannaaaa 8) is child node counting function that calculates the number of child nodes of s in Sandidate"
(d) Selection. Following Operations (a), (b) and (c), CoMCTS traverses the updated reasoning tree to select the next starting node. This selection is guided by the Upper Confidence Bound (UCB) value, which balances search exploration and exploitation. The UCB value of node is computed using the node reward value (s) and the visit count N(s). Among the candidate nodes E S camdidate the
one with the highest UCB value chosen as the starting node s for next search iteration:
log N(s) N(s)
h"" mag
max V(s) + c.
V
(6)
SES- candata
where stands for constant which controls the level of exploration. denotes the parent node of s.
CoMCTS. These four operations, i.e., (a) Expansion, (b) Simulation and Error Positioning, (c)
Backpropagation and (d) Selection, repeated for pre-defined number of iterations until correct reasoning paths are found. This iterative process allows CoMCTS to construct a question-dependenn reasoning tree S with the correct reasoning path Y. and ultimately form multimodal learning- to-reason data triplet {Q, Y,S). By applying our CoMCTS to a set of multimodal questions, we construct collection of multimodal learning-to-Pen to-reesoon data triplets, which provide tree of rich, explicit and well-defined reasoning nodes toward the final answer for each question and enable MLLMs learn to reason step-by-step.
3.2 CoMCTS for reflective reasoning
In this subsection, we extend CoMCTS for reflective reasoning-patte search. Based on the unified reasoning tree constructed by CoMCTS, i.e., {Q, Y, which provides both positive and negative
5

---

reasoning nodes, we identify and integrate negative sibling nodes into effective reasoning paths to build the reflective reasoning path that includes transition from negative reasoning node positive one.
Identifying negative sibling node. Given the effective reasoning path Y, we identify the negative sibling reasoning node for E Y using UCB:
S nog
arg min (sB(s:) - UCB(s), Vs E Y, st (ibbing(())
where Sibling(s) (s) returns all the sibling nodes of s, i.e. the nodes on the same hierarchical level under the same parent node of s. UCB(s) (s) + c.
log N(8) I+N(s)
as in Eq
Constructing reflective reasoning path. Based on Eq. 7 we randomly sample a reasoning node E Y with its negative sibling node Sneg. and concatenate them with reflection prompt to form a reflection trajectory. ie., (Kneg; prompi s). We then use a function Replace(-) that replaces E Y with (Sneg: promphentea: 8) convert Y into the reflective reasoning path Yrefterr:
Yeefe Replace(Y, 8, (Sucg, prompt 6)),
where promphenec denotes reflection prompt, such "The previous reasoning step is wrong and let's rethink again." Then, we can integrate the reflective reasoning path yeteett into our data as quadruplet {Q, Y, Yretee S} E D.
Collective Supervised Fine- Tuning (CoSFT). Given () DD apply standard SFT objective to train our MILMM to learn from D constructed by CoMCTS:
CCosFT (T&)
Σ log (Y
(Q,Y)E
where Y = {s} denotes the effective reasoning path that includes sequence of reasoning nodes collectively conjectured, searched and identified by a group of MLLMs.
CoSFT for reflective reasoning. Given question and its reasoning tree (Q,S) D constructed by CoMCTS, we randomly sample a reflective reasoning path Yeelee from S as in Eqs g and conduct CoSFT for reflective reasoning:
(COSFTR((T))
Σ
log & & (Yretteal QQ),
(10)
ED
where Yrenea = {s) denotes the reflective reasoning path that includes an additional step-wise reflection trajectory.
The goal CCSSTT and LCoSFTERe is to maxi- mize the log probability of effective and reflec- tive reasoning path Y and Yrellea over a tree of reasoning nodes S generated by CoMCTS. In addition, LCoSFT-Re enables to leverage the negative information during CoMCTS search process by learning to calibrate negative rea- soning nodes.
Algorithm Training Mulberry with CoMCTS
Input:
set of policy models {m1, T2, ... TK } ini- tialized by different MLLMs; set of multimodal
questions DQ
for i= to MaxEpoch do
Reasoning Tree Search using CoMCTS: for Q E De do
Collective Monte Carlo tree search: {Q, (. COMCTS((T)) .,,... πK Q)
3.3 Training with Collective MCTS
if found an effective reasoning path then Search and find Yrete from S Add {Q, Y. Y Yrellecs S} into D
Using CoMCTS, we search effective and re- flective reasoning paths for set of multimodal input questions, construct Mulberry-260k multimodal learning-to-reason ao--resso--nnd-rfleeed dataset with a tree of rich, explicit and well- defined reasoning nodes for each question, i.e., a set of quadruplets {Q, Y, Yrefeet S} E D. To learn collective knowledge from Mulberry- 260k, we perform collective SFT to train model, Mulberry, series of Multimodal LLMs with ol-e step-by-step Reasoning and Reflection capabilities.
Remove from DQ
Model Training with CoMCTS Reasoning
Trees:
for k 1 K do
for (Q, Y, Yretect, 5) do Supervised Fine-Tuning:
Optimize
Tr. via (CoSFT(TT)) and
(CoSFT-R((TR)
Output: Trained policy models {ii 52, ,2,... TK)

---

Orangi, Slue Green reasoring nodes geserated by different policy models 1Fy.
Question: Calculate the length of ZU
Level 1
The Egur abouss trangl and Z and un hertrartl botton.
The Siggaa shows night perperdlizzar to
ZY. typoteeuse WY
Level
sangle WCY silluce Well base thr This alficco evunoo teo rgii
Pychopoiig to the base bt
The probice innolos Brilin U2 asx georwere propertes teeee right bittgles specificily Eni...
right tisngle with the right any
Level
Stept: Pythapmen Tecnam oue NUZ: 4p
Reflection
Stept Pazzamin e AUZ:
Stept geomettc property bianphoc a right tranghe.
the lengtee WCZ 6'n WY 4-40 io- = 6it-
Level
The Gial answer 22
The Tinal answer is
The tinal anower
The Final answer 24
The Eiaal answer is
The final answer is
The Enal
The Tirel anower 16
answer
Figure 3: Qualitative illustration of reasoning tree searched by CoMCTS with rich, explicit, well-
defined reasoning nodes.
Experiments
In this section, we first introduce CoMCTTS-generacee dataset, Mulberry-260K, including its sources, construction, and analysis in Sec. 4.1 and provide implementation details in Sec. 4.2) We then present the main results in Sec. 4.3 demonstrating the effectiveness the searched data (i.e.. Mulberry-260k and the trained models (i.e., Mulberry). In Sec. 4.4 we perform comprehensive ablation studies on the impact of effective and reflective reasoning data and the contributions of collective knowledge sources. Sec. 4.5 discusses the effectiveness and efficiency of CoMCTS with other tree search methods.
4.1 Dataset
The Sources of Raw Data. To construct a comprehensive and ceneral-purposs tree-based reasoning dataset, we collect 260K raw multimodal input questions (i.e., a text task instruction with an image as an input question) from wide range of domains, covering General Multimodal Understanding. Math- ematics, Figure Understanding. Realworld Understanding, Science, Medical Image Understanding. etc. The specific data sources are provided in the Appendix 0
Reasoning Data Construction. As detailed in Sec. and Algorithm D and visually illustrated in
Figures 2 and 3 we employ our CoMCTS search effective and reflective reasoning paths for set of raw multimodal input questions as collected from the mentioned "The Sources of Raw Data", ultimately constructing our dataset, Mulberry-2660 2600.. Note we only sample 15K data for reflective reasoning training to avoid overabundance of reflection data.
4.2 Implementation Detail
We implement collective learning in CoMCTS with four models, including GPT-4o, Owen2-VL-7BB LLaMA-3.2-1 ---VViiio--Insteeee and Qwen2-VL-72B, to construct Mulberry-600K, In CoMCTS, we set maximum search iteration to 20 and threshold t in Eq. to O. In each iteration, each model generates one candidate reasoning path to balance search exploration and exploitation. We adopt four popular MILMs as baselines, conducting experiments on Qwen2-VL-7B and LLaMA-3.2-1 IB- Instruct to examine the effectiveness CoMCTS, and on Qwen2-VL-22 VL-BB and LLaVA-NeXT. 8B to study the generalization of sccrrchee data. Training details are in the Appendix B
4.3
Main Results
To examine the effectiveness of searched data (i.e., Mulberry-260k, and trained models (i.e., Mul- berry), we conduct extensive experiments with four powerful baseline models, and comprehensively benchmark Mulberry with various state-of-the-arte including general and reasoning based
7

---

Table Main Results. To examine the effectiveness of the searched data (i.e., Mulberry-260K) and the trained models (i.e., Mulberry), we conduct extensive experiments with four powerful baseline models, and comprehensively benchmark our Mulberry with various state-of-the-arts, including general and casoning-basee MILMs.
Method
VathVist MMStar MMMU ChurlQA DynaMath HallBBench MM-Math MMB
AVG
Closed-Source Model
GPT-4o 42)
63.8
63.9
69.1
85.7
63.7
55.0
31.8
2329
64.5
Claude-3.5 Sonnet
45
67.7
62.2
68.3
90.8
64.8
55.0
1920
Open-Source Model
MM-1,5-7B +4
47.6
41.8
78.6
1861
BlffisS-LLLaMA3-88
45
58.4
55.9
46.6
74.8
1937
1nternV1.2-88 [6
58.3
61.5
$1.8
83.3
39.7
2210
MinCCPM-V226-81
60.6
48.1
2348
Decpsck-WLL2-M0E-44
48
62.8
51.1
86.0
2253
Reasoning Mode!
LLaVA-CoT-IIBB
54.8
57.6
47.8
LLaVA-Reusoner-31
50.6
54.0
40.0
83.0
Insight-V-8B 49
49.8
57.4
42.0
77.4
2069
LLaVA-NcXT--B 50
37.5
42.1
41.7
69.5
22.7
33.4
0.6
1957
39.7
Mulheny--.LaVA-B0
56.3
54.5
43.0
79.5
34.1
47.5
18.9
2021
50,7
Llama-32-1IB-V-Ins..
48.6
49.8
41.7
83.4
34.3
40.3
4.1
1787
45.8
Mulbeny-Llana-lIe
61.1
58.5
45.6
83.5
37.2
48.9
18.7
2035
53,3781
Qwen2-V1.22 2
43.0
48.0
41.1
73.5
24.9
41.7
1.0
1872
42.5
Mulbeny-2B
51.7
51.3
42.0
77.7
30.0
44.9
13.9
2013
47.9541
Owen2-VL-7B 2
58.2
54.1
83.0
42.1
5.9
2327
54.7
Mulberry-B
63.1
61.3
55.0
83,9
45.1
54.1
23.7
2396
MLLMs. The evaluation spans 8 widely used and challenging datasets, covering the fields ranging from general and mathematical reasoning to hallucination and multi-disciplinary understanding and reasoning in Tab. !
Comparison with baselines. We first conduct experiments baselines Qwen2- Owwn22-VL-7B and LLaMA- 3.2-11B- De--ViioonIIntueee that are involved in collective learning of CoMCTS for joint reasoning- path conjecture, search and identification. Trained with jointly-searched Mulberry-260k data, Mulberry- 7B and Mulberry-1 1B bring clear performance improvements against their baselines, i.e., +4.2% Qwen2-VL.-7B and +7.5% over I.LaMA-3.2- 1-B--11---------- 1 -i-Visiion-nnttruct averaged validating the effectiveness of CoMCTS. On the other hand, we examine the generalization of Mulberry-260k by applying it to train other models not involved in CoMCTS, such as W and LLaVA-NeXT-SB. Trained with Mulberry-2600 Mulberry-22 and Mulberry-8B enhance their baselines with +5.4% and + 1.0% gains, respectively, averaged on 8 benchmarks, demonstrating the generalization of CoMCTS.
benchmarks,
Comparison with ressonig-rrespone models. We benchmark Mulberry with various state-of-tt of-the art reasoning-response models. Using the same base model LLa.A-NeeT--8 (50). our Mulberry outperforms LLL
and Insight- V-8B by +5.7% and +6.5% mathematical benchmark Vath and by +3.0% and +1.0% on multi-disciplinary benchmark MMMU, respectively. Besides, Mulberry11I surpasses LLaVA-COT-IID by +6.3% on Vasha under the same baseline LLaMA- 3.2-1IB- -i---iiiio---nttrue The great superiority of Mulberry is largely attributed to our CoMCTS, which conducts collective tree search and generates rich, explicit and well-defined reasoning nodes with flexible numbers of steps.
Comparison with state-of-the-arfs We compare Mulberry with popular state-of-the-art models, both open-source and source. The results Tab. show that Mulberry, trained on CoMCTS- searched data, outperforms most open-sourced MILMS and achieves competitive results against closed-source ones, demonstrating outstanding abilities in step-by-step reasoning and reflection.
4.4 Ablation Study
Ablation Study on CoMCTS. We conduct ablation studies with powerful GPT-4o as the baseline over 1K samples from Geo3K [52) and GeoQA-Plus (53] as shown in Tab. 2 As the core of CoMCTS, we examine how each model in the collective learning group contributes to the overall tree search performance. Tab. reppots the Search Success Rates, and bascline GPT-4o works not

---

Table 2: Ablation Study on CoMCTS. We study how each model in CoMCTS collective learning con- tribute to overall tree search performance in Search Success Rate (S.S.R.).
Table 3: Ablation Study on Mulberry. As Mulberry is trained with effective and reflective reasoning data searched by CoM- CTS, we study their respective contribu- tions.
Direct Pred
CuMCTS
GPT-4o GPT-4oQeeu2-VL-28 L- Qwen2-VL-72B S.S.R.
Methods
Math Vista
58.2
Baseline
43.0
6388
66.2
w/o Reflection Data (+245K)
50.9
69.7
w/ Reflection Data (+15K)
51.7
very well without tree search. It shows that CoMCTS with only GPT-4o improves the performance to 63.8%, largely because our tree search designs like expansion, simulation and error positioning can work even without using collective knowledge. Besides, progressively involving more models into CoMCTS consistently improves the search performance, even with small models like Qwen2-VL. VL-7BB (i.e., +2.4%), demonstrating the effectiveness of CoMCTS in capturing useful collective knowledge not only with large models but also from small models. In final, the inclusion of all four models in the proposed CoMCTS performs clearly the best, i.e., 80.2%, validating the effectiveness of collective learning reasoning tree search.
Ablation Study on Mulberry. We train Mulberry with effective and reflective reasoning data searched by CoMCTS and study their respective contributions to overall reasoning performance. Results in Tab. BJn Math Vista show that incorporating reflection data enhances performance by 0.8%. demonstrating the complementarity of effective and reflective reasoning data searched by CoMCTS.
4.5 Discussion
Table 4: Comparison with other tree search meth- ods. "GPT-4o (direct)" refers to baseline without tree search. CoMCTS outperforms in search effectiveness and efficiency.
Comparison with other tree search meth- ods. We compare our CoMCTS with other tree search methods in search effective- ness and efficiency, including the base- line "GPT-40 direction prediction". "tra- ditional MCTS 77," "ReST-MCTS (24) that enhances MCTS by introducing par- tial search, and "Omega-MCTS 19]" that improves MCTS by designing binary search. Tab. 4 shows the results in search success rate and average search iteration that indicate search effectiveness and efficiency respectively. We can observe that existing tree search methods improve GPT-4o with limited gains. One main reason lies in that traditional MCTS methods generally work by seff-bootstrapping and often get trapped in homogeneous low-quality nodes within the reasoning space of single MLLM. On the other hand, CoMCTS shows great superiority in search effectiveness and efficiency, thanks to the joint expansion mechanism in CoMCTS that allows reasoning- given MLLM itself but also among those of others, benefiting from the synergy of multiple MLLMs while avoiding being trapped within the reasoning space of a single MLLM.
Methods
Search Success Rate
Average Search Iteration.
GPT4n (direct)
58.2
MCTS ReST-MCTS
63.8
42.1
65.6
36.3
Omega-MCTS
66.2
24.3
CoMCTS
80.2
12.7
search not only within the reasoning space of
5
Conclusion
This paper presents CoMCTS, a new learning-to-reant approach for MLLMs, which introduces the concept of collective learning into "tree search" for effective and efficient reasoning-path searching and learning. Based on the CoMCTS, we search effective and reflective reasoning paths for a set of multimodal inputs, and construct Mulberry-2600 with tree of rich, explicit and well-defined reasoning nodes for each question. Using Mulberry-266 we train our model, Mulberry, series of MLLMs with ol-like step-by-step Reasoning and Reflection capabilities. Furthermore, extensive experiments, ablation studies and discussion demonstrate the superiority of our proposed methods on various benchmarks. We hope that CoMCTS along with Mulberry-266 2600k and Mulberry will provides valuable resources and offer new insights for multimodal MCTS search and reasoning.
multimodal
soaming-to-reasonn wawasson-and-rffie -nd-reefeettt dataset

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[2] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -anguug model's perception the world any resolution. arXiv preprint wwX.xi:2209.1219990 2024.
[3] Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang. Ruoming Pang, and Yiming Yang. Improve vision language model chain-of- -fthoght reasoning. arXiv preprint arXiv:2410.16198, 2024.
[4] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:241.100440, 2024.
[5] OpenAL. Introducing openai ol. 2024.
[6] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy Lillicrap. Kenji Kawaguchi. and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405 1,0455 2024.
[7] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games. pages 72 83. Springer, 2006.
[8] Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller IIms stronger problem-solee solvers. arXiv preprint arXiv:2408 2411:9699... 2024.
[9] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng. Jiao Sun. et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0r 266.62522.. 2024.
[10] Mengdi Liu, Xiaoxue Cheng, Zhangyang Gao, Hong Chang, Cheng Tan, Shiguang Shan, and Xilin Chen. Protinvtree: Deliberate protein inverse folding with seward-guider tree search. arXiv preprint arXiv:2506.0092 2025.
[11] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepscek-vl: towards real-world vision- languag understanding. arXiv preprint M.Xi::2403.0552. 2024.
[12] Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song. Mengxi Zhang, Haocheng Feng. Yifan Sun. Zhi- heng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for milms. arXiv prepriat 140i::2400..800000 2024.
[13] Jingyi Zhang, Jiaxing Huang. Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via wise group relative policy optimization. arXiv preprint arXiv:2503. 2503.72937. 2025.
[14] Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang. Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, al. RI sharear Incentivizing reasoning capability of multimodal large language models via share-grpo. grpo. arXiv preprint arXiv:2505. 16673, 2025.
[15] Jingyi Zhang. Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[16] Huanjin Yao, Ruifei Zhang. Jiaxing Huang. Jingyi Zhang, Yibo Wang, Bo Fang, Ruolin Zhu, Yongcheng Jing. Shunyu Liu, Guanbin Li, et al. A survey on agentic multimodal large language models. arXiv preprint arXiv:2510. 10991, 2025.
[17] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang. Kai Zhang, Shengbang Tong. Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro pro: A more robust nulti-discipline multimodal understanding benchmark. arxiv preprint arXiv:2409.02811 2024.
[18] Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thour prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1420011441,, 2024.
[19] Bozhi Luan, Hao Feng. Hong Chen, Yonghui Wang, Wengang Zhou, and Houqiang Li. Textcot: Zoom in for enhanced multimodal text-rich image understanding. arXiv preprint 0xXiv:2404.09797, 2024.
10

---

[20] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 0122824-248377 2022.
[21] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[22] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 17682-17690, 2024.
[23] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems. 35:15476-15488, 2022.
[24] Dan Zhang. Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong. and Jie Tang. Rest-mets"; Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03811 2024.
[25] Sungjin Park, Xiao Liu. Yeyun Gong, and Edward Choi. Ensembling large language models with process reward-guided tree search for better complex reasoning. arXiv preprint arXiv:2412.15797, 2024.
[26] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-35) 2017.
[27] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488, 2021.
[28] Graeme Best, Oliver M Cliff, Timothy Patten, Ramgopal R Mettu, and Robert Fitch. Dec-mets: De- centralized planning for multi-robot active perception. The International Journal of Robotics Research, 38(2-3):31--337, 2019.
[29] Tuan Dam, Georgia Chalvatzaki, Jan Peters, and Joni Pajarinen. Monte-carlo robot path planning. IEEE Robotics and Automation Letters, ((4)311213-1 11220, 2022.
[30] Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ehner, and Xavier Martinet. Hypertree proof search for neural thcorem proving. Advances in neural information processing systems, 35:26337-2663 263499 2022.
[31] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Comera-Paredes Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schritwieserr Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
[32] Yelisey Pitanov, Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and Aleksandr Panov, Monte- carlo tree search for multi-agent pathfinding: Preliminary results. In International Conference on Hybrid Artificial Intelligence Systems, pages
Springer, 2023.
[33] Feiyu Yang. An integrated framework integrating monte carlo tree search and supervised learning for train timetabling problem. arXiv preprint arXiv:2311.00 1.00971, 2023.
[34] Harshil Vagadia, Mudit Chopra, Abhinav Barnawal, Tamajit Banerjee, Shreshth Tuli, Souvik Chakraborty, and Rohan Paul. Phyplan: Compositional and adaptive physical task reasoning with physics-informet skill networks for robot manipulators. arXiv preprint arXiv:2402.15770 2024.
[35] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92-100, 1998.
[36] Shiliang Sun and Feng Jin. Robust co-training. International Joural Pattern Recognition and Artificial Intelligence, 25(07):1113- 1126, 2011.
[37] Shipeng Yu, Balaji Krishnapuram, Rómer Rosales, and Bharat Rao. Bayesian co-training. The Journal of Machine Learning Research, 12:2649-2680, 2011.
[38] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co co-training for semi-supervised image recognition. In Proceedings of the european conference on computer vision (ecev), pages 135-152, 2018.

---

[39] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Ta Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018.
[40] Kaiwen Cui, Jiaxing Huang. Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genca: Generative co-training for generative adversarial networks with limited data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 499-507, 2022.
[41] Jakob Foerster, loannis Alcxandros Assacl, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi- reinforcement learning, Advances in neural information processing systems, 29, 2016.
[42] Aaron Hurst, Adam Lerer, Adam P Goucher. Adam Perelman, Aditya Ramesh, Aidan Clark. AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 2102222410. 2024.
[43] Anthropic. Claude 3.5 sonnet, 2024.
[44] Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning, arXiv preprint arXiv:2409 20566, 2024.
[45] Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024.
[46] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024.
[47] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui Hc, ct al. Minicpm-v: A gpt-4v level milm on your phone, arXiv preprint srXiv:2408.01800 2024.
[48] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu. Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixtureoof-exprts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412. 10302, 2024.
[49] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang. Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chon chain visual reasoning with multimodal large language models. arXiv preprint 144422411.14432, 2024.
[50] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei I Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, May 2024.
[51] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178 2024.
[52] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chur Zhu, Inter- gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguisties (ACL), 2021.
[53] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan I. Liang, Lingho Liu, Eric Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint 145552205514517, 2021.
[54] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282-293. Springer, 2006.
[55] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yonggiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
[56] Tianyu Yu, Yuan Yao, Haoye Zhang. Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-TTo Zheng. Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- -rained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807- 138166 2024.
12

---

[57] Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang. and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405. 207977 2024.
[58] Chuyang Zhao, YuXin Song. Junru Chen, Kang Rong, Haocheng Feng. Gang Zhang, Shufan Ji, Jingdong Wang, Errui Ding, and Yifan Sun. Octopus: A multi-modal um with parallel recognition and sequential understanding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[59] Pan Zhang, Xiaoyi Dong, Yuhang Zang. Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang. Linke Ouyang. Songyang Zhang. Wenwei Zhang. Yining Li, Yang Gao, Peng Sun, Xinyue Zhang. Wei Li, Jingwen Li, Wenhai Wang. Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng A versatile large vision language
Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim- lnternlm-xcomposer-- model supporting long-contextual input and output, 2024.
[60] Hugo Laurençon. Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? arXiv preprint arXiv:2405.0222 2024.
[61] Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint ar KKiv:2403.18814 2024.
[62] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arviv preprint arXiv:2302.0092. 2023.
[63] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong. Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of oftuhughht reasoning. In The Thirty-cight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[64] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufci Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint 11v:2312.113700 2023.
[65] Minjoon Seo, Hannaneh Hajishirzi. Ali Farhadi. Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference empirical methods in natural language processing, pages 1466-1476, 2015.
[66] Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint a7X1v:2212.02776, 2022.
[67] Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402. 18004 2024.
[68] Mchran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241 2023.
[69] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- Kiong Ng, Lidong Bing, and Roy Ka-Wai Lce. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406. 17294. 2024.
[70] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of IEEE conference on computer vision and pattern recognition. pages 5648-5656, 2018
[71] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
[72] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure datasct for visual reasoning. arXiv preprint arXiv:1710.07300 2017.
[73] Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IREE/CVF Winter Conference on Applications of Computer Vision, pages 1527-1536, 2020.
[74] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244. 2022.
13

---

[75] Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicyqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 1697-1706, 2022.
[76] Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hicrarchical tabular and textual data. arXiv preprint arXiv:2206.00 347, 2022.
177] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi- modal models via robust instruction tuning. The Twelfth International Conference on Learning Representations. 2023.
[78] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia. Yizhou Zhao, Wei Zhang. Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:21 2021.
[79] Pan Lu, Liang Qiu, Kai-Wei Chang. Ying Nian Wu, Song-Chun Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209,148 14610, 2022.
[80] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and clementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910. 2017.
[81] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language. visual and mathematical reasoning. arXiv preprint arXiv:2208 D.0x:22008.0538, 2022.
[82] Zhuowan Li. Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14963- 14973, 2023.
[83] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A datasct of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1110, 2018.
[84] Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xic. Pme- vqa: Visual instruction tuning for medical visual question answering. arXiv preprint uxiiv23055.00415 2023.
[85] Aniruddha Kembhavi. Minjoon Seo. Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Ha- jishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 1999-5007,-20 2017.
[86] Aniruddha Kembhavi, Mike Salvato, Erie Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[87] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507 2251, 2022.
[88] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference conferrrce computer vision, pages 2425 2433, 2015.
[89] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[90] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317 8326. 2019.
[91] Danna Gurari, Qing Li, Abigale/ J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018.
[92] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913 13, 2017.
14

---

Appendix
We provide theorem statement and assumptions Section A
implementation details in Section B more discussions in Section C and additional experiments in Section In addition, we discuss the relation to other studies in Section E provide the analysis of reasoning training data in Section F and
qualitative results in Section G Then, we provide the used prompts in Section H and the sources of the raw data in Section
In the end, we provide the analysis with error bars in Section and discussion about the limitations in Section[K
|
A Theorem Statement and Assumptions
RK be a collection of models used in Theorem A.1 (CoMCTS Consistency). Let { ; CoMCTS. Suppose the following conditions hold:
1. The search space is finite, or after pruning below a threshold t, remains effectively traversable.
2. The return Na of each node lies in bounded interval [a, b] C R.
3. Nonzero generation probubility and unbiased evaluation:
(a) For each model Whe there exists c > O such that the probability of generating an optimal branch is at least Ck-
(b) The collective evaluation function
K Σ
m (-|
R(s)
prompt evols Q, Parent(s),
has bounded bias, with controllable variance, approximating the true node value ps-
4. UCB Parameter: In the Selection phase, UCB strategy with >> 0 is used.
5. Repeated- Visit Opportunity: All candidate nodes above threshold be revisited with positive probability; thus, as - oc, each node not pruned is sampled infinitely often if it remains promising.
Under these assumptions, as T oc, the CoMCTS algoritlun visits the optimal path infinitely often, and the value estimate (s) converges the true value Psg for each node.
Proof of Theorem
Lemma A.2 (UCB Consistency 544 In the multi-armed bandit setting with bounded i.i.d. rewards, the UCBI strategy
UCB(k) Xx cV Inn n
ensures that each suboptimal arm is pulled only finitely often, while the optimal arm is sampled infinitely often. Consequently, Xx - μi as n 0o.
Lemma A.3 (Ensemble Near-Unbiasednes: and Variance Control). Suppose p PA unbiased (or bounded-bie -iia) estimators of some μ., each with bounded variance. Then their average remains close to μ in expectation and can significantly reduce variance compared to single estimator.
The proof proceeds by showing four key properties:
1. Optimal children appear in the tree due nonzero expansion probability (c: 0).
2. Optimal nodes are not pruned, via Hoeffding's s inequality demonstrating that R(s*) > t with high probability if μ.,
3. UCB selection guarantees repeated visits for discovered optimal nodes.
4. Value estimates converge to the true values μt under repeated sampling.
15

---

Optimal Children Appear in the Tree
Let s" be child node on the unknown optimal path. Each model my has probability ek of generating s" during Expansion. If these events are approximately independent across k, the joint probability of including s* is
II
(1 ck) 0.
Pexpansion
Hence, after enough expansions, s" almost surely appears in the CoMCTS search tree.
Optimal Nodes Survive Threshold-Based Pruning
Suppose s" is truly optimal, with pe- >> t. In the Simulation and Error Positioning step, CoMCTS computes
K
R(s") = T
Σ
prompteval Q, Parentts(s" );.²)
K
[=1
Let each individual score be X₁ within bounded interval. Under pear-unbiasedness (Lemma A.3 B[)())) ss par
[[X;] ~ po,,,
Define d po t O. Then
K Pr
K
Pr[R(ss) < t] = Pr Σ
< Xi<t
Xi
Σ Xi K - K6.
1=1
By Hoeffding's inequality, for X₁ and approximate independence, there holds
Pr Σ
X1
< Ki] < exp( K 8²).
Therefore, a truly optimal node s* with p4 t pruned with exponentially small probability, implying s" remains in the tree almost surely.
UCB Selection Guarantees Repeated Visits
Nodes not pruned enter the Selection stage, using
In (N))
UCB(s) = V(s) + e 1 Nss)
where is the parent of s. By analogy to multi-armed bandits (Lemma A.2 if is the highest-value child, it gets chosen infinitely often. Hence N(s*) T cc.
Value Estimates Converge
When node 8 returns score R(s), backpropagation updates:
V($) N({) (3) + E ExeChia(s) R(si)
N(3) CountChild ( Candiddnte $)
N(6) N(6) + CountChild St Standidate 8)
As (s*)) oo, the empirical average V(s*) converges to Ma- by standard arguments (e.g. the law of large numbers for bounded returns). Ancestor nodes on the optimal path also receive increasingly accurate updates, ensuring (ss) (4s.
Combining all above results, we conclude that CoMCTS inherits the same consistency guarantees as standard MCTS, with optimal paths ultimately visited infinitely often and V(s) + M..
16

---

B Training Details
B.1 Implementation Details
CoMCTS implementation Details. We implement collective learning in CoMCTS with four models, including GPT-40, Qwen2-VL- VL-7B, LLaMA-3.2-1 112--11B-/ 11-11B-iiss -B-Vision-Instrucct and Qwen2-VL-72B, to construct Mulberry-260k. In CoMCTS, we set maximum search iteration to 20 and threshold in tto O. In each iteration, each model generates one candidate reasoning path to balance search exploration and exploitation. In CoMCTS, we set the temperature to 0.9 for all models to balance creativity and determinism. For all other hyperparameters, we use the same default values as in the respective base models.
Model training details. In this paper, we use Mulberry-2665 -2okerryyy6KKKCoo data searched by CoMCTS to train the Mulberry series models based on four popular MLLMs, including both the search models included in CoMCTS (i.e., Qwen2- VL-7B and LLaMA-3.22 1B- Vision-Instruct) and the models not within the search (i.e., Qwen2-VL. VL-2B and LLaVA-NeXT-SB). We use the LLaMA-factory training framework [55] to train our models, using 8 NVIDIA H100 GPUs.
For the training scheme, all models are optimized for 2 epoch with the AdamW optimizer and cosine learning schedule. We fine-tune the models with batch size of 128 and employ SpepSpeed ZeRO-3 strategy to optimize memory consumption. A unified learning rate of le-5 is used for VA- NeXT-8B and LLaMA-3.2-1IB- 1---------iiii------ Instrut- while for Qwen2-VL-2B and Qwen2-VL-7B, the learning rates are adjusted to 2c-5 and 5c-6, respectively. As the number of tokens in the responses is significantly greater than that of direct simple answers, the training time increases accordingly, as detailed in Tab 5
B.2 Training Loss Curve
We provide more in-depth training details regarding the training loss curve in Fig. 4 Using the Mulberry-2600 dataset for collective supervised fine-tuning, consistent and stable training achieved across all four models, with the training loss steadily decreasing, demonstrating the training stability of Mulberry-260K searched by CoMCTS.
1.0
00
0.7
0.6
03-
3030
2000
2002
4000
1000
2000
2060
3:402
Mtpp
(a) Training Loss of Mulberry- -Larr---LLVVAAB- Model
(b) Training Loss of Mllmery-LLama- 1 IBB Model
argral
09
moothe
sncethed
09
0.-
8
0.6
0.-
0.-
05-
0.4
2008
3000
(c) Training Loss of Mulberry-2D -2B Model
(d) Training Loss of Mulberry-7B Model
Figure 4: Training loss curve.
17

---

Table 5: Training time. We provide the training time for Mulberry series models trained on Mulberry- 260K data generated by CoMCTS. The experiments are conducted on 8 NVIDIA H100 GPUs, using
Llama-Factory framework [55]
Metrics
Mulberry Mulbemy-LLavA-ae LLVVA-8B
Mulberry-Llmma-ll
Mulberry-2B
Mulbery-7B
Training Time (hours)
12.5
11.2
4,5
6.0
C More Discussions
C.1 Detailed Comparison with the Existing COT Training Dataset
We provide detailed comparison of Mulberry-260K with the existing COT training dataset (i.e., and LLaVA-CoT- 100K) in Table Table compares key attributes such dataset size (number of images and QA pairs), average token length of questions and responses, and the inclusion of reasoning and reflection. As shown, Mulberry-260k not only includes both reasoning and reflection, but also provides longer and more informative responses, underscoring its value as CoT training resource.
Specifically, unlike LLaVA-Reasome which generates CoT reasoning by directly prompting GPT-4o, Mulberry- 260K employs the proposed CoMCTS search, producing step-by-step reasoning paths that are not only longer but also feature more precise intermediate steps, benefiting from collective expansion and error positioning in CoMCTS.
Different VLaVO-,,T, which structured reasoning with fixed flow of certain pre-defined stages, Mulberry-260K leverages the proposed CoMCTS to dynamically search for both reasoning and reflection data, resulting in detailed and logically coherent intermediate steps.
Furthermore, thanks to CoMCTS, Mulberry2660K is the first multimodal step-by-step reasoning dataset equipped with reflective capabilitiis, fostering deeper exploration of reflection in MLLMs.
Table 6: Detailed comparison with the existing COT training dataset.
Datascts
Images
QA-Pairs
Avg Question
Avg Response token
Reasoning
Reflection
LLaVA-Reasoner
404K
404K
31.4
122.0
LLaVA-CoT-HO0K
100K
255K
23.2
212.4
Y
N
Mulberry-260K
260K
~~60K
31.9
234.5
Y
Y
C.2 Correctness of the Generated Reasoning Paths
As discussed in Limitation, neither our method nor advanced reasoning models (e.g., ol, QvQ. DeepSeek-R1) can ensure every step is 100% correct. Similar limitations are noted in QvQ. Here, we provide the manual evaluation of 100 reasoning paths. As shown Table our CoMCTS yields much lower enror rate in reasoning paths compared to MCTS.
Table 7: Manual evaluation of the correctness of the CoMCTS -WCTS--eeeerteed reasoning paths.
Methods
Reasoning Path Error Rale
MCTS
42%
CoMCTS
18%
C.3 Training MILMS on CoMCTS vs. Other Tree Search Data.
We examine the quality of the CoMCTS generated data by providng the experiments using data generated from standard MCTS. As shown in Table models are trained on 26K samples searched by MCTS and CoMCTS, respectively. By incorporating collective knowledge, CoMCTS generates more accurate reasoning paths, leading to 2.1 % performance improvement.
18

---

Table 8: Training MLLMs on CoMCTS vs. other Tree Search data.
Methods
Math
LLaMA-V-11B
48.6
MCTS
50.3
CoMCTS
52.4
D Additional Experiments
In this section, we include additional experiments on CoMCTS and Mulberry models. Unless otherwise specified, the experiments in this section are based on the Qwen2-VI. [2] model.
D.1 More Detailed Ablation Study on CoMCTS
We provide more detailed analysis search success rate in the CoMCTS ablation study Table g studying the direct predictions of search models and their performance in CoMCTS. We can observe that due to the model's limited reasoning capability, directly performing reasoning prediction with single model yields poor performance. When using only Qwen2-V1.-7B, the success rate of direct predictions is just 37.4%, and even the powerful closed-source model GPT-4o achieves only 58.2% accuracy in direct reasoning predictions. However, by leveraging tree search and collective knowledge, our proposed CoMCTS significantly improves the search success rate to 80.2% with four search models, demonstrating the effectiveness of CoMCTS.
Table Ablation Study on CoMCTS. We study how each model in CoMCTS collective learning contribute to overall tree search performance in Search Success Rate (S.S.R.).
Direct Prodiction
CoMCTS
S.S.R.
GPT-4o Qwen2 VL. Llama3.2 Qwen2 VL.72B GP Qwen2 VL.
Llama3.2 11B
Qwen2 VL-72B
58.2
37.4
35.4
48.1
63.8
66.2
69.7
80.2
D.2 Parameter Studies
Threshold f. We conduct parameter study on the threshold of Equation in CoMCTS, analyzing its impact on search success rate and average search iterations, as shown in Table [0] In the CoMCTS tree, the node value V represents the quality this reasoning step, and the threshold determines whether the node is removed based on its value, thereby regulating the overall quality of nodes. Setting to a higher value, i.e., 0.2, enhances the quality at each step but also slows down expansion of nodes. As the quality of each node improves, search models are more likely to find the correct path within the collective search space, lcading to an increase in search success rate. However, node expansion slows, the average search iterations increase significantly. Setting t smaller value, i.e.. t -0.2, increases the likelihood nodes containing irrelevant information, disrupting the reasoning process. This results in slight decrease in search success rate and a modest increase in average search iterations. On the other hand, setting the threshold as zero, i.e., t 0, achieves a great trade-off in search success rate and average search iterations, balancing search effectiveness and efficiency.
we also conduct extra hyperpa- rameter studies (e.g., temperature and repetition penalty), indicating our method is tolerant these parameters.
Temperature and repetition penalty. As shown in Tables I 12
19

---

Table 10: Parameter Study of Threshold t. We study the impact of different threshold values in Equation of CoMCTS on the search success rate and average search iterations.
Threshold
Search Success Rate
Average Search Iteration
-022
79.8
12.9
0
80.2
12.7
0.2
80.9
16.3
Table 11: Parameter Study of Temperature.
Temperature
Search Success Rate
0.7
80
(we use)
1,0
1.1
78
D.3 Discussion on Step Separators
We conduct experiments to discuss the impact of different step separators in Table 13 First, the absence of step separators results in disorganized reasoning logic, leading to notable decline in model performance, with an accuracy of only 48.4% on Math This result highlights the importance of step separators in reasoning, as they enhance MI.LMs' ability to comprehend the logical structure within reasoning paths. Next, we compare the special tokens separators (e-g., <step_1>, <step_21 ___ etc) with context separators (e.g., ### as illustrated in Figure The former requires adding special tokens to the vocabulary, while the latter relies on predefined formatting rules. The results in Table [3 show that the context separators method outperform the special token separators, achieving accuracy of 51.7% compared to 50.6%. This performance difference may stem from models' inherent tendency to use markdown formatting (e.g.. ####) to highlight answers and separate steps, which could lower the cognitive associated with learning this pattern in limited data.
D.4 Discussing the Importance of Reasoning
We discuss the importance of step-by step-by-step reasoning in Tab. 14 Specifically, using the same 260K multimodal data, we compare the results trained with direct answers to those trained with step-by-step reasoning searched by CoMCTS. Experimental results show that training with reasoning responses can substantially enhance model performance. For example, on Math Vista, it outperforms direct answer training by a margin of 4.9%. This improvement demonstrates the potential of reasoning to enhance MLLM performance.
D.5 Training with Different Proportions of Mulberry-260K
we conduct the suggested studies by training models using different proportions of Mulberry-2660 data over the base model
As shown in Tab. 15
We can observe that Mulberry consistently improves performance across all Mulberry-260K data proportions, demonstrat- ing its effectiveness limited-data scenarios. Besides, the performance gains increase steadily as more Mulberry-2608 data is used, highlighting its strong scalability.
D.6 Comparison with Base Models Trained using Other COT Data
As shown in Table T6 we conduct the comparison for analyzing the impact of different CoT training datasets on the same base model. Using the LLaMAA--LaVA-NeXT--- base model, despite using less training data, our Mulberry-LLave performance, which is largely because LLaVA- learns reasoning responses directly distilled from GPT-4o while our Mulberry learns from more effective reasoning data searched by the proposed CoMCTS.
outperforms LLaVA- Reassnerr--rr by +3.1 % average
Similarly, with the L.MMA-3.2 2---iiiio--11--eae base model, our Mulberry-LLaMe. better results using only 100K QA pairs, significantly fewer than the 255K QA pairs used by
achieves
20

---

Table 12: Parameter Study of Repetition Penalty.
Repctition penalty
Scarch Success Rate
0.8
78
0.9
79
1.0 (we use)
81
1.2
Table 13: Discussion on Step Separators. We study the impact of different reasoning step separators on model performance.
Methods
Vista Accuracy
w/o Separators
48.4
Special Token Separators
50.6
Context Separators
Table 14: Discussing the Importance of Reasoning. We compared the results of models trained on direct answer data with those trained on Mulberry-2600 reasoning data.
Benehmark
Baseline
Direct Answers
Step-by-step Reasoning
MathVista
43.0
46.8
$1.7
LLaVA-COT-IIB. When scaling our dataset to 260K QA pairs, Mulberyy-LLaMA-IIB outperforms LLaVA-CoT-IIB by +2.7% average, further demonstrating the advantages of CoMCTS-searched reasoning paths over the pre-defined structured reasoning methods in LLaVA-CoT.
E Relation to Other Studies
E.1 Relation to Other MCTS Methods
We summarize the contributions of CoMCTS and highlight the differences compared to previous MCTS methods [24 9 7 (a) This paper presents CoMCTS, new learning- toaiiingt--rrsonn approach for MLLMs, which introduces the concept of collective learning into "tree search" for effective and efficient reasoning- -ath searching and learning. (b) The joint expansion mechanism enables CoMCTS to concatenate reasoning trajectories from multiple MLLMs via iterative search, ultimately constructing a unified reasoning tree comprising diverse and complementary reasoning nodes. This allows reasoning-pale searches across multiple models, leveraging their synergy while avoiding traps in low-quality, homogeneous nodes within single MLLM's 's reasoning space. (c) In each search iteration, CoMCTS skips multiple intermediate steps and selects the last correct step as the next start node in the joint simulation and error positioning operation, largely reducing search time while maintaining search effectiveness. Additionally, collective knowledge enhances error positioning, allowing models to better identify errors through mutual validation.
E.2
Relation to Other Reasoning MILMS
Multimodal large language models 55
57
58
59
60
61 have made remarkable progress,
with recent MLLMs increasingly focusing on reasoning and intermediate steps [62 63 4 3 49
We summarize the contributions of our models and highlight the differences compared to previous reasoning MLLMs: (a) Based the CoMCTS, we search effective and reflective reasoning paths for multimodal inputs, and construct Mulberry-260k, a multimodal reason and reflect dataset with a tree of rich, explicit and well-defined reasoning nodes for each question. Mulberry-2600 is then used to train reasoning model Mulberry. (b) Unlike pre-defined stage-basee based methods, CoMCTS searches for step-level reasoning data of varying lengths. Training Mulberry models on these CoMCTS- reasoning paths equips it with the ability to perform flexible step-by-step reasoning. (c) To the best of our knowledge, Mulberry the first work to explore the reflective capabilities of MI MILMMs through tree search and collective knowledge.
21

---

Table 15: Training with different proportions of Mulberry-260ks
Models
Percentage of Mulberry-2000
QA-Pairs
Math
MMStar
MMMU
AVG
LLaMA-3.2-V1118 (Baseline)
48.6
49.8
41.7
46.7
10%
~226
52.4
53.1
42.8
Mulberry-11B
30%
~88
54.0
56.1
43.6
50%
~ 130k
58.1
57.9
44.9
53.6
100%
~ 260k
61.1
58.5
45.6
55.1
Table 16: Comparison of training with different CoT datasets on the same base model.
Models
Base Model
CoT Dasa
PAiPairs
MathVasa
MMStar
MMMU
AVG
LLaMM33 MAAMMA--LLNA--- NcXT-BB
37.5
42.1
41.7
40.4
LLAMAS-LLaVA NEXT-BB
LL Ressomer-10MK
Mulberry LLaVA-KB
LLaMAS-LLaVA-NANN -
Mulberry-600
-660
56.3
54.5
43.0
$1.3 (+10.9)
1LLMA-3 1is...22-iinnn1-----nnt
48.6
49.8
41.7
46.7
LLaVA-CoT-1IB
LLaMA-3.2 Vision (itioon-11B-aa Instract
M-VA-CTT-100
~255K
54.8
57.6
44.9
52.4 (+5.7)
Mulberry- Liama-lll
LLaMA 3.2 3.2-Viioo-- (ation-11B-a [IB-rastract
Mulberry 100K
~100K
56.8
57.8
44.7
53.1 (+6.4)
Mulberry-
LLaMA-3.2 Vision-118
Mulberry- 260K
-260K
F
Analysis of Reasoning Training Data
F1
Analysis of Reasoning Data Distribution.
We analyze the CoMCTS-searchees reasoning paths in Mulberry-200K by examining the distribution of reasoning steps, as shown in Figure 5 shows that reasoning steps predominantly fall between and 8, with average of 7.5 across the entire Mulberry-260k. For simple reasoning tasks, the chart-related subset of Mulberry-28e 260k,, reasoning steps typically ranges from 6 to 7. averaging 6.8. contrast, for complex mathematical and logical reasoning tasks, such as the geometry- subset of Mulberry-260k, the distribution shifts to 7 and 10 steps, with an average of 8.9. These observations highlight that CoMCTS's collective tree search design generates effective reasoning trajectories with flexible numbers of steps, learning from which allows to train powerful MILMM with reasoning flexibility, ie., a model can "think less and faster" when handling simple questions (i.e., allocate and generate fewer intermediate reasoning steps) and "think more and slower" when tackling complex tasks (i.e., allocate and generate greater number intermediate reasoning steps).
(a) Mulberry-260K
(b) Mulberry-260kssubset
(c) Mulberry-260k-subset chart
50%
Percentage (S3
Parcentage (957
60%
Percentage
30%
20%
30%
10%
5%
12%
0%
30
12
24
16
0%
30
24
1.
0%
30 *SSepp
12
34
Stepp
#Steps
Figure 5: Distribution of reasoning steps in Mulberry-260K data.
F.2
Analysis of Effective Reasoning Training Data
We provide example of effective reasoning data generated by CoMCTS in Figure 6 In this reasoning path, an accurate image analysis is first searched, along with reasonable rationales, allow- ing CoMCTS subsequently search logical and correct step-by-step reasoning based precise perception and rationales, ultimately deriving the correct answer. This clear reasoning largely due CoMCTS's collective expansion and collective enror positioning, allowing search models to complement each other, refine error identification, and iteratively improve reasoning paths.
F.3 Analysis of Reflective Reasoning Training Data
We provide an example of reflective training data generated by CoMCTS Figure In this reasoning path, one search model identifies correct reasoning path after several iterations, while CoMCTS's
22

---

collective enror positioning finds perception error at Step I in another path. CoMCTS then integrates the incorrect Step into the correct reasoning path (i.e., before its sibling nodes) with reflective prompt create reflective data. Here, incorrect Step 1 misjudges the proportion of "Medium Orchid", but later steps identify and correct the error. Fine-tuning with this reflective data equips the model with the ability to reassess previous steps when generating the next step, leading to more accurate reasoning and improves performance.
G Qualitative Results
G.1 Qualitative Comparison
We provide a qualitative comparison of LLaVA-NeXT-8B [50], Qwen2-VL-7B (2], and Mulberry-7B in Figure [8] LLaVA-NeXT-8B and Qwen2-VL-7B generate relatively short predictions without thorough thinking, leading to incorrect answers. contrast, our Mulberry, trained with CoMCTS- searched reasoning data, creates rich, explicit and well-defined reasoning steps with comprehensive understanding, ultimately arriving at the correct answer.
G.2 Reflective Qualitative Results
We present reflective qualitative results of Mulberry in Figure 9 Fine-tuning reflective reasoning data, Mulberry models develop the capability to reflect on previously generated step. Here, Mulberry initially misidentifies the accuracy of the parent but, through continuous reflection, detects the error at Step 2, corrects it, and ultimately arrives at the correct answer.
4dti Image Description:
The image depicts Alaohua standing next to flappole with the shadows of both Xischus's shadow maried measuring meter, while measuring
flegoole's shadow
ene Rationales:
height the properties similar triangles, question weeks height flagpoi.
problem provides messurements
step by step.
identily the relationship between Xizohus's height and her shadow. The ratio Nachuua's gigtt to her shadow is 1.6.1.
4ua shadow same Xichhu''s
Question: K'iachua's shadow measures units at the some time the flagpol's shadow measures 6.D unitt, and Xirohus's height a approsimately 1.8 uniut what the estimated of flagpole?
some ratio flagpole
height
flagoole heght
Set up the proportion using known quantitles:
Xieohue's Height Elagoole's Height Xischna's Shadow
Flagoole's Shiddw
Correct Answer:
values:
t8t Stop S:
for (lagpole's heightt::
Figure 6: An example of effective reasoning training data.
23

---

BAR Description:
segment a label indicating its color, The colors include Gray, Forest Green, Medium Orchid, Comnlower, Gold, and Red.
chart into sie segments representing
titt Rationales:
The pie chart visuallyy represents proportions
oxtegory (color).
The size of each slice ind cates how much of the whole each category accupies.
Since the question asks Medium Orchid is minimum,, need slice against others.
Porest
Medium Orcaid
tan
step by step.
Good
idientilly the sives of all slices.
Red
-Gray: Smallest sitce
smallest
-Meilum Orchid: Third smallest slice
-Cortllowrr Fourth largest slice
-olld: Fifth largest slice
Largest
Question: Medium Orchid minimum?
tat Step
Identify the slice labeled "Medium Orchid" on the pie chart.
Correct Answer: Yes
*a8
Medium Orchid Forest Green, Cornllower, Golc, Red).
compare
other slices
tät Step
Determine if the Medium Orchid slice is the smallest among all the slices.
any tad The final answer Is:
compariion
Orchid stice.
Yos
Figure 7: An example of reflective reasoning training data.
LLaticc We need find the derivaciov vee of 150 o012 and compare it with the derivative offil snd We can do this by using the power rulk
Muliery:
Image Descr pelor:
diggomm divudar oetor being beingrnnddemee tredtreeed lnto mome
difibre on,oon, which states that fill sxch, thon Pogo nehtn Lt Acama, (12) 24g 1) #t5, r(B) EAin 1)
withour The has central angle isteled
a racius loagth
Therefore, derivative thal largger than ahatatseá. Thu correct answer
dircnnforence
understa
Question: Use sector poper sheet w th centre angle and radiss 5.0 roll into bottomles peper sap jaushown in
form
QwenZ-VL: Fire, jen the eiroumieonnee of the
becomes dicunnfrrecce
bose of the conc. circaniferenn ourence of the base equal the wegth of the are of the sector paiper sheet.
nvn Laths think step by step.
asi Stop 1:
Diioubee length offite wethor which
pictuell
mude for ciroumference wherer redius in
diccmiee movee the
Aer Length central argle/3t00 2ns Heem mea and central amgle
bottom perimeter
poper
poper whish
2 3
46-4 Snrm
Choices:
dirunrerrrnce
Simolify
Comeent Answer
However, quettontion asks for bottom ienter the paper which the dicumference of the base of cone. Therefore. the bottom perimeter of the paper an12
Porform calculations: Arc Length 1/3 x 12x An. This meons the chrurfference te of cone's base equats 4n.
ase The final anower is.
So, the sorrect answer S
Qualitative Comparison. Our Mulberry, trained with SOMCTS-searcherr reasoning data, creates rich, explicit and well-defined reasoning steps with comprehensive understanding, ultimately arriving at the correct answer.
Figure 8:
24

---

Hitt imae Descriction
A bar chart shled "Acturacy of different a lggorthmms shows two bors representing
Azurssty of offerent goo thms
algor thmn accuscies. The fist bar labeled "array" has height around 10, while the
100
labeled Ratronales:
80
reoresents names shporihime (array parent).
The y ace meosures accuracy from 0 100. Each algrrithim has one cor respprddinng bar indicating accuracy level.
bast Lot's think step by step.
titt Step 1:
Identify the accuracy levels for each algorthm. Arroy, Approsimately 10%
Parent: Appr oximataly 95%
een step
90%.
carent
Only the "parent" algorithm exceeds this
mistake. ruthink
Question: algortthms have
Step
Compore these volues against the threshold of 90%.
Stop
Correct Answer: Zaro.
Conclude how many algorithms meet or exceed this criterion. Based on the comparison:
Annotation: (10), parent
-No algorithm reaches above 90%.
Zero.
Figure 9: Reflective Qualitative Results. Our Mulberry, trained with CoMCTS-searched reflection data, generated reflective reasoning, ultimately arriving at the correct answer.
NCTS
CalocTs::
Descrattie
The blingl ABC un C wake
dlagere wnegte oringll with right
triencs personttole regonent BC drops point side
Ande du andle du perppradioone sepmeet ee room
A The eeeeeeee andive the
bo
lor
onalee troped aeoog grice.
We de deadingtte with trange where engle CDC dov 001-0t Tharoliro the ciare hester eaparppiine aat. Pryptomas
rock act angie ert iihtt a angi couenite
(007
Muttipla rechels leverage ressering crrive the
percaphull
Correct Answer: 14v3
PRR Suip
iecamect reasiiit
4x58 sepor -
Sole the eqaston
(2)2-7
Sou falv and 14
aad Sup
23/247 / with 22/10
Figure 10: An example to compare CoMCTS with other tree search methods.
G.3 Examples to Compare CoMCTS with Other Tree Search Methods
provided a qualitative comparison between the CoMCTS and single model MCTS.
When single model performs step-by-step reasoning, an еттоr in the initial image understanding can lead to a cascade of incorrect reasoning steps, ultimately resulting when multiple models are used collaboratively for reasoning. if one model fails to correctly interpret the image, others can help identify the correct steps. By cross-verifying and complementing each other's interpretations, the ensemble can jointly search for and arrive at the correct answer.
wrong answer. However,
As illustrated in Figure TO the single-model MCTS failed correctly interpret the image and got trapped in an erroneous node, leading to a reasoning path that consistently failed to arrive at the correct answer.
H Prompt
In this section, we present the prompts involved in different operations of CoMCTS to execute various task instructions. For expansion in operation (a), we use the prompt in Figure [] and Figure [2 The
25

---

prompt in Figure [1 is used to expand the root node for the closed-soure model GPT-4o and all nodes for the open-source models. It is worth noting that, since open-source models can directly and forcefully incorporate preceding reasoning steps, allowing the model to generate subsequent steps based on previous responses, whereas closed-source models cannot. Therefore, we design specific prompts for closed-soure models to expand non-root nodes. For GPT-4o, we use the prompt shown in Figure 12 to expand non-root nodes the existing search tree and generate subsequent reasoning the prompt shown in Figure 13 to evaluate the correctness of each step. Each step is scored as correct, neutral, or incorrect, with scores of I, O, and -1, respectively. The evaluation results for each model are then combined through weighted average. Finally, we use the prompt in Figure 14 to verify if the final answer from reasoning path matches the ground truth.
steps. For error positioning in operation (b),
an image description on the question.
Then, provide rationale to analyze the question.
Next, generate a step reasoning process to solve the problem. Ensure the steps are logical
and
Finally, provide concise summary of the finel answer in the following format: 'The final answer is:
xxx. question multiple choice, provide options allong content. free-
form, directly present the final result. Do provide explanation.
your response with following sections, separated
#BR Image Description:
FAR Rationales:
WMA Let's think by
#AR Step 1:
Step
НMA The final answer is:
[QUESTION]
Figure 11: The prompt for expansion. The prompt expands root nodes in closed-source and all nodes in open-source models.
an image description
question.
Then, provide rationale analyze the question.
Next, generate a step st-step reasoning process to solve the problem, Ensure the steps are logical concise.
Finally, provide concise summary of the final answer in the following format: 'The final answer is: question multiple choice, provide options along
form, directly present the final result. Do not provide any explanation.
your response following sections, separated by
#SM Image Description:
AMH Rationales:
A&M Let's think
A#M Step 1:
Step
AMU The final answer is:
(QUESTION)
Please comp lete response based on the reasoning prefix without altering content.
Reasoning prefix: [REASONING_PPM (RREFKKKK))
Figure 12: The prompt for expansion. The prompt expands non-roo nodes closed-source models.
I The Sources of Raw Data
To construct comprehensive and general-purpose tree-based reasoning dataset, we collect 260K raw multimodal input questions spanning various domain, including
55K Mathematical Data: From GLLaVA
64
GEOS 65 UniGeo [66] GeoQA Plus 53
Geo3K [52] Math Vision 67), GeoMverse 668, and Math V360K [69]. These datasets cover
broad spectrum of mathematical problems, where solving requires extensive reasoning
26

---

Cuestion: (QUESTION)
Ground truth (GT_AN AGT_ANSO SWER)
Н#M Reasoning steps: [REASONING]
Given the question and reasoning steps listed ahove, along with the corresponding ground truth answer, please eve luate the correctness image destription, rationales, and each step of the process.
Requiremente ents:
Output the decision ("correct", "neutraf", "incorrect") for each following the format of Fiiaal Decisian:\ylmage Deseription: [your decision]; Rationales: [your decision]; Let's think step by step: yyour decision):
Iyour decision); Step Iyour decision); ..." Do not provide explanation.
Figure 13: The prompt for error positioning.
Evaluate whether the model's answer matches the correct result.
If does not respond with 'No':
If the model's answer aligns with the correct result, respond with Yes'.
Provide only 'fes' or 'No' the output, with no explanation.
The question is: (QUESTION)
The model's answer MODELLANNSWER)
The correct result is: (GT ANSWER)
Figure 14: The prompt for evaluating the final result.
and multiple logical steps, highlighting their value in Mulberry-260K, We apply CoMCTS search to the raw data from these datasets to generate reasoning paths, which are then used to train Mulberry models, equipping them with advanced mathematical and logical reasoning skills.
116K Figure Understanding data: From DVQA 770. Doc (71), FigureQA [72). PlotQA (73). CharlQA (74), InfoVQA (75), MultiHiert (76), and LRV-Chart [77]. These datasets cover various figure types and understanding tasks, including charts. bar graphs, pie charts, histograms, etc. Training on these data searched by CoMCTS equips the model with figure reasoning capabilities such as table computation, information retrieval, and trend analysis.
41K Math Word Problem Data: From IconQA 78 TabMWP 79,, CLEVR 80 CLEVR-
Math [8]] and Super- CLEVR [82). These datasets span various mathematical word reason- ing tasks. Training on them in Mulberry-266 strengthens the model's 's counting, arithmetic, and ogical deduction skills, enhancing its accuracy and interpretability in solving complex math word problems.
2K Medical Data: From VQA-RAD 33 and PMC-VQA
These datasets focus on medical visual understanding and reasoning, encompassing a variety of radiology images and diseases. Leveraging this subset enhances the model's 's capability in medical image comprehension and diagnostic reasoning.
84
These datasets in CoMCTS plays critical role in enhancing the model's ability tackle complex scientific problems, perform multi-modal reasoning, and interpret scientific illustrations and textual descriptions cohesively.
17K Science Data: From TQA 85), A12D 86), and ScienceQA 87
24K Nature World QA Data: From VQA-AS 88), A-OKVQA
TextVQA 90],
Vizwiz 191), and VQA2.0 (2]. These datasets encompass tasks involving naturalscenes, textual elements, and open-ended visual question answering, challenging models to interpret complex visuals, understand embedded text, and generate accurate responses. This subset
27

---

plays key role in enhancing the model's ability to reason about real-world visual and textual content across diverse contexts.
Error Bars
We provide the error bars of different Mulberry models the Math Vista benchmark. shown in Tab. [7 We conduct five repeated evaluations to obtain the error bars, and the results indicate that the error bars fall within narrow range.
Table 17: Error Bars, The experimental results are based on five repeated evaluations.
Benchmark
MulbenryLLLaAA Mulbeny-Llama-I1B
Mulbeny-28
Mulberry-7B
MathVista
56.3 1 0.3
61.1 1 0.2
51.7 0.3
63.1 0.2
K
Limitations
Mulberry is preliminary exploration work in ol-like MILM, leveraging Collective Monte Carlo Tree Search to enable effective and efficient reasoning-path searching and learning. CoMCTS leverages collective knowledge significantly improve the search success rate and efficiency of reasoning path search. By training on the reasoning data generated through CoMCTS, Mulberry has gained step-by-step reasoning capabilities, leading to a substantial improvement in overall performance. Nevertheless, certain limitations must be acknowledged.
Hallucinations in intermediate steps. Hallucinations still prevalent in MLLMs, whether in closed or open-source models. For instance, the models may generate obvious errors intermediate reasoning steps yet still arrive the correct final answer in CoMCTS. Therefore, although incorporated multiple models to better detect errors, some errors still persist in the intermediate steps because ensuring the correctness of all intermediate steps often requires human checks, which is extremely costly and unaffordable for us.
observed that models struggle to detect their own errors. To address this, CoMCTS employs multiple models to cross-check each other's errors. However, findings also revealed that smaller models often fail to generate effective detection responses, while larger models occasionally exhibit inaccurate error localization. Thus, inaccurate localization may impact the efficiency of the search and recommend using larger models for error localization or exploring better prompts to enable smaller models to localize errors more accurately.
Error localization. During experiments,
Reflective capability. Mulberry is an early-stage work in exploring reflective capabilities within the field of MI..Ms. We would like to clarify that since reflective data constitutes only a small portion of the entire dataset, the Mulberry model generates reflective responses occasionally. We hope to further explore the model's s reflective capabilities in future work.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our contributions and scope both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not include new theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Yes, we provide all necessary information to reproduce the experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
We will provide the anonymous code, and our code and data will be publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil wwwppbblccgguiicuceccccccoe.oopp /eeggudddsseCeebbbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars based on multiple runs on part of the benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. All assets are properly credited and used under their respective licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide anonymous URL.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
34

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, non- standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
35


Reviews:
[{"review_id": "X5SO67Aj7k", "reviewer": "Reviewer_Wx49", "summary": "The paper introduces Collective Monte Carlo Tree Search (CoMCTS), an ensemble-style extension of MCTS that aggregates proposals from multiple MLLMs. Aside from the usage of multiple models for generating and evaluating the steps, another innovation is to use some wrong nodes in the chain to simulate self-reflection via transitioning to the right step. CoMCTS improves the search success rate and reduces the number of iterations over standard MCTS. A large scale dataset is created for finetuning MLLMs, which leads to substantial performance improvements in several MLLMS on multiple standard benchmarks.\n\nOverall, this paper presents a significant advancement in search based multimodal reasoning by combining large model ensembles with MCTS and reflective data generation. The proposed approach is innovative, practical, and well-validated empirically, and the dataset in itself is a valuable contribution. Therefore, I recommend acceptance.", "questions": "I do not have any major questions, please address the concerns raised in the 'Weaknesses' section.", "limitations": "Addressed adequately in Appendix K.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n 1) A new large-scale multimodal dataset is constructed, annotated with structured reasoning trees, error paths, and reflection nodes. This provides rich training signals for developing compositional reasoning abilities in MLLMs. \n\n2) Mulberry demonstrates strong results across multiple multimodal reasoning benchmarks, outperforming open-source models and rivaling some closed-source systems. \n\n3) Ablations clearly demonstrate the utility of each CoMCTS component. The use of erroneous nodes encourages models to self-correct and engage in more human-like, iterative thinking, and combining steps from multiple models is more effective than using each of those individual models.\n\n\nWeaknesses:\n\n1) Running multiple MLLMs in parallel and pruning through simulations remains expensive. A comparison of necessary token usage for creating LLAVA CoT-100k and this dataset would be useful in understanding this aspect thoroughly.\n\n2) The organization of the experimental results could be better. For example, the contents in Appendix C.1 and C.3 are very interesting and it would possibly be better if they are included in the main paper. Otherwise Table 1 gives the impression that Mulberry is only compared to the base models which is not true. Highlighting its effectiveness in comparison to other ways of finetuning the MLLMs would strengthen the paper.", "was_revised": true, "final_justification": "I thank the authors for their response.\n\nAfter rebuttal: W1. The authors show that although CoMCTS uses more tokens/sample compared to LLaVA-CoT, it results in a considerably higher search success rate, this addresses my question about the computational requirement and tradeoff between token usage and efficacy. W2. minor suggestion, addresses satisfactorily.\n\nOverall, after reading the other reviews and rebuttals, my assessment of the paper has not changed considerably (if anything, the authors' rebuttals to other reviewers' questions strengthen the paper). Therefore, I keep my score to indicate an Accept.", "rebuttal": "Dear Reviewer Wx49,\n\nWe appreciate your valuable comments and suggestions.\nWe are encouraged by your recognition of our contributions in constructing a large-scale multimodal reasoning dataset, the strong performance of the Mulberry models, and the clear utility of each CoMCTS component shown in our ablations.\nWe have carefully considered all your suggestions and will revise the manuscript to include a more detailed discussion on token usage and improve the overall organization as suggested.\n\n---\n\n**W1: Comparison of necessary token usage.**\n\nThank you for your suggestion.\nWe agree that including a comparison of token usage will help provide a more thorough understanding of this area.\nTherefore, we provide a comparison between CoMCTS and other CoT data construction methods regarding search token usage per question, search success rate, and token usage per sample for successfully finding the correct answer, as shown in the table below.\nAlthough CoMCTS consumes more tokens during the search phase, it achieves higher search success rates and data quality, leading to superior performance of the Mulberry models.\n\n\n| Models                     | Token usage per sample | Search success rate (%) | Token usage per sample for successfully finding the correct answer |\n|----------------------------|------------------------|--------------------------|----------------------------------------------------------------------|\n| MCTS                       | 10796                  | 63.8                     | 16922                                                                |\n| LLaVA-CoT                  | 212.4                  | 57.1                     | 404                                                                  |\n| BoN (N=10) + LLaVA-CoT     | 2184                   | 61.3                     | 3465                                                                 |\n| CoMCTS                     | 2977                   | 80.2                     | 3713                                                                 |\n\n| Models | Dataset construction method | Avg accuracy |\n|---------|----------------------------|-------------------|\n| LLaVA-Reasoner-8B  | Direct prediction  | 46.1%            |\n| Mulberry-8B  | CoMCTS | 50.7% (**+4.6**)                        |\n| LLaVA-CoT-11B  | Direct prediction  | 51.1%                        |\n| Mulberry-11B  | CoMCTS | 53.3%  (**+2.2**)                    |\n\n\nAdditionally, we manually evaluated the correctness of reasoning paths in 100 cases generated by CoMCTS and the direct prediction method (e.g., LLaVA-CoT). The results show that by leveraging collective error localization, CoMCTS significantly improves the accuracy of intermediate reasoning steps.\n\n| Methods | Reasoning Path Error Rate |\n|---------|----------------------------|\n| Direct prediction   | 51%                        |\n| CoMCTS  | 18%                        |\n\n----\n\n**W2: Better organization of the experimental results.**\n\nThank you very much for your valuable suggestion.\nWe agree that incorporating the contents from Appendix C.1 and C.3 into the main paper could improve the clarity and comprehensiveness of our experimental comparisons.\nIn the revised version, we will move the results and analyses from Appendix C.1 and C.3 into the main text, and we will update Table 1 to clearly indicate that Mulberry is compared with both base models and other reasoning MLLMs using the same backbone.\nWe believe these changes help to better highlight the effectiveness of our approach and address your concern.\n\n----\n\nMany thanks for your professional, detailed, and valuable reviews! Please let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period."}, {"review_id": "bVpa8GvleR", "reviewer": "Reviewer_H5ne", "summary": "This paper develops Mulberry, a Multimodal reasoning model based on Collective Monte Carlo Tree Search. Specifically, the authors introduce collective learning into “tree search” for reasoning-path searching and learning, construct Mulberry-260k dataset, and then finetune a series of multimodal models, aiming to empower the models with reasoning ability. Experiments demonstrate the effectiveness of the models.", "questions": "1. Table 1 has too many missing values.\n\n2. For Table 2's comparison, what results might emerge if all four models were strong foundation models (e.g., GPT-4o)?\n\n3. Would the authors consider open-sourcing the training data and models?\n\n4. Some writing problems: L68 has an extra space. It is recommended to use \\citet{} on L123-125.", "limitations": "No", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well organized and easy to follow.\n2. This work introduces the Mulberry-260K dataset and Mulberry models, which will facilitate future research in multimodal reasoning and related areas.\n3. Research findings indicate that multimodal models, even with small sizes (2B parameters), can benefit from the reasoning and reflective abilities.\n\n\n\nWeakness:\nWhile this work provides a solid incremental contribution, the novelty and insights appear somewhat limited.", "was_revised": true, "final_justification": "The authors provided a clear and detailed rebuttal addressing my main concerns. While I initially felt the contribution was somewhat incremental, the rebuttal provided sufficient justification of novelty, especially around the collective learning mechanism in CoMCTS and its integration with MLLMs. I have accordingly raised my rating to 5 (Weak Accept).", "rebuttal": "Dear Reviewer H5ne,\n\nWe appreciate the reviewer’s valuable comments and feedback.\nThank you for recognizing the clear organization of this paper and for highlighting the contributions of the Mulberry-260K dataset and Mulberry models to future research in multimodal reasoning.\nWe are also grateful for your recognition that even small multimodal models can benefit from enhanced reasoning and reflective abilities.\nWe have carefully responded to each of your concerns and suggestions below, and will revise the manuscript accordingly.\n\n----\n\n**W1: The novelty and insights of CoMCTS and Mulberry Models.**\n\nThank you for your question!\nWe believe our work, including CoMCTS and Mulberry Models, offers unique innovations.\nBelow, we first summarize the key contributions of our paper, and then highlight the differences between CoMCTS and other MCTS-based methods, as well as between Mulberry and other reasoning MLLMs.\n\n**Contributions**:\n**First**, we introduce collective learning into MCTS and propose CoMCTS, which leverages collective knowledge to collaboratively conjecture, search and identify effective and reflective reasoning paths for MLLMs, significantly improving search effectiveness and efficiency.\nTo our knowledge, this is the first work that explores collective learning with MCTS for MLLMs.\n**Second**, we construct Mulberry-260K, a valuable resource for step-by-step reasoning and reflection research in MLLMs.\n**Third**, we develop Mulberry, a series of MLLMs with strong step-by-step reasoning and reflection abilities.\n**Fourth**, extensive experiments validate the superiority of our methods across various benchmarks.\n\n**Relation to Other MCTS Methods**:\nWe summarize the contributions of CoMCTS and highlight the differences compared to previous MCTS methods (e.g., Traditional MCTS, ReST-MCTS):\n(a) CoMCTS introduces collective learning into tree search for MLLMs, enabling more effective and efficient reasoning path searching and learning.\n(b) Its joint expansion mechanism allows CoMCTS to combine reasoning trajectories from multiple MLLMs through iterative search, constructing a unified reasoning tree with diverse and complementary nodes. This leverages the strengths of different models while avoiding traps in low-quality, homogeneous nodes within a single MLLM's reasoning space.\n(c) In each search iteration, CoMCTS generates a complete reasoning trajectory and selects the last correct step as the new starting node, skipping intermediate steps to significantly reduce search time without sacrificing effectiveness. Furthermore, collective knowledge enhances error localization by enabling mutual validation among models.\n\n**Relation to Other Reasoning MLLMs**:\nWe summarize the contributions of Mulberry models and highlight the differences compared to previous reasoning MLLMs:\n(a) Using CoMCTS, we search for effective, reflective reasoning paths for multimodal inputs and construct Mulberry-260k, a dataset containing rich, explicit reasoning trees for each question, which is then used to train the Mulberry models.\n(b) Unlike pre-defined stage-based methods (e.g., LLaVA-CoT, LLaVA-Reasoner), CoMCTS searches step-level reasoning data of varying lengths, enabling Mulberry to perform flexible step-by-step reasoning.\n(c) To the best of our knowledge, Mulberry is the first work to explore the reflective capabilities of MLLMs through tree search and collective knowledge.\n\nWe hope these clarifications help to highlight our contributions and the novelty of our work.\n\n----\n\n**Q1: Supplementing the values in Table 1.**\n\nThank you for your suggestions.\nSome values are missing because the original papers did not report results on certain benchmarks, which resulted in incomplete values in our Table 1.\nMany thanks for your feedback. We have now conducted additional evaluations on these models ourselves to provide a more comprehensive comparison.\nDuring the rebuttal period, we present a part of results in the table below, which shows that the Mulberry models outperform other reasoning MLLMs across the overall benchmarks using same base MLLMs.\nThe remaining missing values will be included in the revised paper.\n\nWe believe offering these missing values would improve the completeness of our comparisons. Thanks!\n\n| Models                | MathVista | MMStar | MMMU  | ChartQA | DynaMath | HallBench | MM-Math | MME_sum | AVG   |\n|-----------------------|-----------|--------|-------|---------|----------|------------|---------|---------|-------|\n| LLaVA-NeXT-8B         | 37.5      | 42.1   | 41.7  | 69.5    | 22.7     | 33.4       | 0.6     | 1957    | 39.7  |\n| LLaVA-Reasoner-8B     | 50.6      | 54.0   | 40.0  | 83.0    | 30.1     | 32.7       | 6.2     | 2011    | 46.1  |\n| Mulberry-8B           | 56.3      | 54.5   | 43.0  | 79.5    | 34.1     | 47.5       | 18.9    | 2021    | 50.7 (**+11**) |\n\n| Models                | MathVista | MMStar | MMMU  | ChartQA | DynaMath | HallBench | MM-Math | MME_sum | AVG   |\n|-----------------------|-----------|--------|-------|---------|----------|------------|---------|---------|-------|\n| Llama-3.2-11B-V-Ins.  | 48.6      | 49.8   | 41.7  | 83.4    | 34.3     | 40.3       | 4.1     | 1787    | 45.8  |\n| LLaVA-CoT-11B         | 54.8      | 57.6   | 44.9  | 82.1    | 36.8     | 47.8       | 10.3    | 2077    | 51.1  |\n| Mulberry-11B          | 61.1      | 58.5   | 45.6  | 83.5    | 37.2     | 48.9       | 18.7    | 2035    | 53.3 (**+7.5**) |\n\n-----\n\n**Q2: CoMCTS results with four strong foundation models.**\n\nThank you for your question!\nIn the table below, we present results of CoMCTS using four strong foundation MLLMs (GPT-4o, Claude-3.5 Sonnet, Qwen2-VL-72B and InternVL2-40B).\nAs model strength increases, CoMCTS is able to generate more accurate reasoning steps and error localization, resulting in higher search success rates.\nThese findings also highlight the long-term potential and scalability of CoMCTS as foundation models become increasingly powerful.\nWe will include these discussions in the revised paper.\n\n| Models | CoMCTS (GPT-4o + Qwen2-VL-72B + LLaMA3.2-V-11B + Qwen2-VL-7B) | CoMCTS (GPT-4o + Qwen2-VL-72B + Claude-3.5 Sonnet + InternVL2-40B) |\n|--------|:--------------------------------------------------------------:|:--------------------------------------------------------------------:|\n| S.S.R  | 80.2                                                         | 88.9                                                               |\n\n----\n\n**Q3: Open source CoMCTS code, data, and models.**\n\nThank you for raising this point. \nWe would like to clarify that we will release all resources, including the CoMCTS search code, training data, training scripts, model checkpoints, and evaluation code, to facilitate reproducibility and further research.\n\n----\n\n**Q4: Addressing writing problems at L68 and L123–125.**\n\nThank the reviewer for pointing out these writing issues, which help improve the clarity and quality of our paper.\nWe have removed the extra space at L68 and revised the citation in L123–125 to use \\cite{} as recommended in the revision.\n\n---\n\nLastly, thank you so much for helping us improve the paper and we appreciate your open discussions! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!"}, {"review_id": "OzrpljxK8W", "reviewer": "Reviewer_GUCu", "summary": "This study introduces Collective Monte Carlo Tree Search (CoMCTS) to equip Multimodal Large Language Models (MLLMs) with step-by-step reasoning and reflective capabilities. Unlike traditional single-node expansion, CoMCTS performs collective expansion, where each MLLM in the group simultaneously generates potential reasoning paths from the current leaf node. This collaborative expansion mechanism aims to (1) generate diverse and complementary candidate reasoning nodes, (2) prevent the search process from getting stuck in low-quality reasoning spaces of individual models, and (3) significantly improve search efficiency by exploring multiple paths in parallel. Furthermore, this study generated a high-quality tree-structured multimodal reasoning dataset (Mulberry-260k), demonstrating that training on this dataset enhances the reasoning abilities of the Mulberry model and improves performance on complex multimodal benchmarks.", "questions": "See the weaknesses.", "limitations": "I could not find any dedicated section of paragraph explaining the potential limitations of this work. I'd recommend adding the explanation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n+ CoMCTS enables multiple models to collaboratively expand reasoning paths and collectively identify and prune erroneous steps, thereby significantly enhancing the effectiveness and efficiency of MLLM search.\n+ Training MLLMs on data generated through CoMCTS, data that includes explicit step-by-step reasoning and reflection paths, leads to improved reasoning abilities and allows the models to self-correct during complex tasks.\n+ High-quality tree-structured multimodal reasoning datasets like Mulberry-260k are valuable for advancing MLLM intelligence beyond direct prediction, toward more robust, explainable, and human-like reasoning processes.\n\nWeakness\n\n+ Although CoMCTS itself is not specific to MLLMs, the evaluation is conducted solely on MLLMs. From the outset of Section 1, the research assumes an MLLM-based setup, but it would be necessary either to include evaluations on tasks like mathematics or to provide a justification for not addressing such tasks.\n+ Table 2 demonstrates the effect of using collective intelligence from different MLLMs. However, the most significant performance gain is observed when Qwen2-VL-72B is added. Therefore, it would be interesting to see the experimental results of applying CoMCTS using only Qwen2-VL-72B. In other words, it is important to distinguish whether the improvement stems from collective intelligence or from the capabilities of the powerful MLLM, Qwen2-VL-72B.", "was_revised": true, "final_justification": "The author response gave empirical evidence that partially addressed my concerns about the limited evaluation of MLLMs and the usefulness of collective intelligence from different MLLMs. I think this paper provides a worthy contribution to the conference.", "rebuttal": "Dear Reviewer GUCu,\n\nMany thanks for your professional review!\nWe are pleased that you have recognized our key contributions: 1) the effectiveness and efficiency of CoMCTS; 2) reasoning and self-correct capabilities of Mulberry; 3) high-quality tree-structured multimodal reasoning datasets Mulberry-260k.\nBelow, we tried our best to respond to your concerns and suggestions one by one.\n\n---\n\n**W1: Additional evaluation of CoMCTS on pure-text LLM \\& More results on mathematics benchmark.**\n\nThank you for your valuable feedback! We are going to respond to this concern in two parts:\n\n> Although CoMCTS itself is not specific to MLLMs, the evaluation is conducted solely on MLLMs.\n\n**Additional evaluation of CoMCTS on pure-text LLM.**\nAs stated in Lines 29-30 and 36-37 and throughout the paper, our work is motivated by the limited reasoning capabilities of existing MLLMs, which prompted us to design CoMCTS for collaboratively searching high-quality reasoning data for MLLMs.\nAccordingly, this paper focuses on multi-modal reasoning data search and MLLM training, and our evaluations have thus been conducted on MLLMs.\nHowever, to address your insightful question, we conducted an experiment using 100 cases sampled from DAPO-Math, showing that collective intelligence also improves search success rates in pure text reasoning tasks using LLMs, as presented in the table below.\n\n|       | MCTS (4o) | MCTS (Qwen2.5-72B) | CoMCTS (4o+Qwen2.5-72B) |\n| ----- | --------- | ------------------ | ----------------------- |\n| S.S.R | 25\\%      | 18\\%               | 39\\%                    |\n\n\n> From the outset of Section 1, the research assumes an MLLM-based setup, but it would be necessary either to include evaluations on tasks like mathematics or to provide a justification for not addressing such tasks.\n\n**More results on mathematics benchmark.**\nWe agree that evaluating mathematics tasks is both necessary and important for assessing the reasoning abilities of MLLMs.\nTo this end, we would like to kindly clarify that we have reported the performance of Mulberry models on mathematics benchmarks such as MathVista, DynaMath and MM-Math in Table 1.\nIn response to your suggestion, we have further expanded our evaluation to include additional mathematics benchmarks, including MathVision, MathVerse and WeMath, as shown in the table below.\nThe results show that Mulberry models consistently achieve substantial improvements over base models and reasoning MLLMs across all evaluated mathematics benchmarks.\nWe believe these results further demonstrate the effectiveness of our approach.\nThank you!\nWe will include these results in the revised version.\n\n| Models                        | MathVista | DynaMath | MM-Math | MathVision | MathVerse | WeMath | AVG  |\n|-------------------------------|-----------|----------|---------|-------------|-----------|--------|------|\n| LLaMA-3.2-V-11B (Base Model)  | 48.6      | 34.3     | 4.1     | 19.3        | 26.9      | 34.6   | 27.9 |\n| LLaVA-CoT-11B                 | 54.8      | 36.8     | 10.3    | 20.2        | 33.4      | 39.3   | 32.5 |\n| Mulberry-11B                  | 61.1      | 37.2     | 18.7    | 26.0        | 36.1      | 44.3   | 37.2 (**+9.3%**) |\n\n----\n\n**W2: Effect of collective intelligence vs. single powerful MLLM (Qwen2-VL-72B).**\n\nThank you for raising this concern.\nWe conducted experiments comparing the search success rate on 100 samples using MCTS with a single GPT-4o, a single Qwen2-VL-72B, and CoMCTS with GPT-4o + Qwen2-VL-72B.\nThe results below show that the search performance of individual models is limited (i.e., 63.8 for GPT-4o, 56.1 for Qwen2-VL-72B), and the main performance gain comes from collective intelligence (i.e., 74.6 for GPT-4o + Qwen2-VL-72B) rather than from a single powerful model.\nAdditionally, we have provided the direct prediction results using only Qwen2-VL-72B in Table 9 of Appendix D.1. These results also show that the reasoning capability of Qwen2-VL-72B alone is limited (48.3), further demonstrating that the benefits of CoMCTS primarily stem from collective intelligence rather than from a single strong MLLM.\n\n| Models | MCTS (GPT-4o) | MCTS (Qwen2-VL-72B) | CoMCTS (GPT-4o + Qwen2-VL-72B) |\n|--------|---------------|---------------------|--------------------------------|\n| S.S.R  | 63.8          | 56.1                | 74.6                           |\n\n\n---\n\nLastly, thank you very much for your constructive feedback and suggestions.\nWe will incorporate these points into the revised paper.\nIf you have any further questions, please feel free to let us know. We will be available throughout the rebuttal period."}, {"review_id": "ERVztz1Kuz", "reviewer": "Reviewer_YpLW", "summary": "This paper introduces Mulberry, a multimodal large language model (MLLM) enhanced with step-by-step reasoning and reflection capabilities, achieved through the Collective Monte Carlo Tree Search (CoMCTS) method. CoMCTS leverages collective knowledge from multiple models to collaboratively search, evaluate, and refine reasoning paths, addressing challenges like search effectiveness and efficiency in traditional MCTS. This paper also constructs a dataset with rich reasoning trees and uses it to train Mulberry, which outperforms existing models on benchmarks requiring complex reasoning.", "questions": "1. What are the innovative aspects of the CoMCTS tree search method designed for MLLMs, and how does it differ from text-based methods?\n\n2. How can the overhead introduced by CoMCTS during the search phase be measured? If feasible, could you estimate the total token cost, GPU hours, or API price for data generation?\n\n3. Could you provide statistics on the contribution of each model in the *Collective* process, such as the similarity in sampling between different models or the proportion of each model in generating correct paths?\n\n4. Is it possible to achieve good performance without introducing smaller models, for example, by only using GPT-4o and Qwen2-VL-72B? For Qwen2-VL-72B and Qwen2-VL-7B, their behaviors might be similar due to shared training data—does integrating both models still provide an advantage?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper introduces Collective Monte Carlo Tree Search (CoMCTS), a novel approach that leverages collective knowledge from multiple models to enhance reasoning-path searching and learning in MLLMs.\n\n2. The construction of Mulberry-260k, a multimodal dataset with rich, explicit reasoning nodes, provides a valuable resource for advancing research in step-by-step reasoning and reflection.\n\n3. The paper demonstrates strong performance of the Mulberry models across various benchmarks, outperforming other models.\n\n## Weaknesses\n\n1. The CoMCTS algorithm merely integrates model ensemble techniques with the MCTS algorithm, lacking significant novelty. Previous work, like LE-MCTS [1], has already investigated this combination in NLP tasks. Furthermore, CoMCTS does not exhibit a specialized design for multimodal models. For example, *expansion, simulation and error positioning, backpropagation, and selection* can also be applied to standard LLMs, and some related research on these approaches already exists.\n\n2. The performance improvement from MCTS alone appears marginal. For instance, in Table 2, GPT-4o direct achieves 58.2, while GPT-4o + CoMCTS reaches only 63.8—a modest gain given the increased computational overhead. The more significant improvement seems to stem from ensembling the Qwen2-VL-72B model rather than MCTS itself.\n\n3. The effectiveness of CoMCTS hinges on the diversity and quality of the models in the collective learning group. Additionally, the reliance on multiple models for collective learning and iterative tree search increases the computational overhead.\n\n[1] Park, Sungjin et al. “Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning.” North American Chapter of the Association for Computational Linguistics (2024).", "was_revised": true, "final_justification": "After reviewing the author's rebuttal, several questions have been addressed, prompting me to increase my overall rating.", "rebuttal": "Dear reviewer YpLW,\n\nMany thanks for your professional, detailed, and valuable reviews.\nWe are encouraged by your recognition of CoMCTS as a novel approach, the valuable resource of the Mulberry-260K dataset, and the strong performance of Mulberry models.\nWe will do our best to address your concerns one by one.\n\n---\n**W1: Discussion of the Uniqueness and Distinction of CoMCTS**\n\nThank you for raising this concern. \nWe respectfully believe that LE-MCTS and our paper are concurrent works (completed within the same week). Given its relevance and excellence, we will include a discussion of the differences between our work and LE-MCTS, and cite it appropriately in our revised paper.\nBelow, we summarize the key distinctions between CoMCTS and LE-MCTS:\n1) Different objectives. Our work uses CoMCTS to generate a novel, effective, and reflective reasoning dataset for MLLM training, whereas LE-MCTS focuses solely on inference for LLMs.\n2) New dataset constructed by CoMCTS: We use CoMCTS to build a large-scale reasoning and reflection dataset, which we hope will benefit and facilitate further research in the community.\n3) Different MCTS design. In each iteration, our model generates a complete reasoning path, followed by pruning error nodes, as detailed in Lines 146-162. We believe this approach is more suitable for LLM/MLLM compared to traditional step-by-step generation, making data generation more efficient. Unlike classic MCTS methods in domains like Go or multi-agent systems that require step-wise search due to external feedback, reasoning tasks with MLLMs allow for generating an entire reasoning chain at once, reducing redundant computation and accelerating the search process.\nAdditionally, our experiments show that CoMCTS significantly reduces the average search iterations compared to LE-MCTS.\n||Le-MCTS|CoMCTS|\n|-|-|-|\n|Average Search Iteration|32.8|12.7|\n4) Reflection design. We introduce a new method that leverages error sibling nodes to construct reflection data. To our knowledge, this paper is the first to explore the reflective capabilities of MLLMs through tree search and collective knowledge.\n\n---\n**W2: Clarification of CoMCTS results**\n\n> The performance improvement from MCTS alone appears marginal. For instance, in Table 2, GPT-4o direct achieves 58.2, while GPT-4o + CoMCTS reaches only 63.8—a modest gain given the increased computational overhead. \n\nThank you! \nWe believe there may be some misunderstanding regarding the value of GPT-4o + CoMCTS in Tab 2.\nThe improvements brought by CoMCTS primarily result from **collaborative search with multiple models**, as shown in the last row of Table 2 (58.2% -> 80.3%) and the search success rates in Table 4 .\nIn Table 2, the “GPT-4o + CoMCTS” setting uses only a single model, but the MCTS configuration is identical to that of CoMCTS, such as the reasoning and error localization prompts. So we named it GPT-4o + CoMCTS. We will revise it to provide more clear clarifications and will change “GPT4o + CoMCTS” to “GPT-4o + MCTS”.\n\n> The more significant improvement seems to stem from ensembling the Qwen2-VL-72B model rather than MCTS itself.\n\nWe conducted experiments comparing the search success rate using MCTS with a single GPT-4o, a single Qwen2-VL-72B, and CoMCTS with GPT-4o + Qwen2-VL-72B.\nThe results below show that the search performance of individual models is limited (i.e., 63.8 for GPT-4o, 56.1 for Qwen2-VL-72B), and the main performance gain comes from collective intelligence (i.e., 74.6 for GPT-4o + Qwen2-VL-72B) rather than from a single powerful model.\nBesides, we have provided the direct prediction results using only Qwen2-VL-72B in Table 9 of Appendix D.1. These results show that the reasoning capability of Qwen2-VL-72B alone is limited (48.3), further demonstrating that the benefits of CoMCTS primarily stem from collective intelligence rather than from a single strong MLLM.\n\n||MCTS (GPT-4o)|MCTS (Qwen2-VL-72B)|CoMCTS (GPT-4o + Qwen2-VL-72B)|\n|-|-|-|-|\n|S.S.R|63.8|56.1|74.6|\n---\n**W3: Effectiveness of collective learning group and computation overhead**\n\nThank you!\n> The effectiveness of CoMCTS hinges on the diversity and quality of the models in the collective learning group.\n\nWe respectfully clarify that a single MLLM for reasoning is ineffective, which motivates us to leverage collective intelligence for collaborative reasoning.\nEmploying stronger models for collaborative reasoning and error localization can further improve the results of CoMCTS (see our response to Reviewer H5ne’s Q2 for details), highlighting the long-term potential of collective intelligence and collaborative learning for future advances.\n\n> Additionally, the reliance on multiple models for collective learning and iterative tree search increases the computational overhead.\n\nRegarding computational overhead, inference with smaller models such as Qwen2-VL-7B is relatively fast (1s per inference), and GPT-4o API calls are also highly efficient. Although larger model Qwen2-VL-72B is more resource-intensive, utilizing vLLM for inference can significantly accelerate the process (2-3s per inference).\nWhile utilizing multiple models for search inevitably increases computational overhead, we believe that a more important benefit is the improvement in the quality of the generated reasoning data.\nWhen training the Mulberry model with data searched via CoMCTS, using the same base MLLMs and a comparable data scale, the results outperform both LLaVA-CoT and LLaVA-Reasoner, as shown in Tab 1, Tab 6, Tab 16, and in our responses to reviewer GUCu's W1 and reviewer H5ne’s Q1.\n\n---\n**Q1: Innovations of CoMCTS and its Relation to Other MCTS Methods**\n\nThank you!\nWe would like to kindly clarify that we discuss the relationship of our paper to other MCTS methods in Appendix E.1 and to other reasoning MLLMs in Appendix E.2. Additionally, we will include the discussions from W1 in the revised version of our paper.\n\nBelow, we summarize the key contributions of CoMCTS and highlight the differences compared to previous MCTS methods:\n1) we introduce collective learning into MCTS and propose CoMCTS, which leverages collective knowledge to collaboratively conjecture, search and identify effective and reflective reasoning paths for MLLMs, significantly improving search effectiveness and efficiency. To our knowledge, this is the first work that explores collective learning with MCTS for MLLMs.\n2) Its joint expansion mechanism allows CoMCTS to combine reasoning trajectories from multiple MLLMs through iterative search, constructing a unified reasoning tree with diverse and complementary nodes. This leverages the strengths of different models while avoiding traps in low-quality, homogeneous nodes within a single MLLM's reasoning space.\n3) In each search iteration, CoMCTS generates a complete reasoning trajectory and selects the last correct step as the new starting node, skipping intermediate steps to significantly reduce search time without sacrificing effectiveness. Furthermore, collective knowledge enhances error localization by enabling mutual validation among models.\n4) Using CoMCTS, we search for effective, reflective reasoning paths for multimodal inputs and construct Mulberry-260k, a dataset containing rich, explicit reasoning trees for each question.\n5) To our knowledge, Mulberry is the first work to explore the reflective capabilities of MLLMs through tree search and collective knowledge.\n---\n**Q2: Measuring CoMCTS search phase overhead**\n\nThank you for your question.\nBelow, we provide detailed estimates of the total token usage, GPU hours, and API costs incurred during data generation in the CoMCTS search phase.\n\n||total token cost (M)|GPU Hours (H)|API price ($)|\n|-|-|-|-|\n|Overhead|965|6528|3200|\n---\n**Q3: Statistics on the contribution of each model in CoMCTS**\n\nThank you for your valuable suggestions.\nWe agree that providing statistics on the contribution of each model to CoMCTS search is important and informative. \n> the similarity in sampling between different models\n\nTo compute similarity, we first encode the reasoning paths generated by different models into embeddings using the all-MiniLM-L6-v2 model. We then calculate the similarity between these embeddings. The results are shown in the table below.\n\n||GPT-4o × LLaMA3.2-V-11B|Qwen2-VL-72B × Qwen2-VL-7B|Qwen2-VL-72B × GPT-4o|Qwen2-VL-72B × LLaMA3.2-V-11B|\n|-|-|-|-|-|\n|similarity|0.8608|0.8914|0.8283|0.8014|\n> or the proportion of each model in generating correct paths\n\nIn addition, we also report the contribution of each model to the generated correct paths, as shown below.\n|Models|GPT-4o|Qwen2-VL-72B|Qwen2-VL-7B|LLaMA3.2-V-11B|\n|-|--|-|-|--|\n|Proportion in generating correct paths|47|32|13|8|\n\n---\n**Q4: CoMCTS results without smaller models**\n\nThank you!\n> Is it possible to achieve good performance without introducing smaller models, for example, by only using GPT-4o and Qwen2-VL-72B?\n\nWe agree that only using larger foundation models can achieve good performance.\nHowever, incorporating smaller models provides additional benefits with minimal computational overhead.\nAs shown below, adding two small models leads to a 5.6% improvement in search success rates.\n\n> For Qwen2-VL-72B and Qwen2-VL-7B, their behaviors might be similar due to shared training data—does integrating both models still provide an advantage?\n\n|Models|CoMCTS (GPT-4o + Qwen2-VL-72B)|CoMCTS (GPT-4o + Qwen2-VL-72B + Qwen2-VL-7B + LLaMA3.2-V-11B)|\n|-|-|-|\n|S.S.R|74.6|80.2|\n\nThe similarity analysis from Q3 also shows that while Qwen2-VL-7B and Qwen2-VL-72B generate relatively similar reasoning responses, they are not identical. They can still collaboratively contribute diverse ideas.\n\n----\nThank you again for your insightful and professional comment, which made our work more complete and solid! If there are any further questions, please let us know.\nIf you feel all questions have been addressed, we would be grateful if you could kindly consider re-rating our work. Thanks!"}]

Author rebuttal:

Meta-review: This paper introduces Collective Monte Carlo Tree Search (CoMCTS), an ensemble-style extension of MCTS that equips MLLMs with step-by-step reasoning and reflective capabilities. By allowing multiple models to collaboratively expand and refine reasoning paths, and by incorporating erroneous nodes as reflective signals, CoMCTS effectively improves both the success rate and efficiency of the search process. The authors further contribute Mulberry-260k, a large-scale dataset with structured reasoning trees and reflection nodes, which significantly enhances the reasoning ability of models and leads to strong empirical results on multimodal benchmarks. Overall, I would recommend this paper for spotlight, as it not only presents a strong empirical contribution but also opens up new directions for exploring reasoning methods in MLLMs. This work highlights the potential of ensemble-style search and reflective data generation, offering an alternative paradigm to the currently dominant distillation-based approaches, and it encourages the community to experiment with more diverse ways of constructing multimodal reasoning data.
Decision: Accept (spotlight)