Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Authors: Qi Yang
Chenghao Zhang
Lubin Fan
Kun Ding
Jieping Ye
Shiming Xiang
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large Vision Language Model
Multimodal Retrieval-Augmented Generation
In-context Learning
Monte Carlo Tree Search
Primary area: deep_learning->algorithms
TL;DR: We propose RCTS, a multimodal RAG framework that enhances LVLMs for VQA tasks by integrating a reasoning-context-enriched knowledge base and tree-search re-ranking, achieving state-of-the-art performance.

Abstract: Recent advancements in Large Vision Language Models (LVLMs) have significantly improved performance in Visual Question Answering (VQA) tasks through multimodal Retrieval-Augmented Generation (RAG). However, existing methods still face challenges, such as the scarcity of knowledge with reasoning examples and erratic responses from retrieved knowledge. To address these issues, in this study, we propose a multimodal RAG framework, termed RCTS, which enhances LVLMs by constructing a Reasoning Context-enriched knowledge base and a Tree Search re-ranking method. Specifically, we introduce a self-consistent evaluation mechanism to enrich the knowledge base with intrinsic reasoning patterns.  We further propose a Monte Carlo Tree Search with Heuristic Rewards (MCTS-HR) to prioritize the most relevant examples.  This ensures that LVLMs can leverage high-quality contextual reasoning for better and more consistent responses. Extensive experiments demonstrate that our framework achieves state-of-the-art performance on multiple VQA datasets, significantly outperforming In-Context Learning (ICL) and Vanilla-RAG methods. It highlights the effectiveness of our knowledge base and re-ranking method in improving LVLMs.

Re-ranking Reasoning Context with Tree Search
Makes Large Vision-Language Models Stronger
Qi Yang 123
Chenghao Zhang Lubin Fan
Kun Ding? Jieping Ye
Shiming Xiang 12
https://github.c http:/..commamaaa/aa http://tttuuu.ccoooaaaaa///aaaaa
Abstract
Vanilla-RAG
RCTS(Ours)
60.0
Recent advancements in Large Vision Language Models (LVLMs) have significantly improved per- formance in Visual Question Answering (VQA) tasks through multimodal Retrieval-Augmented Generation (RAG). However, existing methods still face challenges, such as the scarcity of knowl- edge containing reasoning examples and erratic responses from retrieved knowledge. To address these issues, in this study, we propose multi- modal RAG framework, termed RCTS, which enhances LVLMs by constructing a Reasoning Context-enriched knowledge base and a Tree Search re-ranking method. Specifically, we in- troduce a self-consistent evaluation mechanism to enrich the knowledge base with intrinsic reason- ing patterns. We further propose Monte Carlo Tree Search with Heuristic Rewards (MCTS-HR) to prioritize the most relevant examples. This ensures that LVLMs can leverage high-quality contextual reasoning for better and more consis- tent responses. Extensive experiments demon- strate that our framework achieves state-of-the-art performance across multiple VQA datasets, sig- nificantly outperforming both In-Context Learn- ing (ICI.) and Vanilla-RAre RAG methods. It highlights the effectiveness of our knowledge base and re- ranking method in improving LVLMs.
642 5588
Average Accuracy
100
40.0
(2B) Qwen2-VL (7B)
InternVL-2 (8B)
Figure 1. Comparison various methods huilt on different LVL.S across multiple reasoning datasets. Our proposed RCTS frame- work demonstrates substantial performance gains over conven- tional Zero-Shot and Vanilla-RAs (Lin et al., 2024b) methods.
Recently, large vision language models (LVLMs) (Achiam et al., 2023; Bai et al., 2023; Chen et al., 2024) exhibit remarkable efficacy across diverse visual question answer- ing (VQA) tasks, being capable of processing multiple im- ages concurrently and, furthermore demonstrating the ability for in-context learning (ICL) (Alayrac al., 2022; Wang et al., 2024b). These capabilities facilitate the application of multimodal vetrieval-augmenn et al., 2023), a training-free approach that involves aug- menting the input prompt by retrieving relevant multimodal corpus from an external knowledge base through seman- tic similarity calculation. This approach demonstrates that introducing external knowledge effectively reduces the prob- ability that LVLMs generate incorrect content.
generation (RAG) (Gao
1. Introduction
Existing LVLMs (Achiam et al., 2023; Bai et al., 2023; Chen et al., 2024) are prone to hallucination issues (Huang et al., 2023), which manifest in two primary forms: generating factual inconsistent with real-world facts (e.g., misstating historical events political news), and producing erratic re- sponses misaligned with user instructions questions (e.g., failing to answer user queries). To mitigate factual incon- sistencies, existing multimodal RAG methods (Chen et al., 2022; Caffagni et al., 2024; Yan Xie, 2024) leverage exter- nal knowledge (e.g., Wikipedia or Web Search) to transform L.L.Ms' responses from wnknown (lacking factual ground-
"One example speaks louder than thousand words."
This work was done when Qi Yang was an intern at Alibaba Cloud Computing. 'School of Artificial Intelligence, University of Chinese Academy of Sciences, China *MAIS, Institute of Automa- tion, Chinese Academy of Sciences, China Allibaaa Cloud Com- puting, China. Correspondence to: Lubin Fan <lubbi..fb fiiinnlbbbbe.. @aiibab- inc.com>, Kun Ding <kun.dinge ia.ac.en>..
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
ing) to known (factually supported). However, addressing instruction misalignment poses a distinct challenge. An intuitive approach is to enhance user prompts by prepending few-shot example pairs through in-context learning (Alayrac et al., 2022). While effective, manual curation of such ex- amples limits scalability.
benefits offered by the retrieved samples, reinforcing the efficacy of our tree search method.
To validate the effectiveness of our proposed method, we conduct extensive experiments across multiple reasoning VQA datasets, including ScienceQA (Lu et al., 2022), MMMU (Yue et al., 2024), and Math (Wang et al., 2024a). Our method also excels in non-reasoning VQA datasets such as VizWiz (Gurari et al., 2018) and VSR-MC (Liu et al., 2023). As depicted in Fig. 1, across various sizes and types of LVLMs, our proposed approach significantly outperforms the zero-shot baseline. Besides, compared to the strategy of randomly selecting examples as context, i.c., ICL, our method yields an average of 3% improvement, demonstrating that our framework elevates LVL.Ms from mere known to better understood. Additionally, compared to Vanilla-RAA RAG, our method surpasses performance by more than 3% on all models (4.2% on Qwen2-VI. (7B), 3.9% on VL-e (8B)), indicating that the knowledge base with reasoning contexts and the tree search with answer heuristic rewards effectively re-rank examples that enhance answer accuracy. Qualitative analysis further corroborates the efficacy of our method.
For issue instruction misalignment, a compelling question arises as to whether multimodal RAG can be integrated into in-context learning, transitioning LVLMs' responses from merely known to better understood (know-how reasoning) by prepending it with retrieved examples. Specifically, it can achieve more reliable responses by retrieving and rea- soning over similar examples through in-context learning. However, several possible challenges hinder the practical application of multimodal RAG for addressing this question: i) The retrieved sample question-answer pairs are formatted in rigid, formulaic manner (e.g., "The answer is A' for multiple-choice questions), which limits the LVLMs to cap- ture underlying logical patterns. This inspires us to build a more comprehensive knowledge base with reasoning con- texts. ii) Retrieved examples may not consistently result in positive outcomes, due to the inherent limitations of in- context learning and the diversity of users' queries, which deserves more discussion. Hence, the focus here centers instruction misalignment with two main aspects: Firstly, the construction of the knowledge base with reasoning contexts to optimally enhance generation and facilitate in-context learning. Secondly, the strategic re-ranking of retrieved examples to prioritize more suitable samples, thereby pro- moting efficient and accurate response generation.
Our contributions are summarized as follows:
We introduce a multimodal RAG framework, termed RCTS, to enhance LVLMs by constructing compre- hensive knowledge base with reasoning contexts and re-ranking for highly relevant contexts.
We develop automatically constructed reasoning context mechanism grounded in VQA pairs to con- struct the knowledge base with reasoning contexts, and further propose a tree search strategy with answer heuristic rewards for re-ranking retrieved samples.
In this study, we propose a multimodal RAG framework with Reasoning Context and Tree Search, named RCTS, aiming at constructing a comprehensive knowledge base with reasoning contexts and optimizing the order of con- textual examples to improve the question answering per- formance of LVLMs. For our knowledge base component, introduce an automated reasoning contexts generation method for question-answer pairs, which helps LVLMs ac- quire intrinsic reasoning patterns. For the proposed mul- timodal RAG framework, our method begins with hybrid retrieval for an initial sampling from the knowledge base. Subsequently, we employ re-ranking mechanism to or- ganize the retrieved samples, enhancing the efficacy of in- context learning. The re-ranked Top-K samples with the generated reasoning contexts are then concatenated with the user's question to facilitate optimal answer generation by LVLMs. Regarding the re-ranking process, we propose a tree search approach with heuristic rewards to re-order the retrieved samples. This ensures the identification and priori- tization of the most beneficial contextual examples for the final generation phase, thereby enhancing the overall answer quality. Besides, the reasoning contexts generated before also allows for a quantitative assessment of the potential
Experiments show that our method achieves significant performance improvements on multiple VQA datasets, demonstrating the effectiveness of the reasoning con- text and the proposed re-ranking mechanism.
2. Related Work
Large Visual Language Models. Large Visual Language Models (LVLMs) have emerged as a significant research focus, leveraging the capabilities of powerful Large Lan- guage Models (LLMs) (Touvron et al., 2023; Jiang et al., 2023: Yang et al., 2024; Abdin et al., 2024) to tackle vision-language tasks. These versatile LVLMs demonstrate exceptional performance, particularly in visual question- answering (VQA) tasks (Team et al., 2023; Achiam et al., 2023; Bai et al., 2023; Liu et al., 2024), pointing toward a promising avenue for achieving artificial general intelli- gence. Nevertheless, these models face challenges with knowledge-based VQA due to issues such as hallucina-
2

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Languana Models Stronger
User Question
Retrieval Model df
Retrieval Model xp
Knowledge Base
food web, which granism contains matter that ovantually moves to the bat star? Below a food from ecosystem in Monterey Bay.
Similarity
Eneeder
Search
Text
Ercader
@: Which rhetorical appea
is primurly used this wd? A: liggse (ressson)
Top-N Contexts
The caans tuat the Visglaa
MCTS Re-rankking
Reasoning Contexts
User Question
IB3
#1
LVLM
+
Answer
Figure 2. Overview of the proposed framework. RCTS adopts novel multimodal retrieval- -a-gmeetedd generation framework specifically for visual question answering tasks. Aiming at enhancing the capabilities the large Asioo-language models, our method consists three components. (1) We construct a knowledge base with reasoning contexts by self-consistent evaluation mechanism. (2) To support the multimodal knowledge base, we employ a hybrid embedding strategy for relevant samples retrieval. (3) Given the uncertainty of the retrieved samples, we propose improved Monte Carlo Tree Search algorithm with heuristic rewards for sample re-ranking.
where responses are generated from nonexistent con- tent-and inherent biases (Li et al., 2023). Additionally, the lack of efficient knowledge retrieval mechanisms im- pedes their ability to integrate external knowledge bases for reasoning (Caffagni et al., 2024). In this study, we in- vestigate strategies for constructing comprehensive external knowledge bases to augment the capabilities of LVLMs.
from expansive encyclopedic knowledge bases. RATP (Pou- plin et al., 2024) leverages MCTS and RAG to enhance the self-reflection and self-critique capabilities across numerous private healthcare documents. EchoSight (Yan & Xie, 2024) attempts to address these challenges through a two-stage process, combining visual-only retrieval and multimodal reranking. thereby enhancing the alignment between re- trieved textual knowledge and visual content. However, this method risks losing the association and intrinsic knowledge of visual text due to the conversion of visual information into text. In contrast, our approach considers multimodal in- formation in both the retrieval and reranking stages, therehy preserving the integrity of the knowledge base information more effectively.
Multimodal In-context Learning. Multimodal in-context learning exemplifies paradigm in which model weights re- main unchanged, and improves output quality by adjusting the model's 's input (Dong et al., 2022: Alayrac et al., 2022: Han et al., 2023). A typical in-context learning prompt comprises two elements: demonstrations and new queries. Demonstrations involve multiple VQA pairs, each compris- ing a complete question accompanied by visual information queries
and its corresponding answer. In contrast, consist of questions posed to the model. Leveraging the emergent capabilities of LVLMs, these models can refer- ence demonstrations to some extent to address new ques- tions (Zhao et al., 2023; Zhang et al., 2024b). With the benefit of not requiring fine-tuning model parameters, in- context learning has emerged as a favored paradigm for applying LVLMs. In this study, construct the reasoning context as an integral part of the context based on VQA pairs, to enrich the reasoning knowledge of the context.
3. Methodology
Humans always learn by examples. This cognitive process can be conceptualized as exploring isomorphic structures across diverse examples, thereby improving the extraction of heuristic insights (Van Gog & Rummel, 2010). Drawing inspiration from this cognitive paradigm, we hypothesize that LVLMs can similarly benefit from contexually relevant examples for in-context learning.
3.1. Problem Statement
Multimodal Retreeval-augmmented Generation. While RAG is well-established in LLMs, its application within LVLMs remains relatively underexplored. Systems such as KAT (Gui et al., 2021), REVIVE (Lin al., 2022), and RB- VEAL (Hu et al., 2023) show promise in addressing queries involving common-sense reasoning, yet they struggle with more complex, knowledge-intense inneeniv tasks like Encyclopedic VQA (E-VQA) (Mensink et al., 2023) and Infoseek (Chen et al., 2023). These limitations are largely due to their con- strained ability to fetch and integrate precise information
Existing multimodal eetrieval-augmented generation (RAG) techniques (Yan & Xie, 2024; Li et al., 2024) primarily address open-domain questions that LVLMs fail to answer without an external knowledge base. In contrast, we focus on scenarios where user queries fall within the scope of LVLMs' capabilities, albeit with potential inaccuracies. As shown in Fig. 2, LVLMs can take advantage of relevant examples retrieved from the knowledge base to obtain more precise and reliable responses.
3

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Question Qua
Reasoning Contect C,
Question Qua
Contect
The lwage toose the Groat an Chino, whit it3 onter of forticatione Sul cae cnveral centuries an Inderrtodd the codicall Contert
bocerter for the concis tan
prab
out to protect
The Great Wol
Predicted Beasoning
wodts
the PowerS
X
Loreii an soed en the
Reasoning Contert
""ragg dnaysart The mape Sepics the aiseres mound reioo to
lans
00
The image thows Grean wi Chino, which cotat of fontifcation sult ouer cnveral crenurees petcarity duiing Oin Han canamess
ferss haieeee sage
O
Answer Agpl
answec Ot
"HWrrrrea Contett The Great Vill...
The Great Will
Reasoning Context Cv
*MUnddettadd image
The image depicts the Gneat Wall Crina, tinoricl strecture mrrchhe acren the mourta samain
Ny Predicted Answers
The Geest Wall
and estanded Reasoning
QuestionAnsswer
Question-Ressoring
Self-consistent Reasoning Context Generation
(b) Assessing Predicted Answers Reasoning Context Verification
Figure 3. Illustration of Reasoning Context Generation. The generation method consists of two steps: (a) Utilizing the question question-answer answer pairs from the knowledge base to generate self-consistent reasoning context. (b) Validating the predicted answer based on the quantitative evaluation for optimal reasoning context selection.
Knowledge Base. We define the knowledge base consisting of M visual question-answe answer pairs, denoted as DKB = ( Each x; encompasses an image T,, a question Q,, its corresponding reference answer A,, and an associated reasoning context C; (See Sec. 3.2). Formally, this can be expressed as z; = (4, Qú, A,, Ct).
Specifically, as in Fig. 3 (a), given question-answer pair (Qxs, A&s) from the knowledge base, Nr candidate rea- soning contexts are generated first by LVLMs, denoted as [C:] In Fig. 3 (b), N, predicted answers are generated by combining the question Qko with each [ These predicted answers are evaluated with the ground truth an- swer A&d to obtain a set of prediction scores {Score, Mi Finally, the candidate reasoning context with the highest score is selected as the associated reasoning context.
Goal. Our framework leverages the user's query (lu Qu) to retrieve K pertinent question-answer pairs Xyet = (I1, .2., σK) from the existing knowledge base DKB- Subsequently, the framework generates predicted answer ÿ using large vision-language model G:
3.3. Knowledge Retrieval with Hybrid Embeddings
As shown in Fig. 2, considering that both the knowledge base and user queries contain multimodal information, we employ hybrid-modal retrieval approaches rather than rely- ing solely on a single modality. Following (Lin et al., 2023; 2024b), given user's 's query consisting of an image I, and a question Qu. we first use a text encoder Fz and an image encoder F, with linear function obtain their embeddings with the same dimension d. The formulation is follows:
T ~ G ([4u: Qui Xreel), Xyet S DKB.
(1)
The goal is to develop a multimodal RAG framework that effectively integrates retrieved information with in-context learning to make the predicted answer align closely with the ground-truth response. It is worth noting that our frame- work is training-free and can be adaptively extended to multiple domains by simply expanding the knowledge base.
Er. FL(Qu) E RR/x.d.
(2)
Er, = (I...) E RR.. xd
3.2. Reasoning Context with Self-Consistent Evaluation
Existing knowledge bases usually include visual question- answer pairs without detailed reasoning procedures, which makes difficult to provide valuable context for responses even if relevant examples are retrieved. To alleviate this issue, drawing from Auto-CoT (Zhang et al., 2022), we propose method capable of automatically generating rea- soning contexts for visual question-answer pairs to enhance contextual information during generation. We leverage a self-consisten consistennnyy mechanism of LVLMs to generate candi- date reasoning contexts and utilize mutual answer prediction for reasoning context verification.
where IT, and 4,, denote the total number of tokens of question Qu and image I,, respectively.
To enable hybrid-modal retrieval, all token-level em- beddings are concatenated for retrieval, i.e., E, = Er.,Ey, Ey,.] E t(,),t Similarly, to maintain con- sistency with user queries, we utilize the same text ques- tions and images, excluding answers from the knowledge base for the retrieval process. The knowledge base hybrid embeddings are defined as:
EKB = (E,),M = (ET, 1
(3)

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Retrieval from Knowledge Base
User Question
Self-Consisteng Reward
(In Qu)
%
Action
A
Question:
MCTS-HR
A2
Beasoning
Re-ranking
f13
Mutual Heuristic Reward
Contest:
Qu)
(lar Qn)
Gr
5
1000
#2
A000
Aonte)
Auc(n)
Figure Illustration our Monte Carlo Search with Heuristic Rewards (MCTS-HR), To address the user's query. we initially retrieve Top-N samples candidate actions, which are subsequently selected through our MCTS-HR for sample re-ranking. Additionally, we propose a heuristic reward strategy incorporates two key components, self-consisteney heuristic reward, and mutual heuristic reward, to optimize the reward function within the MCTS framework.
Finally, compute the relevance score between queries embeddings Eu and each knowledge base embed- dings EKB, as follows:
form K-shot prompt, generating response for evaluation. This response is then assessed with reward function R, and the reward value Q is backpropagated to update the tree's value information. Following the standard MCTS procedure, the upper confidence bound for trees (UCT) values of all nodes are then updated to guide further exploration. The algorithm iterates through these stages, re-ranking retrieved samples and refining responses until termination condition, such maximum number of rollouts (referring to the number of simulations) an early stopping strategy, is met. Below, we introduce the key elements of our algorithm.
r(Eu: Eq) = Σ max E
(4)
k=1
where Lu = lr, + Ir_ and 4 represent the number of to- kens in hybrid embeddings, respectively. Therefore, the final relevance scores r(E,, EKB) = [[r(E,. E The Top-N pertinent question-ani annswrr answer pairs Xyet-N (31, 22....; are chosen by relevance scores
Actions Construction and Selection. Unlike most MCTS- based methods (Zhang et al., 2024a; Qi et al., 2024) in LLMs that rely on human-defined prompts as actions to construct the tree, our approach employs question-answer pairs retrieved from the external knowledge base candi- date actions A. Formally, we employ hybrid embeddings to retrieve the N most relevant question-answe pairs, where > K, and constitute the full action space:
3.4. Re-ranking by Tree Search with Heuristic Rewards
This stage aims to re-rank the retrieved samples for selecting the most pertinent samples the context prompt, facilitat- ing efficient and accurate answer generation. Specifically, we adopt the Monte Carlo Tree Search (MCTS) methodol- ogy (Browne et al., 2012), technique primarily employed in designing game-playing bots, to enhance the sample se- lection and re-ranking processes. Since MCTS can effec- tively balance exploring diverse samples and exploiting high- quality ones through simulated trajectories, solving combi- natorial optimization in context selection, formulate the problem and propose
A = {[21 [1), (2), [IN, sn]}.
(5)
where x, Q4, C.), s; denotes the normalized sim- ilarity score between the retrieved pair and user's query.
task as sequential decision-makin a Monte Carlo Tree Search with Heuristic Reward (MCTS- HR) strategy, as shown in Fig. 4. detailed workflow of our proposed MCTS-FF HR is provided in Appendix A.
During the node expansion stage, let C C A be the set of actions that have already been selected (i.e., actions in the parent node). The remaining valid actions available for sampling are Avatid AA \ C. Then, MCTS takes an action ag P(a:) from the action space Avalid using similarity- based probability distribution:
Formally, we initialize a root node with a zero-shot response derived from the user's query. Then, existing nodes are ranked and selected for expansion using greedy sampling strategy based on visit times (a) and node values ((a). During node expansion, action is sampled from an action space constructed from the retrieval samples Xyet- When the maximum depth is reached, the algorithm performs a simulation by concatenating actions and the user query to
P(a:) EjayeAruu 8j
(6)
where the selected action a; serves as the re-ranked example 24, and this process continues iteratively until reaches maximum depth K, thereby completing a branch of the
5

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision- Vision-Language Models Stronger
MCTS. Finally, the sequence of K actions extracted from the current branch is concatenated with the user's query to form K-shot prompt, thereby completing a branch simulation and obtaining the response for this branch.
Table 1. Statistics of multiple VQA datasets divided into evaluation set and knowledge base.
Evaluation Set
Knowledge Base
Name
Size
Name
Size
Coff-sssiittecyy and Mutual Heuristic Rewards. An- other critical component of MCTS is the reward function R, which evaluates the value of each action and directs the tree expansion. Unlike the traditional LCCTS-basel LLM methods (Qi et al., 2024; Zhang et al., 2024a), which di- rectly uses language model as reward function R to score the node response, we propose a selffcoonsisency heuristic reward strategy to get the self-reward value Qs alongside a mutual heuristic reward strategy to get the mutual-reward value QM based on the in-context consistency.
SeienceQAges
4241
Science uniinva
eninval
16967
MMMU-Dev
150
MMMU- Vall
900
Math sostmini
304
Math V sest
2736
VizWizsal
4319
VizWiZuain
20523
USR-MCaest
1181
VSR-MCusain
4440
* No identical samples in evaluation and knowledge base.
where is weighting parameter that controls the impor- tance of the self-reward and mutual-reward values. More details in Section 4.4.
For self-consistency heuristic reward strategy, assuming that the predicted K-shoi response at branch i is denoted as Yj = (A,, Ci). The user questions Qu.) and the pre- diction C₁ are concatenated to generate multiple answers {A!")) j===- ) tent with the originally predicted answers A,. According to
Reward Backpropagation. After obtaining the reward value Q, we then propagate this reward value to its parent and ancestor nodes. Formally, if the reward value of any ele- ment in the child node set Children(p) changes, the reward value of the parent node Q(p) is updated to:
In theory, these answers A(")) should be consis-
the above heuristic rules, the self-reward value Qs,s can be
((()) ND)) + Q(c) Q'(p) = 2 N(p) +1
expressed as follows:
mAx
Q(i)
iCChildren (p)
(10)
Qs.:
1 N:
Σ
R (A (A
.Ai).
where Np)) denotes visit times of the parent node p. Q(c)
represents the reward value of the changed child node c.
maxigChildee(pp) Q(1) represents the highest quality value among all child nodes of parent node p.
where Alt) G (Lui Qui m.). G represents the large vision language model, G(,, n) denotes the random seed in answer generation. Reward function R is calculated through the rule-based evaluator.
This formula takes into account not only the reliability of the answers of all child nodes in the parent node p. but also the reward value of the answer of the most outstanding child.
For mutual heuristic reward strategy, we posit that if the answer question is correct, it will positively contribute to other questions, and vice versa. Therefore, we greedily pick Nm samples ((Tn, ) from the actions space A to serve as subsequent mutual heuristic samples. For branch i, we utilize the user's question and the predicted response yi as contextual prompts, with the selected Nm samples' questions as the reference question and its corresponding answer as the ground truth answer A,((n), The predicted
4. Experiments
4.1. Datasets
In our experimental benchmark, we carry out comprehensive experiments with three common reasoning VQA datasets in extensive domains, including ScienceQA (Lu et al., 2022), MMMU (Yue et al., 2024) and Math (Wang et al., 2024a). Additionally, we compare methods on simpler, non- reasoning VQA datasets using VizWiz (Gurari al., 2018) and VSR-MC (Liu et al., 2023). Following the original splits of these VQA datasets, we construct the knowledge base with the training set and build the evaluation set with the testing set, respectively. Tab. I presents the size statistics of the knowledge base and the evaluation set. Please refer to Appendix B for details and examples of the datasets.
answer (() for the reference question should be consistent
with the ground truth answer. Thus, the mutual-reward value
QM, can be represented as:
1 No
QM N
Σ
R
1 Ag((n))
(m)
(8)
where A(") G ([tuiQui Wi; Ini Qnl). And the final re- ward value Q; for each branch i consists of self-reward value Qs,i and mutual-reward value QM with weighted summation as:
4.2. Implementation Details
The proposed framework is applicable to mainstream LVLMs, thus we evaluate our method on various LVLMS across different scales and types, such as
Q a .s. + (1- - at) Q..s.
6

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Table Comparison results using various L. across different sizes and types the ScienceQA, MMMU, and Math datasets.
Large Vision Language Models
Datasets
Knowledge Base
Methods
Qwen2-VL (2B)
Qwen2-VLL WLe (7B)
Intern VL-2 (8B)
Zero-Shot
67.18
80.33
93.00
ICL (random retrieval)
70.
81.63
93.14 14
ScienceQA
Science traioval
Vanilla-RAG (top retrieval)
71.94
86.68
92.78
RCTS (ours)
78.99
91.44
94.20
Zero-Shot
44.00
51.33
48.00
ICL (random retrieval)
41.33
47.33
47.33
MMMU-Dev
MMMU-Val
Vanilla-RAG (top retrieval)
42.67
50.00
46.67
RCTS (ours)
44.00
53.33
51.33
Zero-Shot
18.75
22.04
21.71
ICL (random retrieval)
17.76
23.35
21.05
Math V sestminii
MathV V
Vanilla-RAG (top retrieval)
20.39
24.67
18.42
RCTS (ours)
22.04
28.95
24.01
(a) Roward infuence diteront dalasets.
(b) Rollout influence on dillerm datasots
Table Comparison results on VizWiz VSR-MC datasets.
1B da Mutual Roward D Hyors-Reward Reward
60.7
90.2
Datasets
Knowledge Base
Methods
Qwen2-VL (7B)
80
Zero-Shot
66.49
Assuany (%)
80
(%)
80
6LSS
32m
ICL
69.01
VizWizeol
VizWiz
8
Vanilla-RAG
69.89
25.7
20.0
25.6
2013
RCTS (ours)
71.50
Sc
Zero-Shot
51.22
Mathh
ICI.
52.32
VSR-MC
VSR-MC
Vanilla-RAG
52.92
SceecCGA
Dalasets
MMMU
Marht/
Number
RCTS (ours)
55.97
Table 4, Ablation study of key components in our method, where
Figure 5. (a) Ablation of reward strategy on different datasets. (b)
Rea. Con. represents Reasoning Context.
Ablation of rollouts on different datasets.
Module
ScienceQA
MMMU
Math-V
MCTS
Rea. Con.
more discussion about these parameters.
X
86.68
50,00
24.67
X
/
88.33
50.60
26.97
X
88.92
49.33
25.65
4.3. Main Results
/
/
91.44
53.33
28.95
Tab. 2 demonstrates the comparison results with represen- tative methods using various LVLMS on reasoning VQA datasets, including ScienceQA, MMMU, and Math As in Tab. 2, Vanilla-RAG (top retrieval) (Lin et al., 2024b) has achieved performance improvement compared to both Zero-Shot and In-Context Learning (ICL) (Han et al., 2023) with random retrieval examples on most datasets, suggesting that semantic- aware example selection is cru- cial for L.LMMs' reasoning. In particular, our proposed RCTS demonstrates substantial gains across all benchmarks. Notably, for Qwen2 VL (2B), RCTS achieves 78. 78.99% on ScienceQA, surpassing both Zero-Shot by +11.819 and Vanilla-RAG by +7.05%. The improvements even more pronounced in the mathematical reasoning dataset, RCTS elevates Qwen2-VI. (7B) from 24.67% (Vanilla-RAG) to 28 95%, establishing new state-of the-art results.
Qwen2-VL (2B/7B) (Wang et al., 2024c), and Intern 2 (8B) (Chen al., 2024). Both models support multi-image input, enabling prompt concatenation with multi image con- text. For efficiency, I.V..Ms with over 7B parameters plemented in 4-bit quantization by AWQ (Lin et al., 2024a) on a single 4090 24GB GPU. Besides, we utilize the frozen BERT-base model and the ViT-L followed by 2-layer MLP both adapted from PreFLMR (Lin et al., 2024b) as our text and vision encoders, respectively. For the setting multi- ple rounds of LVLMs generation, we N. N2- 10, N, Nm 5. For the setting of our MCTS-HR, we adopt the same number of few-shot samples with K = 3, ú.c., maximum tree depth of 3. The number of initial retrieval examples is set to N = 20 as the action space of MCTS-HR. The maximum width of the tree is set to 3 for more action exploration. We set the default rollouts with P = 10, and the reward weight with default a 0.2. Section 4.4 details
im-
Additionally, we evaluate non non-reasone reasonennnng VQA datasets with VizWiz and VSR-MC. Given that responses these datasets

---

Re-ranking Reasoning Context with Tree Scarch Makes Large Vision-Languant Models Stronger
Quecler:
Quantiions
Questorc
Vanilla-RAG
Predicted Answer:
X
The answer B.
RECUUEE: Lets deecte the leogh sex zigle eeleeeeeds sin tatngiis eXh. the property of
User Question
In restangle HKMM the anglo bieettr in intereettctte the diegoral Katin k. The distnnee to tse aand the distance of to kL How longtt JMP Choices:
(4) Bi+ 242.
(B)11
(4) 10
[0) + 2x2
Quantoo:
42
#1
45
RCTS (Ours)
entrri of cuttioe
the vine out Ei3 dre
Predicted Answer:
Lakiem
Donn K17
RecAied left araipa the plaee information and gromonia
The answer A.
Asewer:
Asswer
Answe:
The resseeeo
infermorin
"*ieorily the
Figure 6. Comparison between our RCTS and the Vanilla-RGG (Lin al., 2024b). Wherein the top examples are retrieved from the knowledge base, the below examples are re-ranked by our MCTS-HR.
Table Ablation study of the importance weight a of rewards.
Table 6. The accuracy (%) of the reasoning context on different knowledge bases.
Q
ScienceQA
MMMU
Math-V
VLM
ScienceQA
MMMU
Math
0.0
90,99
50.67
26.64
0.2
91,44
53.33
28.95
Qwen2-Vi VL (2B)
90.56
98.60
85.78
0.5
90.71
54.00
27.67
Qwen2- VL (7B)
100.0
99.89
96.67
0.8
90.17
54.00
25.32
Intern VL-2 VL- (8B)
97.37
96.00
92.84
1.0
86.72
48.67
25.98
typically consist of single word a brief sentence, we only introduce the knowledge base without reasoning context. As presented in Tab. 3, our approach demonstrates consistent effectiveness with + +1.61% % and +3.05% enhancements on VizWiz and VSR-MC respectively compared to Vanilla- RAG, confirming its versatility and robustness.
hybrid-reward on three datasets. Obviously, using hybrid- rewards performs best on all three datasets, validating our design intent. Besides, as shown in Tab. 5, we perform a sensitivity analysis on the importance weight a of hybrid rewards and the default value for a was set to 0.2.
Different Rollouts. The number of rollouts an important factor in RCTS performance. Fig. 5 (b) shows the perfor- mance on the three datasets with different rollouts. It can be seen that as the number of rollouts increases, the perfor- mance on the three datasets shows a consistent trend. We finally set the rollouts with P = 10 to balance the computa- tional overhead and performance.
4.4. Ablation Study
Key Components. To validate the effectiveness of key com- ponents in our RCTS, we separately eliminate the reasoning context and MCTS-HR evaluating on various VQA datasets. As shown in Tab. 4, using MCTS-HR or Reasoning Con- text alone always has positive effect, such MCTS ScienceQA (+2.24%) and Reasoning Context on Math (+2.3%) ) in Qwen2-VI. (7B). The same model applies to the following. Our full method with both MCTS and reasoning context achieves better performance across all datasets, sug- gesting that the designed two mechanisms complement and enhance each other. Besides, the wide variety of questions covered by the MMMU results in limited performance im- provement (+3.33%), attributable to an insufficient number of analogous samples within the knowledge base.
4.5. Discussion
Reliability of Reasoning Context. Tab. demonstrates the reliability and accuracy of the reasoning context generated by our self-consistency evaluation strategy. Specifically, we evaluate the accuracy of the ground-truth answer with the predicted answer, which is generated by splicing the ques- tion and the reasoning context into prompt that yields the corresponding answer. As illustrated in Tab. 6, the gener- ated reasoning context provides precise and comprehensive responses for simpler datasets like ScienceQA. For more complex questions, our strategy still yields substantial
Rewards in MCTS. Fig. 5 (a) shows performance com- parisons using self-reward alone, muwual alone, and
8

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
proportion of correct reasoning context. These results un- derscore the effectiveness of our proposed method.
multimodal AI, such as healthcare, education, and urban planning. As Al continues to permeate daily life, RCTS represents critical step toward creating more transparent, interpretable, and cognitively aligned Al systems, ultimately enhancing their societal impact and adoption.
Qualitative Analysis. To further illustrate our RCTS su- perionity over the baseline method in terms of in-context learning, we present a qualitative analysis comparing our method and Vanilla-RAG in Fig. 6. Although the Vanilla- RAG method (Lin et al., 2024b) combined with reasoning context samples can answer the reasoning information, the retrieved samples are ill- ill-fitting, resulting in incorrect response. In contrast, our RCTS offers more reliable reason- ing contexts by re-ranking the retrieved samples and scoring all the re-ranked context sequences through our heuristic reward mechanism, providing a more reliable answer. Ap- pendix D provides more complete re-ranking processes.
A Acknowledgements
This research was supported by the Strategic Priority Re- search Program of Chinese Academy of Sciences (Grant No. XDA0370305), the National Natural Science Foundations of China (Grant No. 62306310).
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 2tti::2404.4211999 2024.
5. Conclusion
In this work, we introduce multimodal RAG framework, termed RCTS, that focuses on constructing a comprehen- sive knowledge base with reasoning contexts and re-ranking high-quality context. The goal is to enhance the VQA abil- ity of LVLMs by incorporating more relevant reasoning contexts, so that these models go from roughly knowing to better understanding the intrinsic knowledge of the con- text. Specifically, we introduce a self-consistent evaluation mechanism for generating reasoning contexts to enrich the knowledge base. In addition, MCTS-HR is proposed to re-rank the retrieved samples. Experiments on various VQA datasets show that our method is superior to contentex learn- ing and vanilla multimodal RAG methods.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidr, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774 2023.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 35:23716 237366 2022.
Limitations. Although RCTS brings significant perfor- mance improvements, it still depends on whether the pres- ence of helpful samples is within the knowledge base. Be- sides, our method inevitably takes more computational over- head; the trade-off between performance improvement and model overhead is still worth discussing.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint MXXv:2308.11266 2023.
Browne, C. B., Powley, E.,, Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. A survey of monte carlo tree search methods. IEFF Transactions on Computa- tional Intelligence and Al in games, 4(1):1443, 2012.
Impact Statement
The proposed multimodal RAG framework, RCTS, signif- icantly advances the capabilities of LVLMs in VQA by integrating a comprehensive knowledge base, reasoning contexts, and heuristic-based tree search. This innovation is pivotal for applications requiring complex multimodal VQA, such autonomous systems, educational technologies, and Al-driven decision support systems. The framework's 's abil- ity to automatically generate reasoning contexts and opti- mize sample selection through advanced Monte Carlo Tree Search (MCTS) ensures that Al systems can better handle intricate real-world scenarios, fostering trust and usability in AI technologies. Furthermore, the demonstrated improve- ments across diverse datasets, highlight the framework's versatility and potential to revolutionize fields reliant on
Caffagni, D., Cocchi, F., Moratelli, N., Sarto, S., Cornia, M., Baraldi, L., and Cucchiara, R. Wiki-llava: Hierarchical retrieval -ugmentte generation for multimodal 1mms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1818-1826, 2024.
Chen, W., Hu, H., Chen, X., Verga, P., and Cohen, W. W. Murag: Multimodal aetrivall-uugeent generator for open question answering over images and text. arXiv preprint arXiv:2210.02922 2022.
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Rit- ter, A., and Chang, M.-W. Can pre-trained vision and

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
language models answer visual information--seking ques- tions? arXiv preprint ar arXiv:2302. 17713, 2023.
Li, Y., Wang, L., Hu, B., Chen, X., Zhong, W., Lyu, C., Wang, W., and Zhang, M. A comprehensive evaluation of gpt-4v on knowledge-intensivve visual question answering. arXiv preprint arXiv:2311.075366 2023.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, I.., et al. Internvi: Scaling vision foundation models and aligning for generic disual-linguistt tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24195 2024.
Y., Li, Y., Wang. X., Jiang, Y., Zhang, Z., Zheng, X., Wang, H., Zheng, H.-T., Yu, P. S., Huang, F., et al. Bench- marking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arXiv:2411 1.02937. 2024.
Dong, Q.. Li, L., Dai. D.. Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. A survey in-context learning. arXiv preprint DXXiv23301.0023: 2022.
Lin, J., Tang, J., Tang, H.,, Yang, S., Chen, W.-M.. Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awe aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, I.,, Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modalitt models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198- 11201, 2024.
Lin, W., Chen, J., Mei, J., Coca, A., and Byrne, B. Fine- grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neu- ral Information Processing Systems, 36:22820-22840, 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. retrieval-augmenne generation for large language models: A survey, arXiv preprinl arXiv:2312.10997 2023.
Lin, W., Mei, J., Chen, J., and Byrne, B. PreFLMR: Scaling up fine-grained late-interaction multi-modal re- trievers. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5294-5316, Bangkok, Thailand, August 2024b. Association for Computational Linguis- tics. doi: 1/.186653/112224.alll 2024. acl - 1on..lll...2222 289/.
Gui, L., Wang. B., Huang, Q., Hauptmann, A., Bisk, Y., and Gao, J. Kat: A knowledge augmented transformer for vision-and-langagee arXiv preprint arXiv:2 12.0061,,, 2021.
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp.
URL https:
/ //aclanthoolse
Lin, Y., Xie, Y., Chen, D., Xu, Y., Zhu, C., and Yuan, L. Revive: Regional visual representation matters in knowledge-based visual question answering. Advances in Neural Information Processing Systems, 35:10560~ 10571, 2022.
2018.
Han, Z., Zhou, G., He, R., Wang, J., Wu, T., Yin, Y., Khan, S., Yao, L., Liu, T., and Zhang, K. How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investi- gation. arXiv preprint arXiv:2312.0P21
2023.
Liu, F., Emerson, G., and Collier, N. Visual spatial reason- ing. Transactions of the Association for Computational Linguistics, 11:635-651 2023.
Hu, Z., Iscen, A., Sun, C., Wang, Z., Chang, K.-.., Sun, Y., Schmid, C., Ross, D. A., and Fathi, A. Reveal: Betrieval-augmenne
visual-language pre-training with multi-source multimodal knowledge memory. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23369-23333 23379, 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang. H., Chen, Q.. Peng, W., Feng, X., Qin, B., al. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05222. 2023.
Mensink, T., Uijlings, J., Castrejon, L.,, Goel, A., Cadar, F.,
Zhou, H., Sha, F., Araujo, A., and Ferrari, V. Encyclo- pedic vqa: Visual questions about detailed properties of fine- -reigga categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3113- 3124, 2023.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:23 10. 10.06825, 2023.
10

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Pouplin, T., Sun, H., Holt, S., and van der Schaar, M. Re- trieval augmented thought process for private data han- dling in healthcare. arXiv preprint arXiv:2402.07812, 2024.
massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-95611 2024.
Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang. F., and Yang, M. Mutual reasoning makes smaller Ilms stronger problem- solvers. arXiv preprint arXiv:2408.066 06195, 2024.
Zhang, D., Huang, X., Zhou, D., Li, Y., and Ouyang, W. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with Ilama-3 8b. arXiv preprint arXiv:2406.0777 07394, 2024a.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, I.., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., al. Mastering the game of go with deep neural networks and tree search.
Zhang, X., Li, Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the outoof-distiiuutio gener- alization of multimodal large language models. arXiv preprint arXiv:2402.06559 2024b.
nature, 529(7587):484 489, 2016.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M.. Hauth, A., al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., and Chang, B. Mmicl: Empower- ing lanounla model with multi-modal in-context learning. arXiv preprint arXiv:2309.079.. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini arXiv:2302. 13971, 2023.
Van Gog, T. and Rummel, N. Example-based learning: Integrating cognitive and social-cognitive research per- spectives. Educational psychology review, 174,,,1 2010.
Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, Mea- suring multimodal mathematical reasoning with math- vision dataset. arXiv preprint 248. 2024a.
Wang, L.,, Yang, N., and Wei, Learning to retrieve in- context examples for large language models. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Con- ference the European Chapter of the Association for Computational Linguistics (Volume I: Long Papers), pp- 1752-1767, St. Julian's s, Malta, March 2024b. Associ- ation for Computational Linguistics. URL https: / acl anthooooyy wrgg/022.eaccl1eeeeeee wwweeallllaae 105/-
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing languaee model's perception of the world at any resolution. arXiv preprint arXiv:2409. 121111 2024c.
Yan, Y. and Xie, W. Echosight: Advancing visual- language models with wiki knowledge. arXiv preprint MrXiv:2407.127335 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
A. Monte Carlo Tree Search with Heuristic Rewards (MCTS-HR)
We detail the complete workflow of MCTS below.
Tree Initialization: A root node is initialized using native user's query without any retrieved samples, generating zero-shot response for the early stopping strategy.
Node Expansion: The algorithm employs value ((a) and the visits times ((a) rank all nodes that have not been fully expanded. The node with the highest value is selected for further exploration using greedy sampling strategy.
Action Selection:
During node expansion, the MCTS employs action sampling function FAa to sample from the action space A, which serves the expansion node. The action space is constructed using N samples retrieved from the knowledge base.
When the maximum depth is reached, the algorithm performs simulation, often termed "rollouts". This involves concatenating all the actions along the branch with the user query to form a K-shot prompt, which is then used to generate the response for the branch.
Branch Simulation:
The K-shot response is evaluated using a reward function R to obtain reward value Q. This process incorporates self-reward feedback and answers heuristic feedback constraints, via in-context consistency to ensure reliability and fairness in scoring.
Reward Evaluation:
Backpropagation:
The reward value Q of the K'shot response is propagated backward to its parent node and other related nodes to update the tree's value information. If the Q value of any child node changes, the parent node's Q is also updated accordingly.
After updating the Q values of all nodes, a collection C of candidate nodes is first identified for further expansion and selection, then use the Upper Confidence Bound for Trees (UCT) update formula update the UCT values of all nodes for the next stage of exploration following (Silver al., 2016; Zhang al., 2024a). Formally, for node that have not been fully explored, the UCT, is defined as:
UCT Update:
In N ((Father(a)) +
UCT, = Q(a) + ev
(11)
N(a) +
where Q(a) the reward value of node a. (N) is the total visit times of given nodes, is a constant balancing exploitation and exploration, is small constant for avoid devided-by-zero
The algorithm iterates through these stages until termination condition T is met, including maximum rollout constraints or reaching the early stopping strategy, continuously re-ranking the retrieved samples and improve the quality of answers, and exploring new possibilities. The termination function criteria T can derive from several conditions:
Early Stopping: Termination occurs when the answers the root node and the leaf nodes, based on greedy retrieval and initial branching, are consistent.
The search terminates once the number rollouts reaches predetermined limit or all possible combinations of re-ranking samples have been traversed.
Expansion Constraints:
B. Dataset Details
B.1. ScienceQA
Science Question Answering (ScienceQA) (Lu al., 2022) is a benchmark comprising 21,208 multimodal multiple-choice questions drawn from elementary and high school science currieula. As shown in Fig. 7, this dataset is enriched with detailed annotations, including correct answers, corresponding lectures, and comprehensive explanations. The questions span diverse array of topics across three primary subjects: natural science, social science, and language science. The task involves selecting the correct answer from the provided multiple-choice options.
In our experiments, we adhere to the original dataset split, utilizing the training and validation sets, which consist of 16,967 examples, as our knowledge base. The test set, containing 4,241 examples, employed for evaluation purposes.
12

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Languee Models Stronger
Figure Illustrative examples from the ScienceQA dataset (Lu et al., 2022).
B.2. MMMU
The Massive Multi-disciplin Multimodal Understanding and Reasoning Benchmark (MMMU) (Yue et al., 2024) is a novel benchmark that comprises 11 11,550 carefully selected multimodal questions. These questions are divided into 150 for development, 900 for validation, and 10,500 for testing. This dataset is drawn from college exams, quizzes, and textbooks spanning six common disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engincering. This dataset focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks similar to those faced by experts. Besides, as illustrated in Fig. 8, the questions cover a diverse array of topics across 30 subjects and 183 subfields, including 30 highly heterogeneous image types such as charts, diagrams, maps, tables, music sheets, and chemical structures. The task mainly involves selecting the correct answer from the provided multiple-choice options.
Due to the invisibility of the true samples in the test set and the broad domain coverage of the dataset, which results in low similarity between different samples, we utilize the validation set consisting of 900 samples to construct the knowledge base and the development set with 150 examples for evaluation in our experiments.
Projacc
wew
-400 200
13,200
-104
14:00
430
1,000
10
5.50
Ced anse syne
$10311
$3.00
Fate
uvme veleenee
Kraver
Araver
Aasner
Figure& 8, Illustrative examples from the MMMU dataset (Yue et al., 2024).
B.3. Math
HATH-Vision (Math-V) (Qi et al., 2024) is a benchmark designed to evaluate the multimodal mathematical reasoning capabilities of foundation models across a wide range of mathematical tasks with visual contexts. It comprises a total of 3040 multimodal math questions, covering 16 subjects, including Algebra, Analytic Geometry, Arithmetic, Combinatorial Geometry, Combinatorics, Counting, Descriptive Geometry, Graph Theory, Logic, Metric Geometry, Solid Geometry, Statistics, Topology, and Transformation Geometry. As depicted in Fig. 9, this dataset spans five levels of difficulty. The task involves selecting the correct answer from the provided multiple-choice options and outputting the calculated answer straightforwarddly
In our experiments, to ensure the reliability of the knowledge hase, we employ a deduplicated test set as the knowledge base,
13

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
which contains 2736 samples. The test-mini set which contains 304 samples, is employed for evaluation purposes.
cal can snd boces codarr caders
Lyge dec ae chaople on conile sain a salle cudden withe Sooes unide
bect
Image
Image
Cheme
Choise weuasieaaaoooo
Figure 9. Illustrative examples from the MathV dataset (Qi et al., 2024).
B.4. WizWiz
VizWiz (Gurari et al., 2018) is a Visual Question Answering (VQA) dataset designed to assist individuals with visual impairments in better understand visual information in their daily lives. This dataset comprises visual questions from blind individuals seeking answers to everyday visual inquiries. It includes total of 20,523 training samples and 4,319 validation samples. The task in VizWiz involves determining "True" "False" based the provided questions and generating concise phrase to answer each question directly. In our experiments, we utilize the training set as the knowledge base and the validation set for evaluation purposes.
Dusston
Qusstion
Dosstion
Sipple
Choloes
Chotcea
Choicea
Choles
Thee
Trus
top of
toe away from
Meaal Spots Rassoring
Vituaal Gpartal Ressoring MMMisa-Chale//SS0MM:
Figure 10. Illustrative examples from the Visual Spatial Reasoning datasct (VSR) (Liu et al., 2023) and Visual Spatial Reasoning with Multiple-Chote Choile dataset (VSR-MC).
B.5. VSR-MC
Visual Spatial Reasoning (VSR) (Liu et al., 2023) serves as a benchmark that encompasses over 10k natural image-caption pairs, featuring 66 types of spatial relations in English, such as "under", "in front of", and "facing". The primary objective of the VSR task to assess whether the captions accurately reflect the spatial relations depicted in the images by answering True or False, while obstacle the in-context learning.
For this issue, as illustrated in Fig. 10 (b), we developed visual spatial reasoning dataset with multiple-choice format (VSR-MC). Specifically, for each sample instance, we masked the initially correct spatial relationship and then randomly selected five relations from the remaining 65 spatial relations to serve as candidate options, alongside the true spatial relation that functions as the question. The original correct spatial relation is designated as the standard answer.
In our experiments, we applied this pipeline to the training and validation sets of the VSR dataset, comprising 4,440 samples, to construct a comprehensive knowledge base. Similarly, we processed the test set to derive 1,181 test samples for
14

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
performance evaluation. This data processing strategy ensures the rigor and comparability of the experimental results.
C. Prompts in Experiment
C.1. Reasoning Context
This section provide the detailed prompt for Section 3.2.
Get Reasoning Context:
SYSTEM: helpful elearyy lead to the tinal answer.
providing
thought process
thought process
logicallyy step- by step.
USER:
e4Urr Question 44 [Question)
*"Ancwrr
[Anserr)
"*System Prompt. your thought process step by step. manner. ensuring step logicallyy
step.
Get Predicted Answer by Reasoning Context:
SYSTEM: helpfial
USER: eetier Question:*
(Question)
"THOOUGT PROCESS
(Thought Process)
C.2. Answer Prediction
This section provides the detailed prompt for our experiments. Our prompts for different datasets primarily adapted from VLMEvalKit (Duan et al., 2024), with necessary modifications to ensure compatibility across the specific tasks and datasets under investigation.
Zero-Shot Few-Shot (ScienceQA)
SYSTEM:
question image, analyse provdee your below: answer BECAUSE: (Your detailed reasaning]
You helpful
X must be one of the optious.
(Your detilled reasoning] should clearly explain the nrionale behind choice. "Important: Adfere strictly shove format deviations.
USER: (Question)
ASSISTANT: (Answer)
USER: (Question)
Tew.shoo-withhrresonnnngcoo (ScienceQA)
You are helpful aosistant responding to the question accoeding to the contest. When given questicn and image, plense saalyze the content and provide niswer in the specified
SYSTEM:
""TOUUHH "*
[Your toughi process for seriving at the answer]
*FINNLL ANSWFR.**
BECAUSE: [Your detailed ressoning).
thought process arriving answer] provide step by-step process your answer.
he one options:
(Your ditailed reasaning] should clearly explain the mriionle hahind your choice.
** "Importum:** Adhenee striclyy the above format without devistions.
USER: (Qustiion
ASSISTANT: (Answer)
15

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
USER: (Question)
Zero-Shot & Few-Shot (MMMU)
helpful assistant.
USER: (Qusstion) with option letter given (ollowing
"The quoles)
options.
ASSISTANT: (Answer)
(Question)
Answerr with the option letter from the given choices an the following format: "The answer N: (without quotes) where X must be one options.
cew-ssootviibh-raasonnngeoooo (MMMU)
SYSTEM: You are helpful assistant.
USER: [Questicn)
option given following
"The
quetes)
options. step step
answering.
ASSISTANT: (Answer)
USER: [Question)
Answer with the option letter from the given choices in the following format:
"The
X. BECAUSE: xxx' (without where X be options.
Think befor
Zero-Shot Few-Shot (MathV)
SYSTEM: You helpful assistani.
USER:: (Questica)
Anewer the preceding multiple choice question. The format response should follow this format: "The anewer is Aboxed 'X' options "YOUR.
Mhoxed YOUR ANSWER). (withorr quotes), where
"YOULANSWER ANSWER' your conclusion.
ASSISTANT: (Answer)
USER: (Question)
Anewer preeoding multiple question. your response 'X' musi be one of the uptions or 'NSWER is your cunclusion.
format:
nnewer Whoxodd (X) /Maxxdd (OSUI_ANSSER) (withou quetes),
few.sllotwwith--esonnnngccetse VMathV)
SYSTEM: You me a helpful assistant.
(Question)
Answer preceding malliiple choice question. The Cormal of your respone should follow this furmat: TThe answer is Mooodd (X) or /hwoed BECAUSE: xxx' (without quotes), where 'X' must he one of the options or "YOUR_AND YOUURANNSERR your conclusion. Think step by step beforz snswering:
(Answer)
USER: (Qusttion
quntes), preceding maitiple Fommat respouse should "The Mboxed Mhoxed (oOnoddYOURAASWWNY BECALSE: SE: your step by stop answering.
options
16

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
D. Case Example of RCTS
In this section, we primarily illustrate re-ranking process of our proposed Monte Carlo Tree Search with Hybrid Re-ranking (MCTS-HR) framework. As illustrated in the experimental analysis, we present comparative visualizations spanning mathematical reasoning (Fig. and Fig. 12), chart interpretation tasks (Fig. 14 and Fig. 13), and natural scene image comprehension (Fig. 15 and Fig. 16).
Besides, Fig. 15 and Fig. 16 exemplify two distinctive scenarios. Fig. 15 represents an ideal case where near-identical reference sample exist in the knowledge base, enabling the Vanilla-RAG to directly retrieve the matching sample and consequently ensure all candidate branches yield correct answers. Conversely, Fig. 16 demonstrates a challenging scenario where no semantically similar samples are available in the knowledge base, resulting in erroneous outputs across all candidate branches due to excessive dissimilarity between existing references and the query instance. This scenario highlights the critical dependency of retrieval performance on the knowledge base's coverage and semantic granularity.
Figure 11. Illustration of the MCTS re-ranking process math question.
17

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision- Vision-Language Models Stronger
ASA
Figure 12. Illustration of the MCTS re-ranking process math question.
18

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Figure 1.3. Illustration of the MCTS re-ranking process table question.
19

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision- Vision-Language Models Stronger
Figure 14. Illustration of the MCTS re-ranking process chart question.
20

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision- Vision-Language Models Stronger
Figure 155 Illustration of the special case of the MCTS re-ranking process on natural question.
21

---

Re-ranking Reasoning Context with Tree Search Makes Large Vision- Vision-Language Models Stronger
Figure 16. Illustration of failure of the MCTS re-ranking process art question.
22


Reviews:
[{"review_id": "XXhREOHGbE", "reviewer": "Reviewer_x7Ua", "summary": "This work proposed a multi-modal RAG method to retrieve reasoning context examples from the knowledge base. The main components of this work are (1) a CoT knowledge base, (2) knowledge retrieval metrics with hybrid vision-language embeddings, and (3) Monte Carlo Tree Search (MCTS) to retrieve the most related examples. Experiments on several benchmarks and ablation studies demonstrate the effectiveness of the components and the pipeline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive reviews and address your concerns as follows.\n\n**Q1**: Why in-context example retrieval needs such a complex tree-based search strategy? Does the order of retrieved examples matter?  \n**A1**: As demonstrated by Tan et al.[1] and Liu et al.[2], the order of retrieved in-context examples has an important effect on the generative model's response due to path dependency properties, i.e. different example orders implicitly guide the model's attention toward distinct reasoning patterns. Recognizing both the importance of example order and the combinatorial complexity in identifying optimal order, we employ a tree-based search strategy for re-ranking. This approach effectively balances exploration of potential orders with exploitation of high-performing orders through its hierarchical search structure. Besides, we further conduct experiments with shuffle/ordered examples on ScienceQA dataset to empirically validate the impact of order, with the results shown below:\n\n|Model|Retrieval-Order|Retrieval-Shuffle|RCTS-Shuffle|RCTS (ours)|\n|-|-|-|-|-|\n|Qwen2-VL-2B|71.94|71.26|73.90|78.99|\n|Qwen2-VL-7B|86.68|87.95|88.23|91.44|\n|InternVL2-8B|93.00|92.57|93.16|94.20|\n\nWhere, 'Retrieval-Order' refers to the top-3 samples sorted by similarity (i.e., Vanilla-RAG), 'Retrieval-Shuffle' denotes randomly reordered top-3 retrieval samples. 'RCTS-Shuffle' represents the shuffled orders of our RCTS re-ranking. The results demonstrate that different orders exert influence on model's performance.\n\n**Q2**: The complexity of the framework, especially the application of MCTS. And the inspiration behind the application of MCTS.  \n**A2**: We justify our adoption of MCTS from the following aspects: \n1. As mentioned in Q1, the order of the examples is important, which motivates us to optimize the example order. Similar to path planning (Eiffert et al.[3]), this problem can be modeled as a sequential searching problem, which is very suitable to be solved using MCTS.  \n2. MCTS effectively balances exploration of potential orders with exploitation of high-performing orders through its hierarchical search structure, making it more efficient than brute-force-search‌‌. Besides, our approach produce substantial performance gains, clearly justifying the computational investment in MCTS.\n\nTherefore, we adopt MCTS over conventional retrieval methods to transition from exploiting \"similar examples\" to \"relevant examples\" (L28).\n\n**Q3**: Are there any observations on the chain structure, like how the model selects the examples in such orders?  \n**A3**: Through observation of some examples of our RCTS in supplementary materials , we summarize some patterns below:\n1. In the initial expansion of RCTS, RCTS prioritizes expanding from the most similar examples. When the reward value Q is high, RCTS assigns higher priority (Fig. 15). However, if the similar example face significant degradation (i.e., have very low reward values), RCTS will exclude it (Fig. 14).  \n2. During the expansion process of RCTS, as shown in Fig. 11 and Fig. 12, valuable examples (i.e., those with higher reward values) are repeatedly explored and utilized, even if they appear in different orders. In contrast, examples with low reward values are only used once and are not revisited afterward.  \n3. After RCTS reaches the maximum number of simulation rounds, it selects the branch with the highest cumulative reward value Q as the final result, even though other branches may also contain correct answers.  \nNote: For all examples in the supplementary materials, the branch expansion follows a left-to-right sequence.\n\n**Q4**: The time computation cost.  \n**A4**: We discuss this concern at Reviewer **yd6M Q2**.\n\n**Q5**: The novelty and differences of reasoning knowledge base and hybrid embeddings compared to the references.  \n**A5**: 1. For reasoning knowledge base, while both our approach and AutoCoT[4] automate the generation of reasoning chains, our key novelty lies in proposing self-consistency strategy to verify the correctness of the reasoning context, making it suitable for RAG applications.  We elaborate on this motivation in L185–190. 2. For hybrid embeddings, we directly adopt the retrieval module (frozen) from PreFLMR[5] in L211. The core contributions of this work lie in making them suitable with reasoning knowledge base and an innovative tree-based search strategy for re-ranking examples.\n\n**Reference**\n\n[1] Tan et al., 2023, TACL, Lost in the Middle: How Language Models Use Long Contexts\n\n[2] Liu et al., 2024, Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models\n\n[3] Eiffert et al., 2020, ICRA, Path Planning in Dynamic Environments using Generative RNNs and Monte Carlo Tree Search\n\n[4] Zhang et al., 2023, ICLR, Automatic chain of thought prompting in large language models\n\n[5] Lin et al., 2024, ACL, PreFLMR: Scaling up fine-grained late-interaction multi-modal retrievers", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation makes sense.", "theoretical_claims": "This work does not include theoretical proofs.", "experimental_designs_or_analyses": "The overall experimental designs look good to me.", "relation_to_broader_scientific_literature": "I think no.", "essential_references_not_discussed": "Currently no.", "other_strengths_and_weaknesses": "Strengths:\n\n* The performance of the proposed framework looks strong. It outperforms baseline methods by large margins on several benchmarks. The ablations on the components are relatively comprehensive.\n\n* The overall writing of this paper is clear. The figures and tables are well organized and designed.\n\n* The ideas of reasoning knowledge base and hybrid embeddings for retrieval make sense.\n\nWeaknesses:\n\n* My biggest concern is on the complexity of the framework, especially the application of MCTS. In this work, MCTS is used to select most related examples. However, it is not clear to me why in-context example retrieval needs such a complex tree-based search strategy, especially for an ordered chain of examples. Does the order of retrieved examples matter? Are there any observations on the chain structure, like how the model selects the examples in such orders? The application of MCTS looks effective, but the inspiration behind the application is not strong to me. Also, the time computation cost is not provided, and it is not clear whether the overall framework has a heavy time cost.\n\n* Although the other two components, reasoning knowledge base and hybrid embeddings, make sense and work positively to the final performance, the novelty and differences of these two modules compared to the references are not clear.\n\n\n--------------\n\n\nThe rebuttal addressed my major concern on the impact of example order and the necessity of MCTS. Therefore, I am happy to increase my score.", "other_comments_or_suggestions": "Please see Weaknesses.", "questions_for_authors": "Please see Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXaox9XbQP", "reviewer": "Reviewer_eQW2", "summary": "This paper presents a method to refine the selection of retrieved examples for multimodal language model in-context learning. It has two key components. The first component is to ask LLM to generate a set of rationale/reasoning contexts given a QA pair and select the context that has the highest probability of generating the answer given the question and context. The second component is to refine the selection of K candidates from an initial retrieved candidate set. It first select candidates based on the distribution from similarity scores, and then gradually update the distribution by checking 1) consistency between the answer generated from a selected candidate with the question and the actual answer from the selected candidate and 2) whether the answer generated from a selected candidate with the question can positively contribute to the prediction other questions. They conduct experiments on different benchmarks and find that the proposed method outperforms methods without retrieval, vanilla in-context learning with random examples, and vanilla retrieval.\n\n**Update after Rebuttal.** The authors addressed most of my concerns, but my main concern about the knowledge base still partly remains because this work considers training data the knowledge base instead of traditional external knowledge. Although the authors use arguments such as dynamical change of the knowledge or generalization to different knowledge, these arguments are a bit weak considering that the knowledge candidates are large-scale training data. So I believe it is appropriate to keep my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive reviews and address your concerns as follows.\n\n**Q1**: One major concern is the assumption of this paper. Unlike other KB-VQA that mostly focuses on an external knowledge base, the knowledge base in this paper is a large number of QA pairs for each question.  \n**A1**: In RCTS, we propose a novel paradigm with reasoning VQA samples serves as the knowledge base, different from traditional RAG that relies on external sources (e.g., Wikipedia or Online Search), as detailed in L34-40.Specifically, our approach leverages reasoning QA pairs to explicitly model ICL, enhancing the model's ability to solve complex problems.  \nTo address potential concerns about the source of KB, we leverage multiple open-source VQA datasets (e.g., ScienceQA, OK-VQA, VizWiz) and further employ an automated reasoning construction framework to build a high-quality reasoning KB.  \nIn addition, regarding concerns about the assumptions in this scenario, RCTS aims to transition VLMs’ responses from merely known to better understood (know-how reasoning) by prepending it with retrieved reasoning examples, extensive experiments (Section 4.3) demonstrate the effectiveness of our approach, achieving a significant improvement on complex reasoning benchmarks compared to Vanilla-RAG baselines.\n\n**Q2**: It is not quite reasonable to me to not do some fine-tuning, given the large number of examples for each task, while merely using them as candidates for in-context learning. I would expect the authors to provide more explanations on the motivation.  \n**A2**: Thank you for raising this important concern. Actually, the core innovation of our paper lies in exploring a method that leverages massive reasoning QA pairs as a knowledge base for performance improvement without requiring training. We will provide more explanations motivated by three key considerations: \n  1. Generalization Capability    \nAlthough fine-tuning methods (SFT/PEFT) are effective, the debate between SFT and ICL hinges on a trade-off between specialization and generalization. While SFT offers more tailored and often higher performing models for specific tasks, it can lead to loss of the model’s generalization abilities, as discussed by Chen et al.[1]\n  2. Flexible Knowledge Base Construction  \nCompared to fine-tuning, our framework is training-free and can be adaptively extended to multiple domains by simply expanding the knowledge base, as mentioned in L161-164, offering greater universality. Additionally, we have added this explanation in our Introduction to further emphasize our motivation.  \n  3. Experiment Validation  \nWe follow the suggestions and explicitly compared RCTS vs. fine-tuning variants. Specifically, we take VQA pairs from the knowledge base corresponding to MathV as training samples and use Llama-Factory[2] for SFT on Qwen2-VL-2B. As shown below, the results showed that while fine-tuning improved accuracy on our test set, it caused a significant drop on out-of-domain benchmarks, validating our design choice. We believe this trade-off favors applications requiring broad adaptability.\n\n|Method|MathV|ScienceQA|\n|-|-|-|\n|Zero-Shot|18.75|67.18|\n|Fine-tuning-on-MathV-1epoch|22.69|44.56 (OOD)|\n|Fine-tuning-on-MathV-3epoch|23.03|43.67 (OOD)|\n|RCTS (ours)|22.04|78.99|\n\n**Q3**: Expect the authors to provide more explanations on the choices of the two heuristics rewards. For self-consistency rewards, it promotes the selection of QA pairs that the answer matches the generated answer. But what is the rationale behind this heuristics?  \n**A3**: Regarding the self-consistency reward, we primarily leverage the self-consistency property of VLM models (Wang et al.[3]), i.e., 'Self-consistency leverages the intuition that a complex problem typically admits multiple different ways of reasoning path leading to its unique answer.' Thus, the essence of selecting generated answers $\\{A_i^{(n)}\\}$ (L265) that match the predicted answers $\\tilde{A}_i$ (L266) lies in choosing responses where the predicted answers and the predicted reasoning paths remain consistent.  \nFor the mutual heuristic reward, we posit that if the answer to one question is correct, it will positively contribute to reasoning for other related questions, and vice versa (L277-280). Specifically, the reward is based on whether the reasoning context generated by MCTS-HR can generalize effectively to other questions (from KB), thereby selecting robust and transferable response.  \nBesides, we comprehensively take into account the two aforementioned rewards and conducted thorough ablation experiments to verify their effectiveness as shown in table 5.\n\n**Reference**\n\n[1] Chen et al., 2020, ACL, Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting\n\n[2] Zheng et al., 2024, ACL, LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models\n\n[3] Wang et al., 2023, ICLR, Self-Consistency Improves Chain of Thought Reasoning in Language Models", "claims_and_evidence": "One major concern I have for this paper is the assumption of this paper. Unlike other KB-VQA work that mostly focuses on an external knowledge base (e.g., Wikipedia), the knowledge base in this paper is a large number of question-answer pairs for each question. Based on the large number of question-answer pairs, the authors propose a training-free algorithm to use those question-answer candidates as in-context examples to enhance open-source LLMs such as Qwen2-VL and InternVL-2. But it is not quite reasonable to me to not do some fine-tuning (either full parameter fine-tuning or PEFT), given the large number of examples for each task, while merely using them as candidates for in-context learning. I would expect the authors to provide more explanations on the motivation.\n\nThis paper can be much stronger if there is another setting to compare methods that are trained on this dataset. The authors can leverage this inference algorithm to select candidates on top of those fine-tuned models while showing it can still benefit from it.", "methods_and_evaluation_criteria": "The methods of generating reasoning context make sense of this problem, and the motivation of using reranking methods to refine the candidates for answer generation is clear. The benchmarks are also legitimate.\n\nBut I would expect the authors to provide more explanations on the choices of the two heuristics rewards. For self-consistency rewards, it promotes the selection of QA pairs that the answer matches the generated answer. But what is the rationale behind this heuristics?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs are legitimate. It clearly demonstrates the effectiveness of the proposed methods compared to methods without retrieval, vanilla in-context learning with random examples, and vanilla retrieval. And the ablation study also demonstrates the effectiveness of the use of MCTS and reasoning context.", "relation_to_broader_scientific_literature": "This paper can provide a paradigm for improving the quality of multimodal examples when conducting multimodal in-context learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper is well-structured and easy to follow.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6di9pq0145", "reviewer": "Reviewer_wbuQ", "summary": "This paper focuses on RAG for VQA tasks. The authors propose constructing a reasoning-based KB with examples of successful reasoning and then introduce an MCTS-based method for finding the best set of ICL examples, motivated by the fact that existing models can only take a fairly small number of ICL examples (compared to what is retrieved). \n\nThe approach first constructs a KB of reasoning that leads to correct outputs. The main contribution is the MCTS search method, which helps choose which example to select by using the consistency of the answer (were an example to be selected) as heuristic reward, combined with a mutual consistency reward. Here, the authors formulate retrieval as a sequential decision-making task, where the action space consists of selecting examples from the KB. They start by retrieving a set of relevant examples with vector similarity and then rerank according to their heuristic tree-search algorithm. \n\nThe approach is evaluated on multiple datasets: ScienceQA, MMMU, MathV, VizWiz, VSR-MC and across two strong VLMs. The results unequivocally demonstrate improvements over examples retrieved with vanilla RAG and randomly-retrieved examples. \n\nThe authors ablate key parts of their method, including their rewards, showing that both rewards are required for strong performance.\n\n## update after rebuttal\nThe rebuttal has clarified some of the questions I had. I will maintain my positive recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your insightful and valuable reviews and address your concerns as follows.\n\n**Q1**: What dataset is Vanilla RAG retrieving from? Is the info coming from the same KB but without MCTS reranking? And I think the setup for the Vanilla-RAG is not clear. It's later improved by Fig. 6 but this information comes too late.  \n**A1**: Thank you for pointing this out. Yes, the retrieved examples come from the same knowledge base, excluding the reasoning context, as shown in Table 1. Besides, Vanilla-RAG uses the same hybrid retrieval module, which is the same as our RCTS. The difference is that it only relies on the feature similarity without MCTS reranking. And we will revise this explanation earlier about Vanilla-RAG in Section 3.1 and Section 4.2.\n\n**Q2**: L200-201: what is the score being used here? Why do you need SC if you have ground-truth answers?  \n**A2**: As shown in Fig. 3(b), the score is defined as the ratio of correctly predicted answers to the total number of predicted answers, computed as:  $\\text{Score} = \\frac{N_{\\text{correct}}}{N_{\\text{total}}}$, where $N_{\\text{correct}}$ = number of correct responses and $N_{\\text{total}}$ = total number of responses.  \nFor subsequent question, SC serves to build a VQA knowledge base with reasoning contexts. While we possess both questiones and ground-truth answers from the knowledge base, the associated reasoning contexts are unavailable. Additionally, since the model-generated reasoning paths are not entirely reliable, we use the ground-truth to verify and score these reasoning contexts. Table 6 further demonstrates the reliability of our reasoning contexts.\n\n**Q3**: The purpose of rewards (especially the mutual consistency reward) should be made clear earlier on.  \n**A3**: Thank you for pointing this out. We conduct more discussions about the purpose of rewards in Reviewer eQW2 **Q3**. And we will add this explantion earlier on Introduction (L88).\n\n**Q4**: How are you embedding images and text? Which model is used?  \n**A4**: Following Lin et al. [1], we employ the Bert-base model (110M parameters) as our text encoder for generating text embeddings. For extracting image embeddings, we utilize a ViT-L coupled with a 2-layer MLP, totaling 312M parameters. We will add this in Section 4.2.\n\n**Q5**: Which model is used to construct the KB?  \n**A5**: Thank you for pointing this out. We use Qwen2-VL (7B) to construct the KB. We will add this in L290.\n\n**Q6**: L262, does this mean that you are treating selecting a QA pair is an action? And it's not clear what the action space is or why this needs to be a sequential problem. In this case, the authors are selecting a set of ICL examples. It doesn't seem to me like order matters here, i.e. does selecting a different example first makes a difference?  \n**A6**: As illustrated in Eq. 5, our action space comprises multiple retrieved QA pairs, where each QA pair represents a distinct action. Besides, this can be treated as a sequential problem because ICL performance depends critically not only on the quality of the retrieved samples but also on the their orders. Accordingly, our RCTS treats different example orders as sequential decision-making problems within a set of actions (L236). Furthermore, through comprehensive ablation studies, we systematically investigate and demonstrate the importance of example orders in Reviewer **x7Ua Q1**.\n\n**Q7**: Computational cost is mentioned in the limitations, it would be good to have a number to put to this.  \n**A7**: We discuss the computational cost at Reviewer **yd6M Q2**.\n\n**Q8**: Essential references should be discussed.  \n**A8**: Thank you for pointing these important references, we will cite them and add discussions in the revised version. We summarize the differences as follows:\n1. LLM-R [2] introduces an iterative training framework to retrieve high-quality in-context examples for large language models. In contrast, our method employs a training-free MCTS-based reranking approach, offering greater generalization compared to trained methods.\n2. RATP [3] leverages MCTS + RAG to enhance the self-reflection and self-critique capabilities across numerous private healthcare documents. While sharing a similar MCTS + RAG concept, there are differences in design details, such as the setup of proxy rewards and our heuristic rewards, as well as variations in the action space design. Additionally, RATP’s knowledge base consists of document-style data, whereas our method focuses on example pairs with reasoning contexts.\n\n**Q9**: A few small quibbles and typos about the language in the paper.  \n**A9**: Thank you again for pointing this out, we will fix them in our revision.\n\n**Reference**\n\n[1] Lin et al., 2024, PreFLMR: Scaling up fine-grained late-interaction multi-modal retrievers\n\n[2] Wang et al., 2023, Learning to Retrieve In-Context Examples for Large Language Models\n\n[3] Pouplin et al., 2024, Retrieval Augmented Thought Process for Private Data Handling in Healthcare", "claims_and_evidence": "- the claims are clear: the authors are claiming that by selecting examples for ICL via their method they can make better use of their generated KB \n- the evidence largely points to this, however, I think the setup for the Vanilla-RAG baseline is not completely clear. It's later improved by Figure 6 but this information comes too late and it is still not completely clear to me what the source of the examples used in Vanilla RAG is.", "methods_and_evaluation_criteria": "- The evaluation domains make sense, and the method is evaluated on a range of datasets and across two recent models. \n- The proposed method performs strongly but the description of the method is not very clear. There are a few parts that were unclear to me:\n1. L200-201: what is the score being used here? Why do you need SC if you have ground-truth answers? \n2. L256-259: the purpose of these rewards (especially the mutual consistency reward) should be made clear earlier on.\n3. How are images/text embedded? Which model is used?\n4. Which model is used to construct the KB?\n\nOverall, the methods section could benefit from more sign-posting on why particular decisions are being made. Right now it reads like a sequence of somewhat arbitrary decisions; the ablations later show that these decisions help but there is little intuition given on why they should help.", "theoretical_claims": "No theoretical claims made.", "experimental_designs_or_analyses": "Experimental design is sound.", "relation_to_broader_scientific_literature": "The contribution of the paper relates more broadly to the question of data selection under a budget. The idea of using MCTS for ICL selection in this way seems novel and could be applied to other domains as well. In general, picking the right examples from a superset is an important problem.", "essential_references_not_discussed": "- https://arxiv.org/pdf/2307.07164 uses a reward model to learn to retrieve ICL examples \n- https://arxiv.org/pdf/2402.07812 seems to also use MCTS combined with retrieval for guiding model outputs, along with a proxy reward.", "other_strengths_and_weaknesses": "Strengths:\n- overall the results of the paper are strong, with consistent gains across domains \n- the idea of using self-consistency as a signal for MCTS this way is widely applicable \n\nWeaknesses:\n- It's not very clear what the action space is or why this needs to be a sequential problem. In this case, the authors are selecting a set of ICL examples. It doesn't seem to me like order matters here, i.e. does selecting a different example first makes a difference? If not, why is it being modeled as a sequential task? An ablation here on how much order matters would be helpful.\n- Computational cost is mentioned in the limitations, it would be good to have a number to put to this (as the MCTS method seems like it could involve a high number of calls)", "other_comments_or_suggestions": "I have a few small quibbles about the language in the paper:\n- The \"known\" vs \"understood\" claim is not clear and in my opinion doesn't add much to the introduction, the authors should be clearer on what they mean by understanding, which is a pretty hazy concept \n- L162 \"humans always learn by examples\": this is a big claim and not at all settled fact, it either needs more evidence or should be hedged.\n\nTypos:\n\n- L211: \"solely on single-modal.\" -> \"solely on a single modality\"\n- typo in fig 6 caption (retrievd)\n- L609: Native, retrieved\n- L630: Formally\n- L736: backwards quotes (also in the rest of appendix)", "questions_for_authors": "- What dataset is Vanilla RAG retrieving from? Is the info coming from the same KB but without MCTS reranking?\n- L262 \"employs QA pairs retrieved... as candidate actions\", does this mean that you are treating selecting a QA pair is an action?\n- (from methods:) How are you embedding images and text? Which model is used?\n- Which model is used to construct the KB?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3dEBoiVOzJ", "reviewer": "Reviewer_yd6M", "summary": "This paper introduces a novel framework to enhance Large Vision Language models (LVLMs) capability for visual question answering. Two major contributions are demonstrated in this paper. First is about creating a  comprehensive knowledge base enriched with automatically generated reasoning contexts. Second is about employing a tree search-based re-ranking mechanism called MCTS-HR, which strategically orders retrieved examples to improve the accuracy of the LVLMs' responses.\n\nThe experiments demonstrate that RCTS outperforms other methods on various visual reasoning datasets by better guiding the LVLMs to understand and utilize contextual information. Further, ablation studies on the proposed methods also validate the contributions of its key components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews and address your concerns as follows.\n\n**Q1**: From the Table 2, it is not clear why for InternVL-2, the Vanilla-Rag performs worse than the Zero-shot. Is that because of the quantization method for models over 7B?  \n**A1**: Thank you for pointing out this issue. Actually, we initially had similar concerns and conducted multiple experiments, obtaining consistent results. Here are some possible reasons we analyzed:\n1. This issue may not be caused by model quantization, since we use the same quantized models for all experiments. The performance degradation is consistent across all methods.\n2. Since InternVL-2 has been trained on ScienceQA and MathQA datasets, this might be because ICL alters the distribution of the model's outputs, resulting in failed predictions.\n3. Compared to Qwen2-VL, we observe that Qwen2-VL has better performance in multiple rounds of interaction capabilities, while our RCTS implemented ICL through multi-turn conversations. This may be the reason why vanilla-RAG is inferior to zero-shot in InternVL-2.\n4. As for why RCTS did not encounter this issue, this might be attributed to the fact that MCTS-HR incorporates heuristic rewards, which helps to validate the predicted results and can therefore mitigate the impact of ICL on the model distribution to a certain extent.\n\n**Q2**: It is important to show the database building cost, retrieving cost, and the reranking cost to help the readers better understand.  \n**A2**: Regarding the cost issues of our model, all cost experiments are evaluated on an A800 GPU, using vllm 0.6.4, with Qwen2-VL-7B-GPTQ-Int4. Additionally, we randomly select 100 samples from ScienceQA and MathV datasets for cost comparison, and repeat tests 5 times to mitigate hardware fluctuations.\n1. KB Construction Cost. The time required to construct the KB is positively correlated with the difficulty of the problems. This is because once the Score (L202) reaches the maximum value, the current reasoning context is returned as the optimal reasoning path. The cost-time is shown below:\n|Cost-Time (seconds)|ScienceQA|MathV|\n|-|-|-|\n|KB Construction (per)|0.94±0.06|4.28±0.22|\n\n2. Hybrid Retrieval Cost. As mentioned in Q4 by Reviewer wbuQ, we employ Bert-Base + ViT-L (422M parameters) to extract text and image embeddings. Besides, we pre-store the features from KB using Faiss for fast retrieval. The average retrieval time per sample ranges from 5–30 ms, which is significantly shorter than the inference time required for MCTS we discuss below.\n3. MCTS Inference Cost. MCTS inherently requires more simulations due to its rollout mechanism, thus consuming more cost. To enable faster responses for simpler questions, we introduced an early-stopping strategy based on answer consistency (L642–645), i.e., if the initial branch and the greedy retrieval branch yield the same answer, the result is returned, bypassing MCTS’s multi-round simulations. Therefore, our inference cost also varies depending on the difficulty of the problem. The cost time of MCTS-Reranking is shown as below:\n|Cost-Time (seconds)|ScienceQA|MathV|\n|-|-|-|\n|MCTS-Reranking (per)|29.55±4.5|62.32±8.6|\n\n**Q3**: One thing not clear is about the retrieving key of the method.   \n**A3**: Thank you for pointing this out. Actually, we have observed this issue in L60 under the \"challenge ii)\". Specifically, our hybrid retrieval method computes text embeddings using the user's question and the KB questions, without including the long reasoning contexts and corresponding answers (L187-191, L211-213). This design ensures efficient retrieval of the most similar image-text pairs from the KB that align with the user’s query.  \nHowever, relying solely on feature similarity is insufficient, as the information in the KB which is not similar to the query might be an important thinking step to respond to the query. For this issue, we propose a tree search approach with Heuristic Rewards, termed MCTS-HR, which dynamically re-ranks and selects the most related samples rather than similar samples. By evaluating candidate samples through heuristic rewards, MCTS-HR identifies the most pertinent samples (i.e., those beneficial for addressing the user's question) from the candidate feature-similar samples (L209-211). \n\n**Q4**: The discussion of the option of N in the experiments.  \n**A4**: Thank you for pointing this out. We take more ablation studies on N with Qwen2VL-7B, conducted on ScienceQA and MathV dataset.  \nIt can be observed that a smaller N degrades performance, since the reduction in candidate similarity samples narrows the action space of MCTS, limiting its ability for re-ranking. On the other hand, selecting an excessive number of N introduces more noise into the MCTS action space, i.e., samples that are neither similar nor particularly helpful. Taking these into account, we set N to 20.\n|N|ScienceQA|MathV|\n|-|-|-|\n|5|89.4|26.0|\n|10|90.6|27.0|\n|15|90.7|27.6|\n|20|91.4|29.0|\n|30|91.2|26.3|\n|50|91.3|27.6|", "claims_and_evidence": "From the motivation, method and experimental results of this paper, there are three major claims being made:\n1. The proposed RCTS achieves state-of-the-art performance acorss the board: The author shows how the proposed method performs on multiple reasoning VQA benchmarks, like ScienceQA, MMMU, MathV, etc. For example, on ScienceQA, RCTS achieved a 78.99% accuracy with Qwen2-VL (2B), surpassing Zero-Shot by +11.81% and Vanilla-RAG by +7.05%.\nHowever, from the major Table 2, it is not clear why for InternVL-2 method, the Vanilla-Rag performs worse than the Zero-shot on three benchmarks. Is that because of the quantization method for models over 7B? The author needs to add explanations on this.\n2. MCTS-HR effectively re-ranks retrieved examples: By re-ranking the retrieved samples, RCTS ensures that LVLMs leverage high-quality contextual reasoning. In Figure 6, the author shows the comparison between Vanilla RAG and RCTS.\n3. Hybrid rewards in MCTS-HR are beneficial: Figure 5 shows that using hybrid rewards in MCTS-HR leads to the best performance compared to using either self-reward or mutual-reward alone.", "methods_and_evaluation_criteria": "Methods:\n\nThe overall methods are reasonable. One concern after reading the paper, which is also missed in the experiments, is about the RAG cost. I understand the knowledge base could be built beforehand. But it is also important to show the database building cost, retrieving cost, and the reranking cost to help the readers better understand the proposed method.\n\nCriteria:\n\nThe paper follows standard evaluation criteria with other LVLMs and RAG methods, which makes sense to the problem. Also, the author includes both reasoning and non-reasoning VQA benchmarks to show the advancement of the proposed method.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "As I stated in the Claims And Evidence section, my concern is majorly about why, in Table 2 for InternVL-2 method, the Vanilla-Rag performs worse than the Zero-shot on three benchmarks? I could not find the discussion about this in the paper. And does this finding consistant with the other RAG papers using InternVL-2?", "relation_to_broader_scientific_literature": "The paper is related to LVLMs, multimodal information retrieval and RAG. These fields have been discussed in the related works.", "essential_references_not_discussed": "Not aware of important missings in literature.", "other_strengths_and_weaknesses": "1. One thing not clear is about the retrieving key of the method. The embedding of the query image and text might not have very high similarity with correlated information in the database, especially when the query requires long reasoning chains. For example, the information in the database is not word-similar to the query question but might be an important thinking step to respond to the query. It is not clear how this has been considered in the paper.\n\n2. I could not find the discussion of the option of N in the experiments. Will the method still work when N is very small, and will it bring marginal benefits when increasing it?", "other_comments_or_suggestions": "No, I do not have other comments.", "questions_for_authors": "No, I do not have other questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**
This paper proposes a multimodal RAG framework to solve the limitation of instruction misalignments. The authors present empirical study and validate their approach on several dataset such as ScienceQA and MMMU-Dev. 

**Reviewer Consensus and Review Quality**
The submission received four reviews. All reviewers provided positive feedback, highlighting the novelty and good empirical results. 

**Strengths**
- The novelty is good.
- The results of the paper are strong, with consistent gains across domains
- The ablations on the components are relatively comprehensive.
- The overall writing of this paper is clear. 
- The figures and tables are well organized and designed.

**Weaknesses**
- weak experiments to demonstrate the capability of this framework to generalize the external knowledge

**Justification**
Based on the reviewers comments and the author’s response, I believe this paper clearly contributes to the community of multimodality learning. While there exist a few minor issues, such as the complexity of the framework, the strengths of the paper outweigh its weaknesses significantly. In addition, it ranks 1 among 12 papers. Therefore, I recommend **strong accept**.
Decision: Accept (spotlight poster)