ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom

## Abstract

Abstract Large vision-language models (LVLMs) have witnessed significant progress on visual understanding tasks.
However, they often prioritize language knowledge over image information on visual reasoning tasks, incurring performance degradation.
To tackle this issue, we first identify the drawbacks of existing solutions ( i.e. , insufficient and irrelevant visual descriptions, and limited multi-modal capacities).
We then decompose visual reasoning process into two stages: visual perception ( i.e. , eyesight) and textual reasoning ( i.e. , wisdom), and introduce a novel visual reasoning framework named ProReason .
This framework features multi-run proactive perception and decoupled vision-reasoning capabilities.
Briefly, given a multi-modal question, ProReason iterates
proactive information collection and reasoning
until the answer can be concluded with necessary and sufficient visual descriptions.
Notably, the disassociation of capabilities allows seamless integration of existing large language models (LLMs) to compensate for the reasoning deficits of LVLMs.
Our extensive experiments demonstrate that ProReason outperforms both existing multi-step reasoning frameworks and passive peer methods on a wide range of benchmarks
for both open-source and closed-source models.
In addition, with the assistance of LLMs, ProReason achieves a performance improvement of up to 15%
on MMMU benchmark.
Our insights into existing solutions and the decoupled perspective for feasible integration of LLMs illuminate future research on visual reasoning techniques, especially LLM-assisted ones.

## 1 Introduction

In recent years, large language models (LLMs) (Dubey et al., ; Team et al., ; Jiang et al., ) have experienced explosive growth in their capabilities, driving significant advancements across various fields (Shao et al., ; Guo et al., ; Shao et al., ).
This progress has also sparked interest in developing large vision-language models (LVLMs) (Chen et al., ; Bai et al., ), which, like LLaVA (Li et al., ), have achieved remarkable performance in multi-modal understanding tasks.
However, state-of-the-art (SOTA) LVLMs still struggle to integrate visual understanding with textual reasoning simultaneously due to inherent modality differences and deficient training data.
For example, Ghosh et al. () demonstrate that LVLMs often rely more on their prior language knowledge, neglecting visual information in multi-modal reasoning tasks such as visual chart understanding and math reasoning, resulting in performance degradation. Figure 2 illustrates a typical case of this issue, where the reasoning process remains irrelevant to the image.

To address the challenges, a promising solution is to extract visual information from images into textual form to assist LVLMs in reasoning (Yin et al., ; Mitra et al., ).
For example, Ghosh et al. () instruct LVLMs to generate fine-grained captions to facilitate the subsequent reasoning process.
However, these existing methods exhibit two primary limitations:
(i) The visual extraction process is question-agnostic and reasoning-free, that the image description is not targeted for a given question, and reasoning is not involved to deduce extra information for better descriptions. This drawback, termed as “passive”, results in
irrelevant or insufficient information, and subsequent performance degradation.
(ii) The reasoning process entangles visual understanding with textual reasoning abilities of a single LVLM. As mentioned above, the limited capabilities of LVLMs do not always successfully handle both abilities and produce high-quality reasoning.
Hence, both of these limitations hinder the performance of LVLMs in multi-modal reasoning tasks.

To address both shortcomings, we propose a novel multi-step multi-modal reasoning framework named ProReason, which features proactive (i.e., question-oriented and reasoning-involved) visual extraction and disentangled vision-reasoning capabilities.
Specifically, we first decouple multi-modal reasoning capacity into two sub-tasks:
visual perception (i.e., eyesight) and textual reasoning (i.e., wisdom).
The former focuses on understanding images and converting visual details into textual form, while the latter integrates the gathered information to draw final conclusions.
As illustrated in Figure 1, our framework can be divided into three stages: Action, Judgment, and Summary, with each involving different agents.
In the Action stage, a Dispatcher selectively engages a Vision Expert to capture additional visual information, or a Reasoning Expert to derive more insights.
Unlike passive methods, all three agents operate based on the given question and current descriptions, effectively avoiding information redundancy or insufficiency.
Subsequently, a Referee in the Judgment stage evaluates the sufficiency of known information,
deciding whether to return to the Action stage or proceed to the Summary stage, where all information is consolidated for a Summarizer to generate the final answer.
Notably, the disentangled vision-reasoning does not require vision-irrelevant roles (i.e., Reasoning Expert, Referee, and Summarizer) to be performed by LVLMs, enabling the seamless integration of existing LLMs with proven strong reasoning abilities (Chang et al., ), thereby addressing the limitations of LVLMs.

Empirically, we evaluate ProReason across four challenging visual reasoning benchmarks with both open-source and closed-source models.
Extensive experiments demonstrate that ProReason achieves significant and consistent performance improvement over both existing reasoning frameworks and state-of-the-art passive methods across all the benchmarks, with a peak enhancement as 13.2%.
Besides, our comparative analysis highlights the necessity and relative importance of each sub-agent, as well as the advantages of proactive information acquisition.
With the verified superiority over the simultaneous inherent usage of both capabilities, the decoupled perspective not only demonstrates that textual reasoning capabilities outweigh visual understanding for multi-modal reasoning tasks, but also exemplifies the feasibility of LLM-assisted LVLM reasoning, exhibiting performance improvements of up to 15% on the MMMU benchmarks (Yue et al., ).

The main contributions of this work are threefold:

- •
We propose a novel multi-modal reasoning framework named ProReason, featuring iterative proactive perception and decoupled vision-reasoning capabilities.
- •
Empirically, we identify the drawbacks of existing solutions (i.e., insufficient and irrelevant visual descriptions, and limited multi-modal capacities), and validate the successful mitigation of these issues with ProReason by the superior performance over peer methods.
- •
ProReason showcases the remarkable feasibility of integrating LLMs for
boosted visual reasoning, illuminating the potential for LLM-assisted LVLM reasoning in future research.

## 2 Preliminary Observations

We first analyze the behaviors of several existing methods on the MMMU (Yue et al., ) dataset, a challenging multi-modal benchmark requiring comprehensive college-level knowledge and fine-grained reasoning abilities. All experiments are conducted with three recent LVLMs for robustness: Llama3-LLaVA-NeXT-8B (Li et al., ), LLaVA-OneVision-Qwen2-7B-OV (Li et al., ) and Qwen-VL-Chat (Bai et al., ). Further details are provided in Sec. 4.1.

### 2.1 Challenges in Visual Reasoning: Limitations of LVLMs

Chain-of-Thought(CoT) (Wei et al., ) has been extensively verified to enhance the performance of LLMs. Here, we explore its impact on
the reasoning performance of
LVLMs. Similarly, we require the models to “think step by step” before generating final answers. As shown in Table 1, a counter-intuitive phenomenon is observed: compared to “Direct” answering method, the introduction of CoT consistently incurs slight performance degradation across all three models.
Inspired by the findings of Ghosh et al. () that LVLMs often rely on prior language knowledge yet neglecting visual information,
we hypothesize that the degraded performance might be caused by the image-irrelevant reasoning of CoT.

To validate the hypothesis, we design a pipeline referencing Liu et al. () to measure the correlation between model responses and image contents.
Specifically, given a question, we take the reasoning process generated by GPT-4o-mini(^1^11[https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence)), whose superiority on MMMU task is verified in Sec. 4.2, as the golden solution.
Subsequently, a more powerful judge (i.e., GPT-4 (Achiam et al., )) is employed to evaluate the relevance of responses of candidate models (e.g., Llama3-LLaVA-NeXT-8B) and this golden solution with the prompt template in Figure 5,
and gives a comprehensive score from 1 to 5, with 5 indicating the highest relevance. Since the golden solution contains the essential information needed to solve the problem within the image, a higher correlation indicates a stronger relevance to the image content.
As shown in Table 2, the reasoning chains of incorrect responses are significantly less relevant to the images than those of correct answers. An exemplary case is shown in Figure 2, where Llama3-LLaVA-NeXT-8B attempts to use the right-hand rule to solve the task, but its reasoning process is almost unrelated to the image, ultimately leading to an incorrect conclusion.
This highlights the limitation of CoT for multi-modal reasoning tasks, that image-irrelevant reasoning process provokes performance degradation, since the limited capabilities of LVLMs cannot effectively handle both image and text information simultaneously.

**Table 1: Performance of three recent LVLMs on MMMU dataset with different assisting techniques.**
| Method | Model |  |  |
| --- | --- | --- | --- |
| Llama3-LLaVA-NeXT-8B | LLaVA-OneVision-Qwen2-7B-OV | Qwen-VL-Chat |  |
| Direct | 41.8 | 49.1 | 36.2 |
| CoT | 41.5 | 46.5 | 35.4 |
| VDGD | 42.3 | 49.7 | 37.1 |

**Table 2: Image Relevance Score of Chain-of-Thought reasoning traces for “True” and “False” responses of three LVLMs, respectively.**
| Model | Image Relevance Score |  |
| --- | --- | --- |
| True | False |  |
| Llama3-LLaVA-NeXT-8B | 3.70 | 3.36 |
| LLaVA-OneVision-Qwen2-7B-OV | 4.27 | 3.67 |
| Qwen-VL-Chat | 3.63 | 2.80 |

**Table 3: Effectiveness evaluation of passive captions along Detail Level, Question Relevance, and Reasoning Effective Info Inclusion. “True” and “False” denote the response correctness of Llama3-LLaVA-NeXT-8B.**
| Score | Llama3-LLaVA-NeXT-8B |  |
| --- | --- | --- |
| True | False |  |
| Detail Level | 4.43 | 3.93 |
| Question Relevance | 3.87 | 3.30 |
| Reasoning Effective Info Inclusion | 3.91 | 3.57 |

### 2.2 Drawbacks of Passive Information Extraction

Passive visual reasoning techniques suffer insufficient and irrelevant visual information, despite mitigating the oversight of images by converting them into detailed captions, like Visual Description Grounded Decoding (VDGD) (Ghosh et al., ).
To support this claim, we generate fine-grained image captions using GPT-4o-mini with the prompt shown in Figure 6.
We then incorporate these captions into the prompts for LVLMs to facilitate the reasoning process.
As listed in Table 1,
while these image descriptions improve the performance of LVLMs, the gains are marginal, consistently amounting to less than 1%. This underscores the limited utility of captions generated by passive methods.

For further demonstration, we analyze the generated captions along three dimensions: Detail Level, Question Relevance, and Reasoning Effective Info Inclusion, measuring the richness of detail, relevance to the given question, and the inclusion of information necessary for reasoning, respectively.
Similar to Sec. 2.1, we pair the image captions with the reasoning process of GPT-4o-mini,
and employ GPT-4 as a judge to rate the captions from 1 to 5 across these dimensions. The prompt template is shown in Figure 5.
As shown in Table 3, the captions for correct responses of Llama3-LLaVA-NeXT-8B receive higher scores across all three criteria, highlighting the importance of better captions for multi-modal reasoning. Additionally, all captions score significantly lower in the Question Relevance and Reasoning Effective Info Inclusion dimensions than the Detail Level dimension, indicating that while the captions are detailed, they often lack relevance to the questions.
Figure 2 shows a case where Llama3-LLaVA-NeXT-8B utilizes fine-grained captions to solve a question from the MMMU benchmark.
As illustrated, although the caption exhaustively describes the image content, it incorrectly describes the wires in the image as octagons, and misses information about the locations of these wires. This information is irrelevant to the target question, thus offering minimal assistance to LVLMs.
In summary, our analysis highlights the drawbacks of passive visual reasoning enhancement techniques in terms of information insufficiency and redundancy, due to their question-agnostic property.

Figure: (1) ProReason
Refer to caption: x1.png

## 3 Method

As illustrated in Figure 1, ProReason consists of five functionally distinct yet inter-cooperative sub-agents, along with a Memory component. The entire workflow comprises three steps: Action, Judgment, and Summary, effectively integrating the proactive visual understanding (i.e., eyesight) and textual reasoning (i.e., wisdom) capacities for enhanced multi-modal reasoning performance.

### 3.1 Action Step

Action step is the core of question-oriented visual information extraction, driven by three sub-agents: Dispatcher, Vision Expert, and Reasoning Expert. The Dispatcher orchestrates the workflow, selectively directing the Vision Expert to capture specific visual information, or instructing the Reasoning Expert to analyze known information to derive more. The responses from both the Vision Expert and the Reasoning Expert are stored in a textual Memory component.

Formally, given an image $I$ and its corresponding textual question $Q$, the Dispatcher decides to consult the Vision Expert or Reasoning Expert, based on the analysis of $Q$ and the known information in the Memory (if not empty). The Dispatcher then generates a query $q$ for the chosen expert. If the Vision Expert is selected, it takes the image $I$ and query $q$ as input, and generates an answer $A_{v}$, which is then stored in the Memory.
When the Reasoning Expert is selected, it provides a response consisting of the reasoning process and the final answer $A_{r}$ based on the query $q$ and known information in the Memory, before only $A_{r}$ is stored in Memory.
Notably,
the Memory component allows ProReason to keep compact information, and avoids lengthy reasoning traces like CoT and ReAct, thereby suffering less from redundant information.

### 3.2 Judgment Step

Judgment step is conducted by a sub-agent called Referee, which evaluates whether the information stored in Memory is sufficient to answer the question $Q$. The input to the Referee includes the question $Q$ and available information in the Memory. If the Memory contains adequate information to answer the question $Q$, the Referee outputs the identifier “SOLVABLE”; otherwise, it outputs “UNSOLVABLE”.
If the Referee’s output is SOLVABLE, the workflow precedes to the Summary phase. Conversely, if the output is UNSOLVABLE, the Action phase is re-executed to gather more necessary information.
This allows the Referee to collaborate closely with the three sub-agents in the Action step, enabling the framework to proactively acquire the necessary information and prevent omissions, thereby overcoming the drawbacks of passive methods.

### 3.3 Summary Step

The Summary step focuses on integrating the available information in the Memory, and providing the final answer to the question $Q$.
This step is mainly powered by a sub-agent called Summarizer.
Once the Referee determines that the information in the Memory is sufficient to address the question $Q$, and outputs the identifier SOLVABLE, the Summarizer will be called to draw a conclusion, based on the Memory information. This conclusion represents the answer to the question $Q$ of the whole framework, and will be evaluated by the performance metrics.

### 3.4 Advantages of ProReason

#### Reduced information Mission or Redundancy.

With the close collaboration of the Dispatcher, Vision Expert, Reasoning Expert, and Referee agents, ProReason can proactively (i.e., question-orientedly and reasoning-involvedly) extract the necessary and sufficient visual details from images, effectively avoiding information omission or redundancy.
Meanwhile, the Memory component retains only the image captions from the Vision Expert and the reasoning results from the Reasoning Expert, providing a compact textual descriptions. This contributes to minimize the interference of irrelevant information on the subsequent Summarizer.

#### LLM-assisted Multi-modal Reasoning.

In ProReason, the complete multi-modal reasoning process is decomposed into visual perception and textual reasoning stages, each executed by separate agents. These agents are then effectively organized through a designate pipeline to complete the tasks.
This decomposition-before-integration approach not only circumvents the inherent differences between modalities, but also allows visually irrelevant sub-agents to be performed by text-only LLMs.
Consequently, the extensively verified powerful reasoning capabilities of LLMs can be seamlessly synergized to power multi-modal reasoning process for performance enhancement.

## 4 Experiments

In this section, we first evaluate the performance of our ProReason framework against recent baselines on multiple benchmarks, followed by an in-depth ablation analysis of different components.

### 4.1 General Setup

#### Datasets.

To comprehensively validate the performance of our framework, we conduct experiments across four benchmarks: Multi-modal Large Language Model Evaluation (MME) (Yin et al., ), Massive Multi-discipline Multi-modal Understanding and Reasoning (MMMU) (Yue et al., ), MathVista (Wang et al., ), and HallusionBench (Liu et al., ). All of them require visual reasoning capabilities to complete the tasks correctly, and are introduced briefly as follows:

- •
MME is an inclusive benchmark that encompasses 14 subtasks, designed to evaluate perceptual and cognitive abilities. Due to our emphasis on visual reasoning, we select the cognition-relevant tasks, including Commonsense Reasoning, Numerical Calculation, Text Translation, and Code Reasoning.(^2^22To facilitate the comparison across different benchmarks, the results for MME benchmark are calculated by the percentage of correct answers out of the total answers. For the results using the official MME calculation method, please refer to Table 9, Table 10, Table 11, and Table 12.)
- •
MMMU evaluates multi-modal models with multidisciplinary tasks that require college-level domain-specific knowledge and detailed reasoning. It comprises 11,500 questions across 30 disciplines and 183 sub-fields, emphasizing advanced perception and domain-specific reasoning.
- •
MathVista focuses on more challenging mathematical reasoning tasks that demand precise visual recognition and compositional reasoning. It includes 6,141 examples from 31 multi-modal mathematics datasets.
- •
HallusionBench
evaluates models’ ability to reason with images such as statistical charts, emphasizing nuanced visual understanding. It consists of 346 images paired with 1,129 questions, meticulously crafted by experts.

#### Base Models.

We employ GPT-4o-mini, Llama3-LLaVA-NeXT-8B and Qwen2.5-72B- Instruct (Team, ) as our base models, due to their excellent performance, accessibility, and representativeness of model sources.
As one of the most performant LVLMs, GPT-4o-mini demonstrates significant advancements in visual reasoning capabilities, and provides cheap and fast API. In contrast, Llama3-LLaVA-NeXT-8B is a fully open-sourced LVLM developed by the LLaVA team based on the Llama-3-8B LLM (Dubey et al., ) and CLIP vision encoder (Radford et al., ).
Qwen2.5-72B-Instruct is a robust LLM designed to deliver high-quality instruction following and handle complex tasks. It will be utilized for achieving LLM-assisted multi-modal reasoning.

#### Baselines.

Besides the most basic method where models are instructed to answer questions directly, we compare ProReason with two categories of peer methods.
To determine the benefits of proactive information extraction in ProReason, we first consider two SOTA passive visual reasoning methods, VDGD (Ghosh et al., ) and CCoT (Mitra et al., ). Additionally, we choose two multi-step reasoning frameworks of LLMs, Chain of Thought (CoT) (Wei et al., ) and ReAct (Yao et al., ), to demonstrate the effects
of directly migrating LLM solutions to LVLMs.

- •
Direct.
As indicated by the name, models are required to answer questions directly without dedicated prompts. This baseline is set to evaluate the initial performance of base models.
- •
VDGD.
With image caption prefixed to text instruction, VDGD selects tokens that deviate the least from the description using a formula based on Kullback-Leibler divergence.
- •
CCoT.
Given an image and the question, CCOT first generates a scene graph of the image with LVLMs, and then extracts the answer by prompting the LVLMs with the graph.
- •
CoT. CoT is an advanced prompting method that encourages LLMs to break complex tasks down into a series of easy steps, which has been applied broadly and verified to boost the reasoning performance remarkably (Chu et al., ).
- •
ReAct. ReAct is an LLM-specific agent framework, which performs tasks by alternating between reasoning and execution behaviors.
To extend it to multi-modal domain, we use two LVLMs to perform both steps, and rename them as the Vision and Reasoning Experts, respectively. This aligns with our notions for easy understanding, and is shown in Figure 1.

#### Implementation Details.

The prompt templates for all methods are shown in Figures 6, 7 and 8.
Specifically for ProReason, in order to prevent an infinite loop, if Dispatcher selects Vision Expert or Reasoning Expert to obtain information up to 5 times, and Referee still judges that the existing knowledge in Memory is not enough to solve the question, the information collection phase will be terminated immediately, and then Summarizer will directly answer the question based on the available information in Memory.
This setup, refined through multiple trials, is the most effective. Additionally, since we cannot obtain the tokens output by GPT-4o-mini, we omit the step of selecting the token with the smallest deviation from the image description when implementing VDGD for GPT-4o-mini.

**Table 4: Performance of multiple approaches with two base models across four visual reasoning benchmarks. “Hallu.” is the abbreviation of HallusionBench. Based on the performance of the direct method, red and blue signify the improvement and degradation, respectively.**
| Model | Method | Dataset |  |  |  |
| --- | --- | --- | --- | --- | --- |
| MME | MMMU | MathVista | Hallu. |  |  |
| Llama3-LLaVA-<br>NeXT-8B | Direct | 61.5 | 41.8 | 37.1 | 45.8 |
| VDGD | 68.8 (+7.3) | 42.3 (+0.5) | 36.1 (-1.0) | 44.2 (-1.6) |  |
| CCoT | 68.9 (+7.4) | 40.5 (-1.3) | 36.8 (-0.3) | 37.4 (-8.4) |  |
| CoT | 58.8 (-2.7) | 41.5 (-0.3) | 35.9 (-1.2) | 43.1 (-2.7) |  |
| ReAct | 68.5 (+7.0) | 46.7 (+4.9) | 31.7 (-5.4) | 43.6 (-2.2) |  |
| ProReason | 71.5 (+10.0) | 52.5 (+10.7) | 38.8 (+1.7) | 50.9 (+5.1) |  |
| Average | 66.30 | 44.22 | 36.06 | 44.16 |  |
| GPT-4o-mini | Direct | 79.2 | 48.4 | 53.0 | 56.0 |
| VDGD | 82.3 (+3.1) | 51.4 (+3.0) | 51.2 (-1.8) | 52.4 (-3.6) |  |
| CCoT | 80.8 (+1.6) | 54.2 (+5.8) | 53.6 (+0.6) | 56.7 (+0.7) |  |
| CoT | 87.8 (+8.6) | 58.5 (+10.1) | 53.8 (+0.8) | 56.3 (+0.3) |  |
| ReAct | 87.3 (+8.1) | 54.8 (+6.4) | 49.3 (-3.7) | 51.1 (-4.9) |  |
| ProReason | 90.4 (+11.2) | 61.6 (+13.2) | 54.9 (+1.9) | 58.9 (+2.9) |  |
| Average | 84.63 | 54.82 | 52.63 | 55.23 |  |

**Table 5: Performance of Llama3-LLaVA-NeXT-8B Vision Expert, assisted by different text-only LLMs, on various benchmarks under ReAct and ProReason frameworks. The red and blue texts indicate the improvements and reductions relative to the case of using the same method without the assistance of text-only LLMs, respectively.**
| Vision Expert | Textual<br>Sub-Agents | Method | Dataset |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| MME | MMMU | MathVista | Hallu. |  |  |  |
| Llama3-<br>LLaVA-<br>NeXT-8B | Llama3-LLaVA-<br>NeXT-8B | Direct | 61.5 | 41.8 | 37.1 | 45.8 |
| ReAct | 68.5 | 46.7 | 31.7 | 43.6 |  |  |
| ProReason | 71.5 | 52.5 | 38.8 | 50.9 |  |  |
| Qwen2.5-72B-<br>Instruct | ReAct | 71.0 (+2.5) | 50.4 (+3.7) | 34.6 (+2.9) | 40.4 (-3.2) |  |
| ProReason | 81.3 (+9.8) | 56.8 (+4.3) | 48.8 (+10.0) | 52.3 (+1.4) |  |  |
| GPT-4o-mini | ReAct | 73.6 (+5.1) | 48.4 (+1.7) | 36.2 (+4.5) | 46.7 (+3.1) |  |
| ProReason | 84.7 (+13.2) | 54.5 (+2.0) | 41.7 (+2.9) | 53.1 (+2.2) |  |  |

### 4.2 Main Results

ProReason exhibits significant and consistent performance enhancement over baselines across all the benchmarks.
As listed in Table 4, despite better performance than the direct method on MME dataset, VDGD and CCoT fail to demonstrate consistent improvements on the other datasets.
In contrast, ProReason consistently outperforms all the other baselines for both Llama3-LLaVA-NeXT-8B and GPT-4o-mini model across all benchmarks, with a peak improvement as 13.2%, demonstrating the superiority and task robustness of ProReason.

Proactive information acquisition surpasses SOTA passive methods, especially in complex visual reasoning tasks.
Specifically, compared to MME, MathVista and HallusionBench present higher image complexity and question difficulty, and thus require stronger visual understanding and textual reasoning capabilities.
This leads to performance degradation of passive methods (i.e., VDGD and CCoT), highlighting their limited applicability to complex visual reasoning tasks.
In contrast, ProReason achieves notable performance improvements, up to 5.1%, by proactively acquiring visual information from images rather than generating question-agnostic captions. This aligns with out previous obervations in Sec. 2.2 that passive methods introduce substantial information redundancy or omission, misleading subsequent reasoning processes.

Decoupling the visual perception and textual reasoning capabilities of an LVLM outperforms their simultaneous inherent usage. As listed in Table 4,
when both capabilities are utilized concurrently, CoT consistently degrades performance compared to the “Direct” method across all benchmarks with Llama3-LLaVA-NeXT-8B model, consistent with the findings in Sec. 2.1. In contrast, despite the same models, ProReason alternates between visual information acquisition and textual reasoning processes, allowing to leverage each capability more effectively. This enables ProReason to consistently outperform CoT with both Llama3-LLaVA-NeXT-8B and GPT-4o-mini across all benchmarks, demonstrating the effectiveness of capability decoupling.

ProReason outperforms ReAct with even less token consumption. Given the similarity in multi-step reasoning, we compare ProReason and ReAct in terms of performance and cost.
Specifically, as an LLM-specific multi-step reasoning framework, ReAct only outperforms the “Direct” method on MME and MMMU, but underperforms on MathVista and HallusionBench, showing inferior performance compared to the consistent improvements of ProReason.
Furthermore, we compare their average token consumption of GPT-4o-mini.
As shown in Table 8 of Appendix A, ProReason consumes significantly fewer tokens than ReAct on both MME and MathVista tasks, indicating its higher token efficiency and the importance of a compact Memory in reducing token usage.
Coupled with better performance, this suggests the superiority of ProReason over LLM-specific ReAct framework.

Text-only LLMs can be effectively integrated into ProReason for dramatically enhanced performance.
As mentioned in Sec. 3.4, the decoupled visual perception and textual reasoning capabilities facilitate the seamless integration of text-only LLMs. To demonstrate the utility of this advantage, we fix the Vision Expert as Llama3-LLaVA-NeXT-8B, and replace other agents with text-only LLMs.
As listed in Table 5, with the assistance of either Qwen2.5-72B-Instruct or GPT-4o-mini, the Llama3-LLaVA-NeXT-8B Vision Expert receives remarkable performance boost across all benchmarks, particularly by 15% on MMMU and 11.7% on MathVista, compared to directly providing answers.
In contrast, ReAct gains a much smaller improvement. This highlights the unique advantage of ProReason in leveraging existing text-only LLMs for enhanced performance. Notably, this advantage may open new avenues for continuously pushing the performance limits of LVLMs with the assistance of existing powerful LLMs.

### 4.3 Relative Importance of Sub-agents

To assess the importance of each sub-agent within the ProReason framework for visual reasoning tasks, we design five scenarios where Llama3-LLaVA-NeXT-8B acts as Dispatcher, Vision Expert, Reasoning Expert, Referee, or Summarizer, respectively, while the other sub-agents are powered by GPT-4o-mini.
Given that Llama3-LaVA-NeXT-8B exhibits weaker visual understanding and textual reasoning capabilities than GPT-4o-mini, the more significant the performance drop incurred by replacing a sub-agent with Llama3-LaVA-NeXT-8B is,
the more important that sub-agent is. Here we primarily consider the MME and MMMU benchmarks due to their comprehensive question coverage. The experimental results are presented in Table 6.

**Table 6: Performance of ProReason across five scenarios for sub-agent assessment on visual reasoning tasks. For each scenario, one sub-agent is replacing with Llama3-LLaVA-NeXT-8B, while the others are performed by GPT-4o-mini. The blue text indicates the performance decline compared to the scenario with all agents performed by GPT-4o-mini.**
| Dataset | GPT-4o-mini | Llama3-LLaVA-NeXT-8B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dispatcher | Vision Expert | Reasoning Expert | Referee | Summarizer |  |  |
| MME | 90.4 | 88.8 (-1.6) | 84.7 (-5.7) | 88.7 (-1.7) | 89.6 (-1.1) | 84.2 (-6.2) |
| MMMU | 61.6 | 60.9 (-0.7) | 54.5 (-7.1) | 60.2 (-1.4) | 51.5 (-10.1) | 51.0 (-10.6) |

Summarizer is the most crucial sub-agent, closely followed by Referee. The replacement of Summarizer results in the most notable performance decline on both MME and MMMU tasks, reaching 6.2% and 10.6%, respectively.
This highlights the critical function of the Summarizer in integrating all available information to conclude final answers. Besides, the substitution of Referee leads to a 10.1% reduction on MMMU. Given that MMMU is more challenging than MME, this finding underscores the essential role of the Referee in assessing the sufficiency of information, particularly in more complex visual reasoning tasks.

Relatively, Dispatcher and Reasoning Expert are the least essential sub-agents.
Specifically, despite a decline, these two sub-agents exhibit significantly less performance degradation than other sub-agents.
This can be attributed to the easier task of the Dispatcher, which requires minimal textual reasoning capabilities, and the infrequent calls of the Reasoning Expert, which is only activated when additional information needs to be inferred—a situation that is rare in current benchmarks.
Besides, both sub-agents operate within the acquisition loop, allowing for greater error tolerance. Even if some error occurs, subsequent iterations can compensate for the missing information.

In summary, each sub-agent contributes to the performance of ProReason, underscoring their necessity. Relatively, the Summarizer and Referee are the most critical sub-agents, while the Dispatcher and Reasoning Expert have the least impact.

**Table 7: Performance of ProReason with different configurations for the relative importance assessment between visual understanding and textual reasoning capabilities on visual reasoning tasks. The red text highlights the performance improvements brought about by the introduction of GPT-4o-mini.**
| Dataset | GPT-4o-mini | Llama3-LLaVA-NeXT-8B |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Textual Sub-Agents | Vision Expert | All Sub-Agents | COT | Direct |  |
| MME | 84.7 (+13.2) | 77.8 (+6.3) | 71.5 | 58.8 | 61.5 |
| MMMU | 54.5 (+2.0) | 53.4 (+0.9) | 52.5 | 41.5 | 41.8 |

### 4.4 Which One is More Crucial: Visual Understanding or Textual Reasoning?

ProReason effectively decouples the visual understanding
and textual reasoning capabilities of LVLMs. However, it remains unclear which of these two capacities is more critical for visual reasoning tasks. To answer this question, we conduct comparative experiments of the following three scenarios:

- •
Llama3-LLaVA-NeXT-8B as All Sub-Agents. All sub-agents within ProReason framework are performed by Llama3-LLaVA-NeXT-8B model.
- •
GPT-4o-mini as Vision Expert.
Based on the above scenario, we implement the Vision Expert with GPT-4o-mini, while keep the other textual sub-agents unchanged.
- •
GPT-4o-mini as Textual Sub-Agents.
Reversely, we utilize Llama3-LLaVA-NeXT- 8B as the Vision Expert, and GPT-4o-mini for the other vision-irrelevant sub-agents.

Textual reasoning capabilities outweigh visual understanding for multi-modal reasoning tasks, although both are important. As shown in Table 7, replacing either the Vision Expert or the other agents with the more capable GPT-4o-mini achieves consistent performance enhancement, highlighting the significance of both capabilities. However, substituting the textual sub-agents with GPT-4o-mini results in a more substantial performance boost compared to replacing the Vision Expert. This underscores the greater importance of textual reasoning over visual understanding for multimodal reasoning tasks, aligning with our previous analysis in Sec. 4.3 that identifies the Summarizer and Referee as the most crucial sub-agents.

## 5 Related work

#### Large Visual-Language Model.

Recently,
large vision-language models (LVLMs) (Bai et al., ; Chen et al., ; Liu et al., ) have
garnered widespread attention and demonstrated remarkable advancements in understanding and generating multi-modal contents.
In the open-source domain, numerous LVLMs, like LLaVA (Liu et al., ; ; ; Li et al., ; ) and InternVL (Chen et al., ) families, have been extensively developed.
In the closed-source domain, proprietary models such as GPT-4 (Achiam et al., ) and Gemini Pro 1.5 (Reid et al., ) have also achieved significantly success.
Despite these advancements, existing LVLMs still encounter challenges in effectively integrating visual understanding with textual reasoning capabilities simultaneously. This limitation is particularly evident in their diminished attention to image content during visual reasoning process, such as chart interpretation and visual math reasoning, leading to degraded performance (Ghosh et al., ) and motivating more effective solutions.

#### Passive Visual Reasoning.

Extracting information from images into text can effectively assist LVLM in performing visual reasoning tasks.
Visual Description Grounded Decoding (VDGD) (Ghosh et al., ) first describes the image before prefixing this description to the prompt, assisting LVLMs on visual reasoning tasks.
Furthermore, Compositional Chain-of-Thought (CCoT) (Mitra et al., ) directs LVLMs to generate scene graphs (SGs) that serve as a bridge between the visual and textual domains, aiding LVLMs in subsequent tasks.
However, most of these methods employ a question-agnostic and reasoning-free visual extraction process, where image descriptions are not tailored to specific questions, and no reasoning is applied to infer additional information for improved descriptions. These “passive” approaches lead to the inclusion of irrelevant or redundant information, ultimately degrading performance.
In contrast, ProReason adopts question-oriented agents to collect necessary and sufficient information, effectively circumventing these drawbacks.

#### Multi-step Reasoning Framework.

Multi-step reasoning frameworks have been developed for LLMs to achieve better performance by breaking down complex questions into easier ones (Pan et al., ). As a representative method, Chain-of-Thought (CoT) (Wei et al., ) enhances the arithmetic and commonsense reasoning capabilities by explicitly generating intermediate reasoning steps before concluding the final answers. Tree-of-Thoughts (ToT) (Yao et al., ) further refines the CoT mechanism by allowing LLMs to consider multiple reasoning paths and do self-assessment before making decisions. Considering that the inherent knowledge of LLMs may not be sufficient to complete tasks, ReAct (Yao et al., ) integrates information retrieval into the reasoning chains, enabling models to pause to verify results and determine whether additional information is needed before proceeding.
Nevertheless, multi-step reasoning frameworks designed for text-only LLMs are not fully applicable to visual reasoning tasks, and may even impair the performance of LVLMs (Ghosh et al., ).

## 6 Conclusion

In this paper, we first validate that existing multi-modal reasoning approaches still suffer insufficient and irrelevant visual descriptions, as well as limited multi-modal capacities.
To address these issues, we decompose the visual reasoning process into visual perception and textual reasoning stages, and introduce a novel visual reasoning framework named ProReason, featuring multi-run proactive perception and decoupled vision-reasoning capabilities.
Empirically, extensive experiments demonstrate the superiority of ProReason over both passive image information acquisition methods and multi-step reasoning frameworks for text-only LLMs across multiple visual reasoning benchmarks with both open-source and closed-source models.
Notably, our method showcases the remarkable feasibility of integrating LLMs for multi-modal reasoning with dramatically improved performance, highlighting the great potential for LLM-assisted LVLM reasoning in future research.

## Appendix A Appendix

### A.1 Demonstrative Examples

Figure: Figure 2: An example with three reasoning frameworks: fine-grained caption, chain-of-thought, and ProReason. ProReason enables LVLMs to proactively acquire necessary information in a question-oriented manner, and predicts answers based on the collected information. Apparently, ProReason is superior to previous methods, which often describe question-irrelevant visual details, or overlook informative elements. Green indicates correct information or conclusions, while red signifies incorrect ones.
Refer to caption: x4.png

Figure: Figure 3: Additional examples of images-unrelated Chain-of-Thought reasoning.
Refer to caption: x5.png

Figure: Figure 4: Additional examples of question-unrelated passive image captions.
Refer to caption: x6.png

### A.2 Prompts

Figure: Figure 5: Prompt templates of Relevance Score and caption effectiveness evaluation.
Refer to caption: x7.png

Figure: Figure 6: Prompt templates of Chain-of-Thought, fine-grained image captions generation, Compositional Chain-of-Thought (CCoT), and ReAct.
Refer to caption: x8.png

Figure: Figure 7: Prompt templates of Dispatcher.
Refer to caption: x9.png

Figure: Figure 8: Prompt templates of Vision Expert, Reasoning Expert, Referee, and Summarizer.
Refer to caption: x10.png

### A.3 Supplementary Results

**Table 8: Average token consumption of ProReason and ReAct with GPT-4o-mini model on the MME and MathVista benchmarks.**
| Method | MME | MathVista |  |  |
| --- | --- | --- | --- | --- |
| Input | Output | Input | Output |  |
| ProReason | 1286.8 | 327.2 | 2238.6 | 788.6 |
| ReACT | 1645.0 | 197.0 | 3092.8 | 845.1 |

**Table 9: Performance of multiple approaches with two base models across four visual reasoning benchmarks. Different from Table 4, MME scores are calculated with the official method here.**
| Model | Method | Dataset |  |  |  |
| --- | --- | --- | --- | --- | --- |
| MME | MMMU | MathVista | HallusionBench |  |  |
| Llama3-LLaVA-<br>NeXT-8B | Direct | 359 | 41.8 | 37.1 | 45.8 |
| VDGD | 374 | 42.3 | 36.1 | 44.2 |  |
| CCoT | 354 | 40.5 | 36.8 | 37.4 |  |
| CoT | 258 | 41.5 | 35.9 | 43.1 |  |
| ReAct | 385 | 46.7 | 31.7 | 43.6 |  |
| ProReason | 416 | 52.5 | 38.8 | 50.9 |  |
| GPT-4o-mini | Direct | 552 | 48.4 | 53.0 | 56.0 |
| VDGD | 584 | 51.4 | 51.2 | 52.4 |  |
| CCoT | 535 | 54.2 | 53.6 | 56.7 |  |
| CoT | 729 | 58.5 | 53.8 | 56.3 |  |
| ReAct | 712 | 54.8 | 49.3 | 51.1 |  |
| ProReason | 779 | 61.6 | 54.9 | 58.9 |  |

**Table 10: Performance of Llama3-LLaVA-NeXT-8B Vision Expert, assisted by different text-only LLMs, on various benchmarks under ReAct and ProReason frameworks.Different from Table 5, MME scores are calculated with the official method here.**
| Vision Expert | Textual<br>Sub-Agents | Method | Dataset |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| MME | MMMU | MathVista | Hallu. |  |  |  |
| Llama3-<br>LLaVA-<br>NeXT-8B | Llama3-LLaVA-<br>NeXT-8B | ReAct | 385 | 46.7 | 31.7 | 43.6 |
| ProReason | 416 | 52.5 | 38.8 | 50.9 |  |  |
| Qwen2.5-72B-<br>Instruct | ReAct | 428 | 50.4 | 34.6 | 40.4 |  |
| ProReason | 603 | 56.8 | 48.8 | 52.3 |  |  |
| GPT-4o-mini | ReAct | 473 | 48.4 | 36.2 | 46.7 |  |
| ProReason | 613 | 54.5 | 41.7 | 53.1 |  |  |

**Table 11: Performance of ProReason across five scenarios for sub-agent assessment on visual reasoning tasks. For each scenario, one sub-agent is replacing with Llama3-LLaVA-NeXT-8B, while the others are performed by GPT-4o-mini. Different from Table 6, MME scores are calculated with the official method here.**
| Dataset | GPT-4o-mini | Llama3-LLaVA-NeXT-8B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dispatcher | Vision Expert | Reasoning Expert | Referee | Summarizer |  |  |
| MME | 779 | 746 | 613 | 743 | 762 | 641 |
| MMMU | 61.6 | 60.9 | 54.5 | 60.2 | 51.5 | 51.0 |

**Table 12: Performance of ProReason with different configurations for the relative importance assessment between visual understanding and textual reasoning capabilities on visual reasoning tasks. Different from Table 7, MME scores are calculated with the official method here.**
| Dataset | GPT-4o-mini | Llama3-LLaVA-NeXT-8B |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Textual Sub-Agents | Vision Expert | All Sub-Agents | COT | Direct |  |
| MME | 613 | 512 | 416 | 286 | 359 |
| MMMU | 54.5 | 53.4 | 52.5 | 41.5 | 41.8 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions are in the weaknesses. Here is the short summary:
1. Detailed evaluations on the Vision expert and Reasoning expert.
2. Calculating the relevance score between standard answers and the information stored in memory could enhance the evaluation of ProReason’s information accuracy.
3. Provide more details in the example for greater clarity.
soundness: 3
strengths: + The paper provides a detailed analysis of the limitations in existing models, highlighting how current LVLMs tend to rely more on language information than on visual cues. This analysis points out issues such as insufficient and irrelevant visual descriptions and limited multi-modal capabilities.
+ The proposed ProReason framework can iteratively generate proactive perception and effectively decouple vision and language capabilities.
+ Extensive results on four benchmarks demonstrate ProReason's effectiveness. Additionally, ProReason’s design allows future integration with LLMs to enhance visual reasoning, showcasing the modulization and the potential for LVLM with LLM-assisted.
summary: This paper discusses the performance degradation problem of Large Vision-language models (LVLMs), which tend to rely more on language knowledge than image information in visual reasoning tasks. The authors first analyze the limitations of current solutions, then introduce the ProReason framework, which incorporates multi-run proactive perception and decoupled vision-language capabilities. It allows integration of the existing LLMs to compensate for the reasoning deficits of LVLMs. In detail, ProReason contains 5 components, including a dispatcher, a vision expert, a reasoning expert, a referee, and a summarizer.  The dispatcher selects a vision expert or a reasoning expert. After several iterations, a referee will decide if the information is enough for a summarizer to answer the question. To demonstrate the effectiveness of the proposed framework, experiments on several benchmarks show that ProReason outperforms existing multi-step reasoning frameworks and passive peer methods.
weaknesses: - The authors claim to 'decouple' multi-modal reasoning into visual perception and textual reasoning. However, they do not provide evaluations on key aspects such as the frequency with which the dispatcher selects the Vision Expert versus the Reasoning Expert, the content of the generated memory from each expert, and the relevance between the memory generated by the Vision Expert and the Reasoning Expert and standard answers.
- The authors compute relevance scores and evaluate caption effectiveness across different methods. However, since ProReason answers questions based on information stored in memory, calculating the relevance score between standard answers and the information within the memory could be an effective way to evaluate whether ProReason captures accurate information.
- In Figure 2, the example doesn’t fully clarify each component’s function, and the prompts differ from those in Figures 7 and 8, creating some inconsistency. Also, it would be great to show how memory is stored and given to each component.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the weaknesses section.
soundness: 2
strengths: + The proposed method presents an effective way to extract the necessary and sufficient visual details from images for the further multi-model reasoning step. 
+ Extensive and comprehensive experiments demonstrate the superiority of the proposed method.
summary: Briefly, this paper presents a multi-step multi-modal reasoning framework to overcome the limitation of LVLMs for visual reasoning tasks by introducing visual perception (i.e., eyesight) and textual reasoning (i.e., wisdom). The experimental results are strong. Besides,  the authors demonstrate the drawbacks of existing solutions (i.e., insufficient yet irrelevant visual descriptions and limited multi-modal capacities).
weaknesses: 1) Sub-optimal Design and Assumption. In Sect. 2.2, the authors argue that a detailed caption of the given image cannot provide sufficient and relevant information for Visual Reasoning (VR). However, some works (e.g., [s1]) concentrate on optimizing the caption. It might be more reasonable to include an optimized caption for the proposed Action step rather than use Q-I as input. 

2) Insufficient Experimental Evaluation. 
   + Why isn't GPT-4V included for comparison? It is somewhat difficult to fairly evaluate the proposed method on some benchmarks (e.g., HallusionBench) that have already reported the performance of GPT-4V.
   + Since the proposed method includes many steps during the inference, it might suffer from accumulated errors, e.g., wrong answers from the action step. The ablation study should include an extra experiment with a detailed discussion.
   + Missing the experiments on the traditional open-domain VQA benchmark (e.g., GQA).  It seems that the time complexity of the proposed method is high for some simple VQA tasks.

3) Due to the poor presentation of the Summary step, the proposed method might only works well on the VR task with multiple choices provided. 

[s1] Hu et al., PROMPTCAP: Prompt-Guided Image Captioning for VQA with GPT-3. In CVPR 2023.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How does the framework handle cases where the required visual information is implicit or requires complex inference from multiple parts of the image? For example, in scenarios where understanding spatial relationships or temporal sequences is crucial?

2. Could the authors elaborate on how the framework might be extended to handle multiple images or video inputs, where temporal reasoning and cross-frame information integration become important?
soundness: 3
strengths: I majorly conclude there are two strengths in this paper.

The paper presents a compelling approach to separating visual perception from textual reasoning, addressing a fundamental limitation of current LVLMs. This decomposition allows for more effective handling of each capability and enables the integration of specialized models for different aspects of the task.

The framework's ability to seamlessly integrate existing LLMs for improved reasoning capabilities is particularly valuable, as it allows organizations to leverage their existing investments in language models while enhancing multi-modal capabilities.
summary: This paper introduces PROREASON, a novel multi-modal reasoning framework that decouples visual perception and textual reasoning capabilities in large vision-language models (LVLMs). The framework features a multi-agent system that proactively collects visual information based on questions and performs reasoning through separate specialized components. The authors demonstrate that PROREASON outperforms existing approaches across multiple benchmarks, with improvements up to 13.2% on standard metrics.
weaknesses: There are two main weakness about this paper.

1. The paper doesn't thoroughly discuss how the framework handles cases where the Vision Expert and Reasoning Expert disagree or provide conflicting information. Suggestion: Add a section analyzing failure cases and how the framework handles conflicting information between agents.

2. The evaluation focuses on specific benchmark tasks, but doesn't extensively explore how the framework scales with increasing complexity of visual scenes or reasoning requirements. Suggestion: Include experiments with varying levels of visual and reasoning complexity to demonstrate scalability.

Meta-review: Paper addresses degradation of the performance of VLMs that stem from them relying less on visual information and more on language priors. The proposed approach amounts to Chain-of-Reasoning with multiple agents, which, in-part decouple visual information gathering and reasoning to produce answers. Paper was reviewed by three expert reviewers and, post rebuttal, received 3 x marginally above the acceptance threshold ratings. Main concerns of reviewers stem from: (1) lack of explanation and motivation for individual components, (2) evaluation of decoupling the vision and language information in processing, (3) lack of experiments with GPT-4V, and (4) lack of more comprehensive evaluation that ensures that intermediate reasoning steps are valid. Authors have provided a comprehensive rebuttal that reviewers have generally found compelling with [zwCa] increasing the rating. However, no reviewer was willing to champion the paper and, unfortunately, [BRG2] did not engage in discussion. 

AC has reviewed the reviews, rebuttal, discussion that followed and the paper itself. Generally, while overall performance of the model is indeed strong and impressive, AC, similar to reviewers, is concerned that internal reasoning may not in fact be consistent and using final performance as a proxy of this (as is suggested by authors in the rebuttal) is generally insufficient. The overall design is also similar to various recent approaches that either do CoT or use tools (e.g., VisualProgramming CVPR'23 or VisualScatchpad from NeurIPS'24). It is not clear which specific designs make the proposed framework more suited for in-depth reasoning in complex scenarios. 

Given the borderline nature of the paper, this paper was also discussed at length between AC and the Senior AC. This discussion has converged to a consensus of Reject recommendation at this time. It was felt that novelty was limited and justification for performance could use improvements. As such, authors are encouraged to revise the paper and to resubmit it to a future venue.

Decision: Reject