Visual Agents as Fast and Slow Thinkers

## Abstract

Abstract Achieving human-level intelligence requires refining cognitive distinctions between System 1 and System 2 thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce FaST , which incorporates the Fa st and S low T hinking mechanism into visual agents. FaST employs a switch adapter to dynamically select between System 1/2 modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a flexible system , hierarchical reasoning capabilities, and a transparent decision-making pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that FaST outperforms various well-known baselines, achieving 80.8% accuracy over V ⁢ Q ⁢ A v ⁢ 2 𝑉 𝑄 superscript 𝐴 𝑣 2 VQA^{v2} italic_V italic_Q italic_A start_POSTSUPERSCRIPT italic_v 2 end_POSTSUPERSCRIPT for visual question answering and 48.7% G ⁢ I ⁢ o ⁢ U 𝐺 𝐼 𝑜 𝑈 GIoU italic_G italic_I italic_o italic_U score over ReasonSeg for reasoning segmentation, demonstrate FaST ’s superior performance. Extensive testing validates the efficacy and robustness of FaST ’s core components, showcasing its potential to advance the development of cognitive visual agents in AI systems. The code is available at https://github.com/GuangyanS/Sys2-LLaVA .

## 1 Introduction

Figure: Figure 1: Working Pipeline. FaST represents a solution rooted in system switching, demonstrating pronounced capabilities in hierarchical reasoning and ad-hoc explainability.
Refer to caption: x1.png

In the field of artificial intelligence, System 2 delineates a cognitive mode distinguished by deliberate, analytical, and consciously reasoned processes [, , , , ]. This mode is juxtaposed to System 1, which embodies intuitive, automatic, and unconscious cognition. Achieving human-level intelligence in AI systems necessitates the deliberate cultivation and refinement of these cognitive distinctions. This process is crucial for the development of advanced reasoning and decision-making capabilities [, , ].

The emergence of foundation models marks a significant turning point, where large language model (LLM) based agents have made remarkable strides in many areas, showcasing human-like intelligence across diverse tasks [, , ]. However, this achievement is primarily attributed to some features of foundation models: overparameterization and the availability of vast, general-purpose datasets [, ]. It is imperative to note that while these models exhibit human-like traits ($e.g.,$ inductive and deductive reasoning [, , ]), these characteristics do not equate to the processes of System 1/2 thinking [, ] and are far less intelligent than human thinking.

In practice, visual agents often encounter challenges when moving from controlled, structured benchmarks to complex, real-world environments [, ]. This difficult circumstance will result in spurious reasoning pathways, akin to hallucinations, where they struggle to acknowledge their limitations or uncertainties [, ]. Such an issue arises from the absence of explicit modeling of the fast and slow cognitive processes, reminiscent of human System 1 and System 2 [, ]. Consequently, when faced with intricate inquiries, the Multimodal Large Language Model (MLLM) frequently offers overly confident yet inaccurate responses [, , ]. Addressing this problem entails reassessing MLLM algorithms to incorporate insights from the interplay between fast and slow thinking (System 1/2) observed in human cognition. This design philosophy guides our work.

In this study, we introduce the Fast and Slow Thinking (FaST) mechanism into visual agents. More concretely, we design a switch adapter to determine whether the encountered problems are best addressed using which thinking mode. Simple tasks require only fast thinking (System 1 thinking) for a straightforward problem-solving pipeline, while complex tasks necessitate the slow, deliberate processing of System 2 (see Figure. 1). Specifically, System 2 is triggered when we encounter visual challenges that have: ① Uncertainty: When the model has low confidence in directly identifying the object to which the complex query is referring. For example, the query asks “the appliance for storing and cooling food” instead of “refrigerator,” and ② Invisibility: When dealing with minuscule-sized objects that evade detection by standard visual encoders, where normal visual agents cannot tell what it is. This switch adapter is achieved by designing negative contextual data to re-adjust the model’s confidence and ignite world knowledge (see detailed analysis in §4.2). Subsequently, a proposal adapter is engaged to outline regions that are related to the questions. This allows visual agents to leverage the newly acquired data, thereby facilitating a more detailed and precise response. Further, if the inquiry necessitates detailed insights into particular instances, a seg adapter provides segmentation masks, offering additional contextual information for deeper analysis (see detailed analysis in §4.3).

FaST enjoys a few attractive qualities. ❶ Flexible system: Building on a foundation that explicitly models System 1/2 thinking, our proposed method adeptly handles complex visual tasks, demonstrating competitive performance in a streamlined pipeline (see §3.2). FaST’s core epistemology combines an intuitive mechanism for straightforward cases with deliberate analytics for more intricate scenarios, thereby enhancing the development of a human-like visual agent. ❷ Hierarchical reasoning: FaST perceives visual tasks with a top-down granularity, encompassing image-level cues, box-level candidates, and pixel-level targets (see Fig. 2). This progressive approach facilitates a sensible understanding of visual content, starting from global concepts, progressing through region-specific candidate assessment, and culminating in precise target identification. Each stage of this process involves developing concrete ideas and establishing a coherent “chain of evidence” to support the final inference. ❸ Transparent pipeline: FaST’s decision-making process embodies a neuro-symbolic essence, yielding intermediate step outputs as readily interpretable symbols ($e.g.$, RoI-driven boxes or masks), facilitating direct visual inspection by humans. This inherent reasoning mechanism enables ad-hoc explainability of the model’s behavior (see Fig. 3), distinguishing FaST from prior approaches [, ] that lack precise explication of their operational mechanisms regularly.

We conducted a series of experiments to validate the efficacy of our proposed method. In §4.1.1, we apply FaST to visual question answering and multimodal benchmarks. FaST demonstrates significantly improved performance over baselines such as LLaVA-v1.5 [], achieving performance gains on benchmarks like TextVQA [] with a 2.5% increase in accuracy and a total score improvement of 6.7 on MME []. In §4.1.2, we explore the versatility of our approach through its application to tasks such as referring and reasoning segmentation, with performance gains including an increase of 4.1% $CIoU$ with LLaVA-v1.5, and improvements of 3.2% $CIoU$ and 2.7% $GIoU$ on the ReasonSeg dataset over LISA-7B []. The robustness and effectiveness of the core components of our framework are further substantiated through a series of ablation studies, as elaborated in §4.3.

## 2 Related Work

LLM as Visual Agents.
With the capabilities that LLMs have demonstrated in language understanding and generation [, , , , ], the research community has progressed to explore how LLMs can be enhanced with vision input for multimodal tasks as visual agents [, , , , , , ]. There are two paradigms for LLM-based visual agents: end-to-end based and tool-using visual agents. Following the principle of instruction tuning, end-to-end visual agents are trained with a curated visual instruction tuning dataset to digest features from multi-modality, unlocking the capability to answer visual questions [, , , , , , , , ].
For other visual tasks ($e.g.$, Segmentation, Detection, etc), end-to-end trained tailored agents can further perform downstream tasks [, , , , , , , , , ].
Recent research has focused on leveraging improved vision encoders and fostering more detailed visual understanding, yielding promising results [, , ]. While these approaches can be implemented with direct instruction tuning data, they represent a ‘System 1’ type of training. This type of training primarily relies on the dataset’s quality and tends to provide direct answers that are prone to hallucinations, a consequence inherent to the nature of System 1 instruction tuning data. For the second paradigm, tool-using models are built on top of a frozen LLM with access to pretrained visual perception tools [, , ]. In this scenario, the LLM first selects visual tools and then decides by thoroughly analyzing the fine-grained information extracted by visual tools [, , ]. Although the utilization of external visual tools increases the interpretability of the reasoning process, inaccuracies can arise in light of the system’s complexity. In addition, due to the complexity of reasoning, the vast amount of information generated by the agent may overwhelm key ${}_{\!}$information ${}_{\!}$related to the question query, ${}_{\!}$leading to incorrect responses.

Our research introduces a novel and adaptable framework designed to enhance response accuracy by adopting distinct slow thinking cognitive modes. Unlike traditional end-to-end visual agents, our framework, FaST, systematically assesses information sufficiency, thereby mitigating the risk of overconfidence. When System 2 (slow, analytical thinking) is activated, FaST employs multiple experts to construct a coherent chain of evidence. This approach ensures the generation of accurate and interpretable responses, significantly advancing the reliability and transparency of visual agents.

System 2 in AI. Recently, LLMs have been engineered to produce text that mimics the step-by-step reasoning process characteristic of human cognition, akin to the analytical and deliberate thought processes associated with what is termed as System 2 in the human cognition process [, , , , ]. The systematic approach to problem-solving is a hallmark across various domains, including mathematical word problems [, , , , , , ], logical reasoning [, , , , , , ], and multi-modal reasoning [, , ]. In Explainable AI, this systematic method is emulated by the model as it generates a text-based elucidation of its reasoning and decision-making process through step by step reasoning process ($e.g.$, chain of thought) [, , , ]. However, it is crucial to recognize that LLMs, while powerful, are not exempt from encountering challenges when facing complex problems. One such challenge is the issue of hallucination [, , , , , ], which can distort the model’s reasoning process and lead to inaccuracies in the explanations provided. Initially, LLM reasoning is seen as only a linear chain of thoughts, where each step in the reasoning process is clearly articulated. As models evolve, they adopt more complex structures like hierarchical trees [, ] and intricate graphs [], which enable them to handle much more complex problems but also restrict their general applicability because of increased topological complexity [, , , , ]. Moreover, these complex structures can lead to errors that propagate through the model’s reasoning, causing a cascade of mistakes []. To counter this, incorporating feedback from intermediate reasoning steps and employing iterative refinement, which is similar to human reflection, could help mitigate errors [, , , , , ]. In unsupervised scenarios, such feedback is vital for enhancing the reasoning capabilities of LLMs and reducing errors [].

Our key contribution is the introduction of the chain of evidence within multimodal reasoning frameworks. This methodology enriches each reasoning step with accurate, image-based cascading information, effectively mirroring human visual and System 2 cognitive processes. Our approach not only enhances accuracy but also significantly improves interpretability and generalization capabilities.

## 3 Methodology

### 3.1 Preliminary

Figure: Figure 2: Slow Thinking Mode of FaST. Our slow thinking mode comprises three core modules: Switch Adapter, which selectively activates a slow and analytical thinking mode when encountering complex visual queries, supplementing with extensive world knowledge to provide missing objects and contextual clues; Proposal Adapter, which identifies and emphasizes regions of interest within the visual inputs; Seg Adapter, which delivers precise pixel-level segmentation, enhancing depth of the visual analysis. The outputs from each module are integrated into a chain of evidence (see Fig. 3), providing a methodical and accurate response. FaST represents a neural-symbolic approach that combines the strengths of symbolic reasoning, ensuring that our system is effective and interpretable.
Refer to caption: x2.png

Notation. The integration of components in visual agents $\mathcal{F}$ (based on the Large Language Model) typically involves a visual encoder, denoted as $\mathcal{E}_{V}$, a nature language encoder, represented by $\mathcal{E}_{L}$, and a language model such as Vicuna []. Initially, the visual agent is presented with an image $\mathcal{I}$ and an accompanying textual prompt $\mathcal{Q}$, which could be a question or instruction. Then the visual agent combines these multimodal tokens into a united space. Finally, the visual agent outputs a textual response $R$ given the textual and image input. The generation process can be expressed as Eq. 1:

$$ $R=\mathcal{F}\left[\mathcal{E}_{V}(I),\mathcal{E}_{L}(Q)\right]$ (1) $$

###### Definition 1

(System 1 and System 2)
System 1 and System 2 are two different systems of thinking proposed by Nobel Laureate Daniel Kahneman in his book Thinking, Fast and Slow [].

System 1 (Fast Thinking System): Unconscious, automated thinking processes, fast, intuitive, effortless, responsible for automatic responses and basic cognitive operations in daily activities, vulnerable to heuristic biases and errors, $e.g.$, recognizing familiar faces, and knowing the location of objects.

System 2 (Slow Thinking System): Conscious, energetic thinking processes, slow, effortful, logical, and analytical, responsible for complex calculations, reasoning, and decision-making, can monitor and control System 1 processes, $e.g.$ filling out a tax form, finding the position of a word in a sentence.

###### Definition 1

### 3.2 FaST

We present FaST (see Fig. 2), a novel framework designed to efficiently handle both simple and complex visual queries. FaST features a dynamic system switch mechanism that enables rapid responses to straightforward questions (System 1) and accommodates deliberate reasoning for intricate scenarios (System 2). During slow thinking, the system uses contextual clues to identify a relevant region, facilitated by a proposal adapter. The adapter generates a bounding box around the target object, and if needed, a pixel-level mask adapter refines the proposal for further details. Finally, we summarize the gathered information from the whole system to provide a comprehensive answer.

System Switch.
Current works on visual agents mostly rely on visual question-answering data, which gives direct answers (System 1) after inquiry as Eq. 1. However, attempting to answer questions directly in this way can compromise the reliability of the responses. Agents tend to hallucinate over questions that require more deliberate reasoning and visual details. To reduce hallucination and make the model reliable, we utilize a system switch trigger to tell when to require more visual information. Specifically, for a question $\mathcal{Q}$ and an image $\mathcal{I}$, we define a MLLM with switch adapter $\mathcal{S}$ and formulate the fast $\mathcal{F}_{fast}$ and slow thinking process $\mathcal{F}_{slow}$. When the query is easy, the frame does not need the switch adapter $\mathcal{S}_{adapter}$ and only output result $\mathcal{R}$ by $\mathcal{F}_{fast}$ as Eq. 2.

$$ $\mathcal{R}=\mathcal{F}_{fast}\left[\mathcal{E}_{V}(I),\mathcal{E}_{L}(Q)\right]$ (2) $$

###### Remark 3.1

(Switching-friendly dataset) A Negative Data for Target Objects Reasoning Dataset $\mathcal{D}$ of 100,000 (image, question, answer) triples was constructed to facilitate the identification of target regions or objects required to answer a question. The dataset constructs questions about the absence or details of certain objects, deliberately made too small to be perceived by the visual encoder.

###### Remark 3.2

(Switch Adapter) A light-weight adapter that is fine-tuned with both positive fast-thinking data and negative data (Def. 3.1) to acquire system switching capability. When the adapter encounters harder questions, the switch mechanism will be triggered for later slow thinking.

Note that a slow thinking process is not always activated. The system switch adapter as Remark. 3.2 $\mathcal{S}_{adapter}$ will determine whether the question for the particular image is sufficient to give a direct answer. If so, the fast mode $\mathcal{F}_{fast}$ will give a quick and direct response as Equation 2. If there is any missing information about the question that current agent cannot solve, the switch adapter will be activated and find the pattern to elicit all the possible missing objects $\mathcal{O}_{missing}$ related to question and context clues $\mathcal{C}_{clue}$ which is the possible location of the missing objects as Equation 3.

$$ $\mathcal{O}_{missing},\mathcal{C}_{clue}=\mathcal{S}_{adapter}\left[\mathcal{E }_{V}(I),\mathcal{E}_{L}(Q)\right]$ (3) $$

Specifically, we use negative data that contain missing objects $\mathcal{O}_{missing}$ and context clues $\mathcal{C}_{clue}$
for training the system switch adapter for triggering the slow mode as Def. 3.2. The slow mode should deal with question-image pairs that are 1) uncertain in pinpointing the specific object in question, and 2) too small to perceive for the standard visual encoder. So we utilize triplet data in dataset Def. 3.1 (image, question, answer) as where the question requires objects that are not in the image or too small to be perceived by the visual encoder. The threshold is set to be $20\times 20$. We require the model to tell that certain objects are missing instead of a direct answer, and we utilize the world knowledge to list all the objects and also the context information for later deliberate reasoning.

Hierarchical Reasoning.
We use a top-down scheme to reason over multi-scale granularity images effectively in order to reason and take advantage of world knowledge progressively. Similar to humans would look for some context clue to find specific objects relating to questions and zoom in if they think the answer lies in a particular region, we model this process with system switch adapters as Eq. 3.2 to focus on the context clue $\mathcal{C}_{clue}$ generated from the switch adapter as Eq. 3.

We denote the MLLM as a proposal adapter $\mathcal{P}_{adapter}$ (visual agent). In System 2, FaST uses many visual agents to accomplish hierarchical reasoning. The frame tries to narrow down the search space by using the question $\mathcal{Q}$ and the previously obtained clue $\mathcal{C}_{clue}$ to let the proposal adapter output a region $Region$ that aligns with the question and the context clue as Eq. 4.

$$ $Region=\mathcal{P}_{adapter}\left[\mathcal{E}_{V}(I),\mathcal{E}_{L}(Q), \mathcal{C}_{clue}\right]$ (4) $$

After getting the region, the visual agents $\mathcal{P}_{adapter}$ will be asked to focus on a more specific target with a bounding box $[Bboxes]$ complemented by the context clue $\mathcal{C}_{clue}$ and region $Region$ get from Eq. 4. This process can reveal the step-by-step reasoning and be modeled as Eq. 5.

$$ $[Bboxes]=\mathcal{P}_{adapter}\left[\mathcal{E}_{L}(Q),Region,\mathcal{C}_{ clue}\right]$ (5) $$

###### Remark 3.3

(Proposal Adapter)
A lightweight adapter that is fine-tuned with proposal data to acquire the capability of finding the corresponding region given the context clue or object name.

###### Remark 3.4

(Pixel-level mask decoder)
The Pixel-level mask decoder is the decoder of segment anything(SAM []). The pixel-level mask decoder is fine-tuned to produce target masks based on the hidden embeddings.

When we have a more specific target proposal(bounding box $[Bboxes]$), FaST will apply a fine-grained pixel-level mask decoder $\mathcal{P}_{Seg}$ as Eq. 3.4 to output the specific mask part $[Mask]$ of the target proposal $[Bboxes]$ to focus on as Eq. 6. We name this whole process from $Region$ to $[Mask]$ chain of evidence as Def. 3.5 similar to thinking more and more deeply by humans.

$$ $[Mask]=\mathcal{P}_{seg}\left[\mathcal{E}_{L}(Q),[Bboxes],\mathcal{O}_{missing }\right]$ (6) $$

Figure: Figure 3: Chain of Evidence. FaST represents a solution rooted in switching, demonstrating pronounced capabilities in hierarchical reasoning and ad-hoc explainability.
Refer to caption: x3.png

###### Remark 3.5

(Chain of Evidence) Chain of evidence is like the chain of thought in a large language model. But we define it as a deeper and deeper step of thinking based on correct evidence in our frame FaST. The completion of the chain of evidence needs many visual agents to work together.

After getting the target proposal (bounding box $[Bboxes]$) from context clue $\mathcal{C}_{clue}$ with proposal adapter and specific mask part $[Mask]$ by missing objects with seg adapter, a chain of evidence is constructed as Remark 3.5 and Fig. 3. Our FaST framework then summarizes all this information ($\mathcal{I}$ and $\mathcal{Q}$) and the chain of evidence with switch adapter to give the final correct reasoning answer $Ans$ as Eq. 7

$$ $Ans=\mathcal{F}_{Slow}\left[\mathcal{E}_{L}(Q),\mathcal{E}_{V}(I),[Bboxes/Mask ]\right]$ (7) $$

The decision-making process inherent in FaST is distinguished by its neuro-symbolic nature, which generates intermediate outputs as easily interpretable symbols, including region-of-interest (RoI) driven boxes and object-driven masks. This capability allows for direct visual inspection by humans, thereby augmenting the transparency of the model’s operations. Moreover, the intrinsic reasoning mechanism of FaST enhances ad-hoc explainability of its behavior, shown in Fig. 2.

###### Remark 3.1

###### Remark 3.2

###### Remark 3.3

###### Remark 3.4

###### Remark 3.5

### 3.3 Implementation Details

The framework of FaST as Fig. 2’s implementation details and in are shown in this section below.

$\bullet$ Visual Agents. We choose the architecture and configuration of LLaVA-v1.5 [] as our visual agent. The most important component in a visual agent is the visual encoder $\mathcal{E}_{V}(I)$: A CLIP-ViT-L-336px model [] is used, where input images are resized or padded to $336*336$ pixels, learning to associate visual features with corresponding textual descriptions. An MLP projection with channels of $[256,4096,4096]$ is used for connecting image representations into the word embedding space.

$\bullet$ Mask Decoder. The mask decoder $\mathcal{P}_{seg}$ architecture is identical to SAM. Besides, it is fully fine-tuned with a collection of semantic segmentation [, , , , ] and referring segmentation [, ] datasets to efficiently map the <seg> token representations to a mask if the FaST need to segment.

$\bullet$ Chain of Evidence. When we apply the chain of evidence as 3.5 in the large language model to get the answer as the final step like Eq. 7. The whole sequence of the chain of evidence is too long to load in the $\mathcal{F}_{Slow}$. So FaST needs a visual sampler based on cross-attention that is trained to decrease the number of image tokens to a suitable length (from 256 to 32), apart from MLP projection.

## 4 Experiment

We utilize eight popular benchmarks to evaluate our framework FaST comprehensively, categorized into general visual question answering (VQA) datasets and multimodal benchmarks. The VQA benchmarks include VQA-v2 [], GQA [], ScienceQA [], and TextVQA [] which focus on optical character recognition. For multimodal benchmarks evaluation, we use the hallucination benchmark POPE [], along with comprehensive benchmarks such as MME [], MM-Vet [], and SEED []. We compare our model with the baseline LLaVA-v1.5 [], and other multimodal large language models. To thoroughly assess our model’s understanding of pixel-level instances, we evaluate its performance on referring segmentation and grounding benchmarks, including refCOCO [],
refCOCO+ [], and refCOCOg []. Further, to examine the model’s reasoning capabilities on FaST framework, we consider the Reasoning Segmentation benchmark [].

### 4.1 Main Results

#### 4.1.1 Experiments on VQA and Multimodal Benchmarks

Training. In developing the Switch Adapter, we employed the LLaVA-v1.5 [] framework, conforming strictly to its established training protocols. We incorporated negative samples from $V^{*}$ [] with contextual cues to enhance system switching capability to amplify multimodal inferential and world knowledge. This augmented dataset was combined with LLaVA-v1.5’s supervised dataset and trained for one epoch. For the Proposal Adapter, we augmented the LLaVA-v1.5 dataset with region-specific bounding boxes based on contextual cues and queries, then fine-tuned for one epoch to optimize proposal generation. The Segmentation Adapter utilized the LISA [] architecture integrated with the LLaVA-v1.5, employing SAM as the mask decoder. The adapter was fine-tuned using the same datasets as Lisa, including semantic segmentation, referring segmentation, and reasoning segmentation. This fine-tuning process involved 10,000 steps to improve the model’s segmentation capabilities. Throughout developing the Switch Adapter, Proposal Adapter, and Segmentation Adapter, we employed the LoRA (Low-Rank Adaptation) technique []. By leveraging LoRA, we introduce minimal additional parameters while preserving the original multimodal large language model’s architecture and efficiency. All experiments used 8 NVIDIA TESLA A100-80GB GPUs.

Metric.
In model evaluation across diverse datasets, various performance metrics are utilized.

*Accuracy.* The primary evaluation metric utilized in the $VQA^{v2}$, GQA, TextVQA, ScienceQA, and SEED benchmarks is accuracy.
Accuracy is a performance measure that quantifies the exact match percentage between predicted and acceptable ground truth answers, indicating a model’s precision.

*F1 Score.* The POPE dataset uses the F1 Score to balance precision and recall, providing a comprehensive assessment by harmonizing the trade-off between positive prediction accuracy and recall.

*Total Score.* The MME evaluation metrics include accuracy (based on individual questions) and accuracy+ (considering both questions per image), reflecting a stricter and more comprehensive model understanding. Random accuracies for these metrics are 50% and 25%, respectively. Perception scores, calculated as the sum of these metrics across subtasks, total 2000 for perception.

*GPT-Evaluation.* In the MM-Vet dataset, performance is evaluated by GPT-4 through a comparative analysis of predicted and ground truth answers, generating a score to quantify alignment.

Results.
As depicted in Table 1, FaST demonstrates superior performance across multiple VQA datasets and multimodal benchmarks when compared to established methods. Notably, FaST surpasses the LLaVA-v1.5 model, with marked improvements in performance across all datasets. Specifically, in the VQA datasets, our model achieves an enhanced accuracy of 2.3% in $VQA^{v2}$, 1.8% in GQA and 2.5% in $VQA^{T}$ comparing to LLaVA-1.5. Similarly, in the multimodal benchmarks, FaST enhances performance notably with an increment of 6.7 in the MME, 1.5 in the SEED and 0.5 in the MM-Vet score, underlining its effectiveness in handling diverse domains. This consistent performance underscores the robustness of FaST, especially in complex visual and textual tasks.

Table: Table 1: Main results on eight VQA and multimodal benchmarks. Our FaST consistently outperforms the baseline LLaVA1.5 model across all evaluated benchmarks, denoted with line $\Delta$.

#### 4.1.2 Experiments on Referring and Reasoning Segmentation

Training.
The training settings for the Switch Adapter and Proposal Adapter remain consistent with those previously described as §4.1.1. During the training phase of the Segmentation Adapter, certain specific datasets are intentionally omitted to uphold an unbiased evaluation of referring and reasoning segmentation datasets. This strategic exclusion is a crucial measure implemented to prevent any potential data leakage, thereby ensuring the integrity and reliability of the evaluation results.

Metric.
Following prior research on segmentation [, ], two evaluation metrics are employed: Generalized Intersection over Union ($GIoU$) and complete Intersection over Union ($CIoU$).

*$CIoU$.* The $CIoU$ is calculated based on the cumulative intersection over the cumulative union across all images in the dataset. This approach can introduce a significant bias towards larger objects or images with more objects, as they contribute more to the cumulative union area.

*$GIoU$.* The $GIoU$ is computed as the average per image $IoU$, where the $IoU$ is calculated for each individual image, and then the average is taken across all images in the dataset. This metric provides a balanced assessment by treating all images equally, regardless of their size or the number of objects.

**Table 2: Main results on referring and reasoning segmentation benchmarks. Our FaST exhibits competitive results in referring segmentation tasks like refCOCOg+ while showcasing superior performance in reasoning segmentation, particularly when evaluated against LISA-7B.**
|  | Referring Segmentation | Reasoning Segmentation |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | refCOCO | refCOCO+ | refCOCOg | ReasoSeg |  |
|  | CIoU | CIoU | CIoU | CIoU | GIoU |
| LAVT[CVPR22]​ [] | 72.7 | 62.1 | 61.2 | - | - |
| OVSeg[CVPR23]​ [] | - | - | - | 28.5 | 18.6 |
| GRES[CVPR23]​ [] | 73.8 | 66.0 | 65.0 | 22.4 | 19.9 |
| X-Decoder[CVPR23]​ [] | - | - | 64.6 | 22.6 | 17.9 |
| SEEM[NeurIPS24]​ [] | - | - | 65.7 | 25.5 | 21.2 |
| LISA-7B[CVPR24]​  [] | 74.1 | 62.4 | 66.4 | 44.4 | 46.0 |
| LLaVA w Seg Adapter | 70.8 | 57.5 | 64.0 | 43.0 | 41.0 |
| FaST (Ours) | 73.3 | 64.4 | 67.0 | 47.6 | 48.7 |
|  |  |  |  |  |  |

Results. Table 2 presents the results of referring and reasoning segmentation benchmarks. FaST compares favorably to recent visual agents like LISA-7B. Notably, FaST outperforms LISA-7B on the refCOCO+ and refCOCOg benchmarks by a substantial margin of 2.0% and 0.6% $CIoU$, respectively. For the more challenging reasoning segmentation task, FaST achieves a significant improvement over LISA-7B, with 3.2% $CIoU$ and 2.7% $GIoU$ gain, demonstrating its efficacy across diverse segmentation benchmarks and showcasing its robustness in handling complex visual reasoning queries.

### 4.2 Analysis of System Switching Adapter

Figure: Figure 4: System 1 Mode Analysis. We investigate the system switching ratio, along with fast thinking performance on easy or hard queries defined by the switch adapter.
Refer to caption: x4.png

Our study investigates the efficacy of the switch adapter mechanism in balancing accuracy and computational efficiency. As depicted in Fig.4, our analysis illustrates the system’s adeptness in discerning between the System 1 and System 2 cognitive modes triggered by query complexity. Notably, the switch adapter activates the System 1 mode in 41.8% and 64.4% of cases for the MME and GQA datasets, respectively. Notably, responses generated under the System 1 mode exhibit significantly higher accuracy rates (78.8% for MME and 64.7% for GQA) compared to those under System 2 mode (52.2% for MME and 56.8% for GQA), highlighting the adapter’s ability to differentiate query complexities and optimize task performance accordingly, emphasizing the importance of maintaining a fast thinking mode for prompt and confident responses.

**Table 3: Runtime Analysis and Comparison on only System 1 (fast), our FaST and only System 2 (slow).**
|  | MME | GQA |  |  |
| --- | --- | --- | --- | --- |
| Method | Runtime | Result | Runtime | Result |
| System 1 Only | 734ms | 1508.7 | 737 | 61.9 |
| System 2 Only | 2938ms | 1518.6 | 2937ms | 64.0 |
| Ours | 2023ms | 1517.4 | 1475ms | 63.8 |

Table 3 offers valuable insights into the runtime comparison of different system configurations.
System 1 Only operates with just a switch adapter, with the slow thinking process disabled. For System 2 Only, we force to construct chain of evidence for every single query. Notably, System 1 only operates efficiently, requiring only one-time inference. In contrast, System 2 only demands significantly more computational resources. Meanwhile, our FaST offers a strategic advantage by optimizing the balance between computational efficiency and performance. It operates approximately 31% faster than System 2 Only for MME and about 50% faster for GQA, while maintaining comparable performance outcomes. FaST enhances the overall efficiency of cognitive processing tasks, offering a pragmatic approach to optimizing computational resources.

### 4.3 Ablation Study

**Table 4: Key Component Analysis**
| Algorithm Component | GQA | POPE | MME |
| --- | --- | --- | --- |
| Baseline | 62.1 | 85.7 | 1509.2 |
| \hdashline $+$ Proposal Adapter | 63.2 | 86.0 | 1516.5 |
| $+$ Seg Adpater | 62.8 | 85.8 | 1514.4 |
| \hdashline<br>Ours (both) | 63.8 | 86.2 | 1517.4 |

Key Component Analysis.
We investigate the core elements of our novel framework FaST, focusing on the proposal adapter for contextual region localization and the seg adapter for pixel-level mask segmentation. To establish a Baseline, we deploy a model devoid of proposal adapter and seg adapter, relying solely on a switch adapter to supply missing objects and context clues. As shown in Table 5, after integrating the proposal adapter, seg adapter, or both, we observe progressive and significant enhancements in performance across various evaluation metrics (62.1% $\rightarrow$ 63.8% accuracy on $VQA^{v2}$). These findings underscore the pivotal roles played by the proposal and seg adapters in augmenting the model’s capabilities, thereby affirming their significance within the FaST framework.

Furthermore, we analyze the output components of our switch adapter in FaST, focusing on including missing objects and context clues. We construct Baseline* model which operates without these enhancements. As illustrated in Table 5, adding the missing objects component shows a noticeable improvement in both MME and refCOCOg scores. Similarly, the integration of context clues also leads to further performance gains, albeit not as significant as missing objects. When both components are integrated, the model demonstrates the best result metrics. These results highlight the significant impact of incorporating missing objects and context clues on the model’s overall effectiveness.

### 4.4 Qualitative Comparisons of FaST

Figure: Figure 5: Qualitative Comparisons of FaST. The top row shows the VQA results on FaST compared to LLaVA-v1.5. The bottom row presents the segmentation results compared to LISA-7B.
Refer to caption: x5.png

In Fig.5, we present qualitative comparisons to illustrate the enhancements introduced by FaST. The top row above the dotted line shows the results for the VQA task, comparing FaST with LLaVA-v1.5. LLaVA-v1.5 fails to identify the correct focusing areas, resulting in incorrect responses. In contrast, FaST leverages the extensive world knowledge of LLMs to construct a chain of evidence ($e.g.$, it can identify a woman on the street or a monitor on the roof). Then FaST determines the appropriate focus area with object-level pixel masks for an accurate and deliberate answer. The bottom row under the dotted line displays segmentation results compared with LISA-7B. LISA-7B exhibits confusion in accurately distinguishing items that are comparatively smaller and require more extensive reasoning to discern. Conversely, FaST effectively isolates the relevant objects with greater granularity and accuracy. This demonstrates the superior capability of FaST in both VQA and segmentation tasks.

## 5 Conclusion

In this study, we introduced FaST, an innovative framework that integrates the concepts of System 1 (fast, intuitive) and System 2 (slow, deliberate) thinking into visual agents, aiming to enhance their reasoning and decision-making capabilities. FaST dynamically adapts to queries of varying complexity by employing a flexible system switch mechanism, ensuring rapid responses for straightforward questions and employing hierarchical reasoning for more intricate scenarios. Our results across a wide array of benchmarks demonstrate significant improvements in performance, highlighting the efficacy of FaST’s chain of evidence methodology in reducing hallucination and enhancing interpretability. Our ablation studies highlight the crucial role of contextual clues and pixel-level adapters in achieving comprehensive visual reasoning and understanding. FaST significantly advances visual agents’ capability, paving the way for the more reliable, accurate, and human-like AI cognition process.

## Appendix A Implementation Details and Pseudo-code of FaST

Hyper-parameters. We follow established methodologies [] and utilize LLaVA-v1.5  [] as the foundational visual agent. The image resolution is preprocessed to 336 × 336 pixels to accommodate the clip-vit-large-patch14-336 vision encoder []. The AdamW optimizer [] is employed with the DeepSpeed ZeRO 2 (^1^11[https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)) configuration for fine-tuning the switch, proposal, and segmentation adapters with LoRA []. For the LoRA configuration, we set the rank to 128 and alpha to 256, consistent with the settings of LLaVA-v1.5. Additionally, we adjust the learning rate of the vision encoder projection layer to 2e-5 to achieve better alignment. An MLP projection with channels of [256, 4096, 4096] is used to connect image representations into the word embedding space for the projection layer. An additional resampler projection layer is used to reduce the number of image tokens.

Training Data for Switch Adapter. Consistent with the pretraining stage of LLaVA-v1.5, we initially pretrain Vicuna-v1.5 as a base frozen large language model and for the MLP projection layer and sampler layer of the CLIP vision encoder using a 558K subset of the LAION-CC-SBU dataset (^2^22[https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain)) with BLIP [] captions. During the fine-tuning stage, we integrate the negative dataset acquired from $V^{*}$ [] with the original LLaVA-v1.5 instruction tuning 665k data(^3^33[https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)) for LoRA based finetuning.

Training Data for Proposal Adapter To determine the corresponding region for a query, we use LRP++ [] for data construction, similar to Chain of Spot []. Our initial prompt is as follows:

The question $Q$ and the context clue $C$ are formatted to get the answer in terms of a bounding box. In this format, $w_{0},w_{1}$ represent the left and right boundaries, respectively, while $h_{0},h_{1}$ denote the upper and lower boundaries. To identify the correct region, we sampled one question per image from the LLaVA instruction tuning data, consisting of a total of 665k data for proposal finetuning.

Training Data for Seg Adapter. Adopting an approach similar to LISA [], the training data for our model comprises three distinct segments: a semantic segmentation dataset, a referring segmentation dataset, and a reasoning segmentation dataset. We deliberately exclude visual question-answering datasets to enhance the model’s segmentation performance. The semantic segmentation segment includes the ADE20K [], COCO-Stuff [], and LVIS-PACO [] part segmentation datasets. The referring segmentation datasets encompass refCOCO[], refCOCO+ [], refCOCOg [], and refCLEF []. The reasoning segmentation dataset includes ReasonSeg []. It is important to note that the referring segmentation and reasoning segmentation datasets are carefully excluded during the evaluation of the segmentation benchmarks to prevent any potential data leakage.

Pseudo-code Implementation.
The pseudo-code of FaST is given in Pseudo-code 1.

Figure: Algorithm 1 Pseudo-code of FaST in a PyTorch-like style.

## Appendix B More Qualitative Results for Visual Question Answering

Figure 7 presents additional qualitative results for Visual Question Answering (VQA). Our FaST framework consistently demonstrates remarkable performance across various challenging scenarios. Notably, in the bottom right corner of Figure 7, our FaST leverages extensive world knowledge to identify the keyboard, which subsequently aids in discovering the hidden computer mouse and providing the correct answer. This ability to integrate and utilize contextual information showcases the model’s advanced capabilities and highlights its potential for practical applications. The qualitative results further underscore FaST’s robustness and versatility in handling diverse VQA tasks.

Figure: Figure 6: More qualitative results for Visual Question Answering.
Refer to caption: x6.png

Figure: (a) The model fails to trigger System 2 thinking mode.
Refer to caption: x8.png

## Appendix C More Qualitative Results for Segmentation

Figure 7 showcases further qualitative results for the Segmentation task. Our FaST model excels in various challenging scenarios, accurately locating difficult targets and performing complex reasoning for more demanding queries. For instance, in the bottom right corner, the model successfully identifies an appliance that can be turned on when feeling hot by recognizing relevant contextual clues that suggest the appliance should probably appear on the wall, thereby resulting in the correct answer. This example demonstrates the model’s advanced understanding, adaptability, and precision.

## Appendix D Failure Case

In Fig. 8, we present an overview of the most notable failure cases, providing insights into the distinct patterns that lead to suboptimal outputs in our FaST model. These challenges include difficulty in triggering the System 2 thinking mode, constructing adequate contextual clues, generating appropriate proposals, and providing accurate pixel masks. The model often fails to recognize the need for deliberate reasoning, relying instead on System 1 thinking, which leads to incorrect responses, as seen in Fig. 8. Inadequate contextual clues generated by the switch adapter impair the model’s focus on the correct region, resulting in vague or incorrect responses, as illustrated in Fig. 8. The proposal adapter’s inaccurate identification of regions of interest, as shown in Fig. 8, leads to proposals that do not correspond to the query. Additionally, the segmentation adapter struggles with producing precise masks, particularly for small or occluded objects, causing erroneous conclusions, as highlighted in Fig. 8. These failure cases underscore the urgent need for refinement in our FaST framework, emphasizing the importance of significantly enhancing the precision of the system switch adapter, improving contextual clue construction, and optimizing the proposal and segmentation adapters to achieve more reliable and consistently accurate responses in complex visual and textual tasks.

## Appendix E Limitation and Future Work

While the FaST framework has demonstrated significant advancements in emulating human-like cognitive processes in visual AI through its fast and slow thinking mechanisms, several limitations warrant attention. Firstly, the system’s reliance on a predefined set of negative data for training the switch adapter may not encapsulate the full spectrum of real-world complexities. Firstly, this could lead to suboptimal performance when faced with novel or unexpected scenarios. Secondly, despite its fine-grained analysis capability, the pixel-level mask decoder might struggle with highly textured or patterned images where segmentation becomes challenging. Lastly, the generalizability of FaST across various domains and tasks necessitates further validation to ensure its robustness and reliability in diverse applications. We plan to develop advanced learning mechanisms that will allow the model to generalize more effectively beyond the predefined negative dataset. Additionally, we will focus on optimizing it for real-time applications to reduce computational overhead and response times.

## Appendix F Social Impacts

The development and deployment of FaST carry significant implications for the field of AI, especially in augmenting the human-like cognitive capabilities of LLM-based visual agents. On the positive side, FaST’s emulation of human cognitive processes holds promise for creating more advanced AI systems that can assist in critical areas like general vision question dialogue and security surveillance, where precise visual comprehension is paramount. Moreover, its transparent decision-making pipeline contributes to the explainability of AI, fostering trust and facilitating ethical AI practices. However, it’s imperative also to acknowledge potential negative social impacts. The reliance on extensive models and datasets could perpetuate biases if not meticulously curated, potentially resulting in unjust or discriminatory outcomes. Addressing these ethical considerations and establishing guidelines for the responsible utilization of such sophisticated AI systems are paramount requirements.

## Appendix G Ethical Safeguards

In our paper introducing a novel framework FaST, we implement rigorous ethical measures to prevent potential misuse and promote responsible application. These measures are delineated in comprehensive protocols accompanying the final release of models and datasets. Our protocols encompass stringent usage guidelines, access controls, incorporation of safety filters, and monitoring systems. These concerted efforts reflect our steadfast dedication to upholding the utmost ethical standards in scientific exploration. Our objective is to protect the rights and privacy of all stakeholders involved, thereby fostering a culture of responsible and ethical research within our community.

## Appendix H Reproducibility

Our FaST framework is implemented in PyTorch []. All the experiments are conducted on eight NVIDIA A100-80GB GPUs. Our full implementation shall be publicly released upon paper acceptance to guarantee reproducibility. The codes are available at the anonymous link [https://anonymous.4open.science/r/Sys2-LLaVA-8B0F/](https://anonymous.4open.science/r/Sys2-LLaVA-8B0F/) for the review process.

All Experiments (switch, proposal, and seg adapter) are conducted on eight NVIDIA A100-80GB SXM GPUs(^4^44[https://www.nvidia.com/en-sg/data-center/a100/](https://www.nvidia.com/en-sg/data-center/a100/)). Reproducing the fine-tuning process would require approximately 15 A100 GPU days.

## Appendix I Licenses for existing assets

All the methods we used for comparison are publicly available for academic usage. The switch adapter is implemented based on the released code ([https://github.com/penghao-wu/vstar](https://github.com/penghao-wu/vstar)) with an MIT license. The proposal adapter is implemented on the released code ([https://github.com/dongyh20/Chain-of-Spot](https://github.com/dongyh20/Chain-of-Spot)) with an Apache-2.0 license. The seg adapter is implemented on the released code ([https://github.com/dvlab-research/LISA](https://github.com/dvlab-research/LISA)) with an Apache-2.0 license.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Please see weaknesses above. I believe the paper in its current form already presents promising ideas+results but can be made still stronger.

– It would be good include a discussion / conceptual comparison to recent “large reasoning models” that propose scaling test-time compute eg. the OpenAI o1 model. While not (yet) multimodal, such methods are speculated to be trained w/ reinforcement learning and to produce “internal” (i.e. not necessarily interpretable) chain of thought – what are some of the tradeoffs of this type of approach against FaST?

– [Minor] As I understand, FaST does not provide increased transparency when it opts for “fast” thinking – it would be good to clearly demarcate this.

– [Minor] L538 typo: extra "A"
soundness: 3
strengths: – The paper presents a highly novel solution to an important problem

– The paper is very well-written and easy to follow

– The motivation for the approach is laid out very clearly and is compelling

– The proposed approach  is designed intuitively, and shows convincing improvements over prior work

– The neuro-symbolic nature of the approach directly translates to improved transparency

– The paper presents a comprehensive set of results and ablation studies on visual question answering and referring and reasoning segmentation benchmarks. The qualitative results included in the paper are also very compelling

– The paper includes a detailed appendix and code implementation which will aid in reproducibility
summary: Proposes FaST, a Fast-and-Slow-Thinking mechanism for vision-language models that models both “system 1” (fast) and “system 2” (slow) thinking, dynamically switching between the two based on query complexity. The method demonstrates improvements over baselines on standard benchmarks in addition to improved reasoning and transparency.
weaknesses: – [Complexity] One concern I have is the complexity of the approach, especially considering the large number of moving parts. Do the same set of method hyperparameters generalize across settings, or was a large amount of tuning required to achieve the results reported in the paper? 

– [Clarity] I found some of the method details to be lacking eg. L259-260 mentions “a fast sampler based on cross-attention” but I could not find any further details for this component.

– [Performance] While FaST does seem to consistently outperform the LLaVA-1.5 base approach that it builds on top of, it is not exactly an apples-to-apples comparison considering the additional models /  datasets that are used (including large ones such as SAM). Meanwhile, its gains over methods reasoning-focused approaches like Visual CoT and Chain-of-Spot are modest (and sometimes lags considerably eg. behind VCoT on VQAT) – a deeper analysis of / comparison against these methods (eg. average runtime etc.) would be helpful to establish the pros/cons of using FaST. The comparison against CoVLM and CogCOM appears more significant (Fig 4) but is missing a comparison to the LLaVA-1.5 baseline w/ the stronger encoder. 

– [Analysis] The results presented in Figure 5 are very interesting, but are missing baseline performance of the base LLaVA-1.5 model: what is its accuracy on the subset of system 2 / system 1 questions (as judged by FaST) in GQA/MME? How much does “slow” thinking actually improve performance on such questions (assuming that the switch adapter is accurate – it would also be good to validate this eg. by collecting question complexity labels)?

– [Analysis] It would be valuable to include an evaluation of the correctness/quality of the “chain-of-evidence” generated by the approach, perhaps via a human study and/or model interpretability methods. Does the chain-of-evidence appear faithful or is it mostly a post-hoc rationale? How often does a “correct” chain of evidence lead to the right conclusion, and vice versa? Is the chain-of-evidence used for the model “optimal”, or does it include unnecessary steps or logical leaps? This will help build confidence around the improved transparency claims made in the paper.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Can the authors provide more details in the synthesis of the dataset used for switch adapter finetuning? Specifically, how are the negative samples from V* augmented to cover the two cases that are expected to trigger a slow thinking mode switching (uncertainty in pinpointing objects or the targets are too small)?    
2. Will the datasets collected by the authors be released? The reviewer believes that these datasets are crucial for reproducing the FAST framework proposed in the paper, and they will also be highly beneficial for future related work.
3. Can the authors provide some experimental results for other reasoning tasks, like raven, bongard or physical reasoning tasks?
soundness: 3
strengths: 1. FaST strikes a balance between accuracy and efficiency for visual agents by dynamically integrating fast and slow reasoning. The motivation to incorporate different thinking modes based on the scenario is intuitive, and the primary concept is presented clearly and logically.
2. The interpretability and transparency of the proposed framework are improved through the introduction of a chain-of-evidence mechanism and the generation of symbolic intermediate results as feedback.
3. Experimental results across various tasks validate the effectiveness of the proposed framework.
summary: The paper presents FaST, a framework for improving the cognitive capacities of visual agents.   
Inspired by System 1 and System 2 thinking in human’s cognition, FaST incorporates a switch adapter that dynamically toggle between a faster, intuitive response mode and a slower, analytical reasoning mode depending on the task complexity. 
FaST also features a high interpretability by integrating hierarchical reasoning strategies and symbolic intermediate outputs into the framework.
Experiments show that the proposed outperforms baselines in various VQA and multimodal benchmarks.
weaknesses: 1. The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks (e.g., the Bongard problem, physical reasoning tasks) to further validate the generalization capabilities of the proposed method.
2. The efficacy of the switch adapter can be further analyzed in Section 3.2:
  * No investigations have been conducted on the switching ratio and performance on reasoning segmentation and other tasks, which is pivotal to demonstrate the universal compatibility of the switch adapter mechanism across different families of tasks.
  * The description of accuracy rates “under System 2 mode” contains some ambiguities. The authors should clarify in the text that accuracy rates reported under System 2 are also evaluated using fast thinking mode, otherwise, the claim regarding the adapter’s ability to differentiate problem complexities is not adequately supported. 
  * The accuracy rates mentioned in the passage do not appear to match those shown in Figure 5.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: The major concerns are listed in the weakness part.

Minimal concern: It is definitely not a good idea to use "fast" as an acronym for "fast" and "slow".

Potential Typos:
- Lines 151-156: Format like "Equation 1" does not align with Sec.2.2 "Eq.3"
- Line 160: "Def.2.2", only Remark 2.2 exists
- Line 196: "Eq.2.1", only Remark 2.1 exists
- Line 215: "Eq.2.4", only Remark 2.4 exists
soundness: 3
strengths: - The method is built based on the idea of Kahneman's system 1/2 theory from cognitive science, which is clearly-motivated and reasonable.

- The approach is clearly written and its implementation is reasonable.

- The authors provide comprehensive evaluation across multiple benchmarks and tasks. And the method outperforms baselines.

- The authors provide clear visualizations of the model performance.
summary: The paper proposes a slow-fast paradigm for MLLMs based on Kahneman's System 1/2 therory. The framework consists of a "fast" module built upon MLLM and vision foundation models. A switch adaptor will determine if it's necessary to use "slow" module, which will gather all detailed clues to find a more accurate result. The approach shows impressive results across multiple benchmarks and outperforms baselines models.
weaknesses: - The analogy between human cognitive system 1-2 and the "fast" "slow" module in the proposed method lacks rigor justification, or even misaligned. In human cognition, System 1 usually handles low-level tasks and is much faster, while System 2 is slow but handles complexity. However, in FaST, both the fast and slow modules solves same problem, but only have a tradeoff between efficiency and performance. Considering that the "slow" module is only 4x slower, we can abandon the switch adaptor and use the slow module all the time for a maximal performance.
- From another perspective, if the only purpose of two stage model is the computation efficiency, then the proposed approach is more like an early-exit trick of a complex model (referring to the "slow" module). THis diminish the novelty of the paper.
- The framework essentially implements a manual Chain of Thought (CoT) approach rather than dual cognitive system. If reframing the method in to CoT paradigm, a progressive visual cue integration would be potentially developed (more than 2 layers). This approach would gradually add visual evidence into the reasoning process, which could be both more efficient and accurate.
- The complexity burden is only **shifted** to the switch adaptor, rather than being alleviated or addressed. So what is the performance of the switch adaptor itself?
- The details of the datasetin remark 2.1 is missing. It is critical to introduce the construction process of the dataset.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Overall, I found the paper very interesting, my main concern is within the weak points. Another question I have is, if this FaST can outperform other models by using only System 2, what are the advantages of switching between System 1 and System 2 instead of just System 2?  I think time cost would be an answer, but I'd be curious if the author provides more answers.




typo: In Line 538, AFurthermore should be Furthermore
soundness: 3
strengths: 1. The motivation for incorporating systems 1 and 2 in visual agents seems promising and convincing. Especially the design of system 2, the introduction of clue, seg instance and chain of evidence can assist the reasoning process based on experiment results. The construction of Negative Data for Target Objects Reasoning Dataset seems useful for training the adapter switcher. 
2. The hierarchical reasoning capabilities that start with global image cues and drill down to pixel-level analysis provide a well-rounded method of handling visual inputs. The symbolic approach can provide more explainability on the model reasoning process.
2. A comprehensive experiment, including many current models, proves the effectiveness of FaST. The ablation study on pipeline structure and the time comparison highlight the advantage of FaST.
3. Overall, I found the paper well written, with clear definitions and formulas to demonstrate the complex system. The motivation based on human cognitive science is well presented.
summary: The paper introduces FaST, a framework based on human reasoning that incorporates fast (system 1) and slow (system 2) thinking mechanisms into visual agents. FaST involves a switch adapter to change between systems 1 and 2 based on the complexity of the query question. While System 1 reasons the question in a short or one-step approach, system 2 uses a switch adapter to provide clues and object names, which later inspires the proposal adapter to provide region boxes and seg adapter to provide seg instances. In the end, all additional information will be part of a chain of evidence and prompt LLMs to do "slow" reasoning to output an answer. The paper does a comprehensive experiment comparing FaST with current models and demonstrates the effectiveness of FaST on both visual question answering and segmentation reasoning. Further ablation studies prove the significance of each component within the Fast.
weaknesses: 1. Result Demonstration: based on the paper, I think we can assume the previous model mainly uses fast thinking on reasoning questions. A better way to illustrate the effectiveness of FaST is to split Table 1 with the model's system 1 and system 2 performance. In other words, for all those questions in which FaST use system 1 reasoning, compare the performance in one table and compare all questions with system 2 in another table.  I expect the improvement of FaST mainly comes from system 2, which means FaST will show similar performance with other models in questions that require system 1 reasoning.
2. Dependence on Specific Datasets: I appreciate the work constructing the “Negative Data for Target Objects Reasoning Dataset" to train the switch adapter. However, I am concerning the training data can not reflect all real-world scenario, which may limit generalizability. Is it possible for us to gain the signal from the model's inner representation to decide the switch between systems 1 and 2?
3. Failure case: the paper should also explore scenarios where it fails or underperforms compared to baseline models. As the FaST use a hierarchical reasoning pipeline, it is possible the error of some component can make the whole system collapse or lead to a cascade of issues. An analysis of a failure case can provide insight into the model's weaknesses and future directions.

Meta-review: This work presented a new reasoning mechanism which can adaptively switch between "system 1" and "system 2" for multimodal LLMs. The authors proposed a top-down reasoning pipeline for visual agent which starts from the global view of the visual input and gradually drill down to the detailed visual observatrions to derive the final outputs, either for visual question answering or reasoning segmentation. To make the visual agent adaptively, the authors proposed a switch module and train it with carefully curated data to enable automatic. The experimental results demonstrate that the proposed system-1/2 visual agent achieve the best of both world for both simple tasks that do not require long-run reasoning and complicated tasks that necessate the reasoning process. Some further visualizations can clearly demonstrate the reasoning steps which enhances the interpretability substantially.

The main contribution of this work is the proposed fast and slow thinking mechanism for the multimodal LLMs. The paper is well-written and movtivated. The method is also appeadling and systematic per the comments by reviewers. Overall, the ACs think this work is a great way of enhancing the explicit reasoning capability for multimodal LLMs, which mirrors the current tremendous efforts to improve the reasoning capability in language domains for coding and mathematical tasks.

After the engaging discussion period, most of the concerns raised by the reviewers were addressed by the authors. From the ACs' perspective, the point made by reviewer GSxm that "The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks." makes much sense as image QA and segmentation tasks are mostly perception tasks without reliance on complicated reasoning as also pointed out by the authors and other reviewers. To really make the vision model as an agent as claimed by "Visual Agents", it is supposed to come with more  capable system-1/2 thinking for more agentic tasks. 

Overall, three reviewers gave positive ratings on this submission, and one reviewer gave 5 but with most of the concerns addressed according to the reply. The ACs also agree that this is a nice work on a systematical way of enabling system-1/2 thinking for multimodal LLMs, despite it should not count as a visual agent per se. As a result, the ACs recommend a clear acceptance of this work.

Decision: Accept (Poster)