Visual Agents as Fast and Slow Thinkers

Authors: Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, Dongfang Liu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal Large Language Model, System2 Thinking, Language Agent

Abstract:
Achieving human-level intelligence requires refining cognitive distinctions between \textit{System 1} and \textit{System 2} thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce \textbf{\textsc{FaST}}, which incorporates the \textbf{Fa}st and \textbf{S}low \textbf{T}hinking mechanism into visual agents. \textsc{FaST} employs a switch adapter to dynamically select between \textit{System 1/2} modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a \textit{flexible system}, \textit{hierarchical reasoning} capabilities, and a \textit{transparent decision-making} pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that \textsc{FaST} outperforms various well-known baselines, achieving 80.8\% accuracy over $VQA^{v2}$ for visual question answering and 48.7\% $GIoU$ score over ReasonSeg for reasoning segmentation, demonstrate \textsc{FaST}'s superior performance. Extensive testing validates the efficacy and robustness of \textsc{FaST}'s core components, showcasing its potential to advance the development of cognitive visual agents in AI systems.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730431748382

thread_type:
official_review

<thread_object>
creator: reviewer_knej

<content>
summary:
The paper introduces FaST, a framework based on human reasoning that incorporates fast (system 1) and slow (system 2) thinking mechanisms into visual agents. FaST involves a switch adapter to change between systems 1 and 2 based on the complexity of the query question. While System 1 reasons the question in a short or one-step approach, system 2 uses a switch adapter to provide clues and object names, which later inspires the proposal adapter to provide region boxes and seg adapter to provide seg instances. In the end, all additional information will be part of a chain of evidence and prompt LLMs to do "slow" reasoning to output an answer. The paper does a comprehensive experiment comparing FaST with current models and demonstrates the effectiveness of FaST on both visual question answering and segmentation reasoning. Further ablation studies prove the significance of each component within the Fast.

strengths:
1. The motivation for incorporating systems 1 and 2 in visual agents seems promising and convincing. Especially the design of system 2, the introduction of clue, seg instance and chain of evidence can assist the reasoning process based on experiment results. The construction of Negative Data for Target Objects Reasoning Dataset seems useful for training the adapter switcher. 
2. The hierarchical reasoning capabilities that start with global image cues and drill down to pixel-level analysis provide a well-rounded method of handling visual inputs. The symbolic approach can provide more explainability on the model reasoning process.
2. A comprehensive experiment, including many current models, proves the effectiveness of FaST. The ablation study on pipeline structure and the time comparison highlight the advantage of FaST.
3. Overall, I found the paper well written, with clear definitions and formulas to demonstrate the complex system. The motivation based on human cognitive science is well presented.

weaknesses:
1. Result Demonstration: based on the paper, I think we can assume the previous model mainly uses fast thinking on reasoning questions. A better way to illustrate the effectiveness of FaST is to split Table 1 with the model's system 1 and system 2 performance. In other words, for all those questions in which FaST use system 1 reasoning, compare the performance in one table and compare all questions with system 2 in another table.  I expect the improvement of FaST mainly comes from system 2, which means FaST will show similar performance with other models in questions that require system 1 reasoning.
2. Dependence on Specific Datasets: I appreciate the work constructing the “Negative Data for Target Objects Reasoning Dataset" to train the switch adapter. However, I am concerning the training data can not reflect all real-world scenario, which may limit generalizability. Is it possible for us to gain the signal from the model's inner representation to decide the switch between systems 1 and 2?
3. Failure case: the paper should also explore scenarios where it fails or underperforms compared to baseline models. As the FaST use a hierarchical reasoning pipeline, it is possible the error of some component can make the whole system collapse or lead to a cascade of issues. An analysis of a failure case can provide insight into the model's weaknesses and future directions.

questions:
Overall, I found the paper very interesting, my main concern is within the weak points. Another question I have is, if this FaST can outperform other models by using only System 2, what are the advantages of switching between System 1 and System 2 instead of just System 2?  I think time cost would be an answer, but I'd be curious if the author provides more answers.




typo: In Line 538, AFurthermore should be Furthermore
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667115841

thread_type:
official_review

<thread_object>
creator: reviewer_uxt1

<content>
summary:
The paper proposes a slow-fast paradigm for MLLMs based on Kahneman's System 1/2 therory. The framework consists of a "fast" module built upon MLLM and vision foundation models. A switch adaptor will determine if it's necessary to use "slow" module, which will gather all detailed clues to find a more accurate result. The approach shows impressive results across multiple benchmarks and outperforms baselines models.

strengths:
- The method is built based on the idea of Kahneman's system 1/2 theory from cognitive science, which is clearly-motivated and reasonable.

- The approach is clearly written and its implementation is reasonable.

- The authors provide comprehensive evaluation across multiple benchmarks and tasks. And the method outperforms baselines.

- The authors provide clear visualizations of the model performance.

weaknesses:
- The analogy between human cognitive system 1-2 and the "fast" "slow" module in the proposed method lacks rigor justification, or even misaligned. In human cognition, System 1 usually handles low-level tasks and is much faster, while System 2 is slow but handles complexity. However, in FaST, both the fast and slow modules solves same problem, but only have a tradeoff between efficiency and performance. Considering that the "slow" module is only 4x slower, we can abandon the switch adaptor and use the slow module all the time for a maximal performance.
- From another perspective, if the only purpose of two stage model is the computation efficiency, then the proposed approach is more like an early-exit trick of a complex model (referring to the "slow" module). THis diminish the novelty of the paper.
- The framework essentially implements a manual Chain of Thought (CoT) approach rather than dual cognitive system. If reframing the method in to CoT paradigm, a progressive visual cue integration would be potentially developed (more than 2 layers). This approach would gradually add visual evidence into the reasoning process, which could be both more efficient and accurate.
- The complexity burden is only **shifted** to the switch adaptor, rather than being alleviated or addressed. So what is the performance of the switch adaptor itself?
- The details of the datasetin remark 2.1 is missing. It is critical to introduce the construction process of the dataset.

questions:
The major concerns are listed in the weakness part.

Minimal concern: It is definitely not a good idea to use "fast" as an acronym for "fast" and "slow".

Potential Typos:
- Lines 151-156: Format like "Equation 1" does not align with Sec.2.2 "Eq.3"
- Line 160: "Def.2.2", only Remark 2.2 exists
- Line 196: "Eq.2.1", only Remark 2.1 exists
- Line 215: "Eq.2.4", only Remark 2.4 exists
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668781282

thread_type:
official_review

<thread_object>
creator: reviewer_gsxm

<content>
summary:
The paper presents FaST, a framework for improving the cognitive capacities of visual agents.   
Inspired by System 1 and System 2 thinking in human’s cognition, FaST incorporates a switch adapter that dynamically toggle between a faster, intuitive response mode and a slower, analytical reasoning mode depending on the task complexity. 
FaST also features a high interpretability by integrating hierarchical reasoning strategies and symbolic intermediate outputs into the framework.
Experiments show that the proposed outperforms baselines in various VQA and multimodal benchmarks.

strengths:
1. FaST strikes a balance between accuracy and efficiency for visual agents by dynamically integrating fast and slow reasoning. The motivation to incorporate different thinking modes based on the scenario is intuitive, and the primary concept is presented clearly and logically.
2. The interpretability and transparency of the proposed framework are improved through the introduction of a chain-of-evidence mechanism and the generation of symbolic intermediate results as feedback.
3. Experimental results across various tasks validate the effectiveness of the proposed framework.

weaknesses:
1. The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks (e.g., the Bongard problem, physical reasoning tasks) to further validate the generalization capabilities of the proposed method.
2. The efficacy of the switch adapter can be further analyzed in Section 3.2:
  * No investigations have been conducted on the switching ratio and performance on reasoning segmentation and other tasks, which is pivotal to demonstrate the universal compatibility of the switch adapter mechanism across different families of tasks.
  * The description of accuracy rates “under System 2 mode” contains some ambiguities. The authors should clarify in the text that accuracy rates reported under System 2 are also evaluated using fast thinking mode, otherwise, the claim regarding the adapter’s ability to differentiate problem complexities is not adequately supported. 
  * The accuracy rates mentioned in the passage do not appear to match those shown in Figure 5.

questions:
1. Can the authors provide more details in the synthesis of the dataset used for switch adapter finetuning? Specifically, how are the negative samples from V* augmented to cover the two cases that are expected to trigger a slow thinking mode switching (uncertainty in pinpointing objects or the targets are too small)?    
2. Will the datasets collected by the authors be released? The reviewer believes that these datasets are crucial for reproducing the FAST framework proposed in the paper, and they will also be highly beneficial for future related work.
3. Can the authors provide some experimental results for other reasoning tasks, like raven, bongard or physical reasoning tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742732064

thread_type:
official_review

<thread_object>
creator: reviewer_m2ti

<content>
summary:
Proposes FaST, a Fast-and-Slow-Thinking mechanism for vision-language models that models both “system 1” (fast) and “system 2” (slow) thinking, dynamically switching between the two based on query complexity. The method demonstrates improvements over baselines on standard benchmarks in addition to improved reasoning and transparency.

strengths:
– The paper presents a highly novel solution to an important problem

– The paper is very well-written and easy to follow

– The motivation for the approach is laid out very clearly and is compelling

– The proposed approach  is designed intuitively, and shows convincing improvements over prior work

– The neuro-symbolic nature of the approach directly translates to improved transparency

– The paper presents a comprehensive set of results and ablation studies on visual question answering and referring and reasoning segmentation benchmarks. The qualitative results included in the paper are also very compelling

– The paper includes a detailed appendix and code implementation which will aid in reproducibility

weaknesses:
– [Complexity] One concern I have is the complexity of the approach, especially considering the large number of moving parts. Do the same set of method hyperparameters generalize across settings, or was a large amount of tuning required to achieve the results reported in the paper? 

– [Clarity] I found some of the method details to be lacking eg. L259-260 mentions “a fast sampler based on cross-attention” but I could not find any further details for this component.

– [Performance] While FaST does seem to consistently outperform the LLaVA-1.5 base approach that it builds on top of, it is not exactly an apples-to-apples comparison considering the additional models /  datasets that are used (including large ones such as SAM). Meanwhile, its gains over methods reasoning-focused approaches like Visual CoT and Chain-of-Spot are modest (and sometimes lags considerably eg. behind VCoT on VQAT) – a deeper analysis of / comparison against these methods (eg. average runtime etc.) would be helpful to establish the pros/cons of using FaST. The comparison against CoVLM and CogCOM appears more significant (Fig 4) but is missing a comparison to the LLaVA-1.5 baseline w/ the stronger encoder. 

– [Analysis] The results presented in Figure 5 are very interesting, but are missing baseline performance of the base LLaVA-1.5 model: what is its accuracy on the subset of system 2 / system 1 questions (as judged by FaST) in GQA/MME? How much does “slow” thinking actually improve performance on such questions (assuming that the switch adapter is accurate – it would also be good to validate this eg. by collecting question complexity labels)?

– [Analysis] It would be valuable to include an evaluation of the correctness/quality of the “chain-of-evidence” generated by the approach, perhaps via a human study and/or model interpretability methods. Does the chain-of-evidence appear faithful or is it mostly a post-hoc rationale? How often does a “correct” chain of evidence lead to the right conclusion, and vice versa? Is the chain-of-evidence used for the model “optimal”, or does it include unnecessary steps or logical leaps? This will help build confidence around the improved transparency claims made in the paper.

questions:
Please see weaknesses above. I believe the paper in its current form already presents promising ideas+results but can be made still stronger.

– It would be good include a discussion / conceptual comparison to recent “large reasoning models” that propose scaling test-time compute eg. the OpenAI o1 model. While not (yet) multimodal, such methods are speculated to be trained w/ reinforcement learning and to produce “internal” (i.e. not necessarily interpretable) chain of thought – what are some of the tradeoffs of this type of approach against FaST?

– [Minor] As I understand, FaST does not provide increased transparency when it opts for “fast” thinking – it would be good to clearly demarcate this.

– [Minor] L538 typo: extra "A"
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210229426

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all Reviewers

comment:
To all reviewers:

Thank you for your thorough review and insightful comments. We have revised our paper according to the suggestions. The major changes are summarized as follows:
1. [**Performance Evaluation**] As suggested by Reviewer M2ti, we performed additional experiments to compare System 1 and System 2 performance in FaST, detailed in Appendix Sec. B. The results demonstrate that FaST achieves significant improvements on System 2 tasks while maintaining competitive performance on System 1 tasks.
2. [**Dataset Construction and Analysis**] In response to Reviewer GSxm and uXT1, we added a detailed description of the dataset construction process in Appendix Sec. A, including methodologies for dataset synthesis and augmentation. Additionally, we confirmed plans to release all datasets upon publication.
3. [**Failure Case Analysis**] Addressing Reviewer GSxm and KNeJ, we included a comprehensive analysis of failure cases in Appendix Figure 9, identifying key scenarios where the reasoning pipeline underperforms. This provides insight into limitations and areas for improvement.
4. [**Clarity and Transparency**]
Per Reviewer M2ti’s feedback, we clarified details regarding the fast sampler mechanism, which is elaborated in Appendix Sec. A. We also revised descriptions of System 2 accuracy to avoid ambiguities and ensured consistency with Figure 5. Transparency differences between System 1 and System 2 were further emphasized in the revised Introduction.
5. [**Switch Adapter Performance**]
As suggested by Reviewer uXT1, we evaluated the efficiency and robustness of the switch adapter. Results in Appendix Sec. B show minimal computational overhead, validating its lightweight and effective design.
6. [**Extended Experiments**] Following Reviewer KNeJ’s suggestion, we conducted experiments using the Bongard problem to evaluate FaST’s generalization capabilities for abstract reasoning tasks. These results are included in Appendix Sec. C and demonstrate incremental improvements over baseline models.

All modifications have been marked in blue in our revised submission.

Sincerely yours,

Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presented a new reasoning mechanism which can adaptively switch between "system 1" and "system 2" for multimodal LLMs. The authors proposed a top-down reasoning pipeline for visual agent which starts from the global view of the visual input and gradually drill down to the detailed visual observatrions to derive the final outputs, either for visual question answering or reasoning segmentation. To make the visual agent adaptively, the authors proposed a switch module and train it with carefully curated data to enable automatic. The experimental results demonstrate that the proposed system-1/2 visual agent achieve the best of both world for both simple tasks that do not require long-run reasoning and complicated tasks that necessate the reasoning process. Some further visualizations can clearly demonstrate the reasoning steps which enhances the interpretability substantially.

The main contribution of this work is the proposed fast and slow thinking mechanism for the multimodal LLMs. The paper is well-written and movtivated. The method is also appeadling and systematic per the comments by reviewers. Overall, the ACs think this work is a great way of enhancing the explicit reasoning capability for multimodal LLMs, which mirrors the current tremendous efforts to improve the reasoning capability in language domains for coding and mathematical tasks.

After the engaging discussion period, most of the concerns raised by the reviewers were addressed by the authors. From the ACs' perspective, the point made by reviewer GSxm that "The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks." makes much sense as image QA and segmentation tasks are mostly perception tasks without reliance on complicated reasoning as also pointed out by the authors and other reviewers. To really make the vision model as an agent as claimed by "Visual Agents", it is supposed to come with more  capable system-1/2 thinking for more agentic tasks. 

Overall, three reviewers gave positive ratings on this submission, and one reviewer gave 5 but with most of the concerns addressed according to the reply. The ACs also agree that this is a nice work on a systematical way of enabling system-1/2 thinking for multimodal LLMs, despite it should not count as a visual agent per se. As a result, the ACs recommend a clear acceptance of this work.