Visual Agents as Fast and Slow Thinkers
Authors: Guangyan Sun
Mingyu Jin
Zhenting Wang
Cheng-Long Wang
Siqi Ma
Qifan Wang
Tong Geng
Ying Nian Wu
Yongfeng Zhang
Dongfang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
System2 Thinking
Language Agent
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Achieving human-level intelligence requires refining cognitive distinctions between \textit{System 1} and \textit{System 2} thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce \textbf{\textsc{FaST}}, which incorporates the \textbf{Fa}st and \textbf{S}low \textbf{T}hinking mechanism into visual agents. \textsc{FaST} employs a switch adapter to dynamically select between \textit{System 1/2} modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a \textit{flexible system}, \textit{hierarchical reasoning} capabilities, and a \textit{transparent decision-making} pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that \textsc{FaST} outperforms various well-known baselines, achieving 80.8\% accuracy over $VQA^{v2}$ for visual question answering and 48.7\% $GIoU$ score over ReasonSeg for reasoning segmentation, demonstrate \textsc{FaST}'s superior performance. Extensive testing validates the efficacy and robustness of \textsc{FaST}'s core components, showcasing its potential to advance the development of cognitive visual agents in AI systems.

Published as a conference paper at ICLR 2025
VISUAL AGENTS AS FAST AND SLOW THINKERS
Guangyan Sun Mingyu Jin ; Zhenting Wang", Cheng-Long Wang*, Siqi Ma², Qifan Wang', Tong Geng Ying Nian Wu Yongfeng Zhang" Dongfang Liu
Rochester Institute of Technology University of Rochester & Rutgers University University of California, Los Angeles Meta AI KAUST Westlake University
ABSTRACT
Achieving human-level intelligence requires refining cognitive distinctions be- tween System and System thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cogni- tion. Transitioning from structured benchmarks real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident re- sponses. To address the challenge, we introduce FAST, which incorporates the Fast and Slow Thinking mechanism into visual agents. FAST employs switch adapter to dynamically select between System 1/2 modes, tailoring the problem- solving approach to different task complexity. objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a flexible system, hierarchical reasoning capabil- ities, and a transparent decision-making pipeline, all of which contribute to its ability emulate human-like cognitive processes in visual intelligence. Empir- ical results demonstrate that FAST outperforms various well-known baselines, achieving 80.8% accuracy over V VQA" for visual question answering and 48.7% GToU score over ReasonSeg for reasoning segmentation, demonstrate FAST's superior performance. Extensive testing validates the efficacy and robustness of FAST's core components, showcasing its potential to ad advance the development of cognitive visual agents Al systems. The code available at this link.
tackles uncertain and unseen
INTRODUCTION
In the field of artificial intelligence, System de- lineates cognitive mode distinguished by delib- crate, analytical, and consciously reasoned pro- cesses (Wei et al., 2022; Wang et al., 2023d; Ze- likman et al., 2022: Zhou et al., 2023; Hua & Zhang, 2022). This mode is juxtaposed to System 1, which embodies intuitive, automatic, and uncon- scious cognition. Achieving human-level intelli- gence in AI systems necessitates the deliberate cul- tivation and refinement of these cognitive distinc- tions. This process is crucial for the development of advanced reasoning and decision-mait making capa- bilities (Zhang et al., 2023d;e; Hao et al., 2023).
Q1: Which holiday do you Think they are celebrating?
Q2: Is the telephone on the left right side the hand lamp?
Fest
Slow
Ansi Maybe it's Christmasl
Ans: The telophone the right.
Chain Evidence Construction
The emergence of foundation models marks a sig- nificant turning point, where Large Language Mod- els (LLMs) based agents have made remarkable strides in many areas, showcasing human-like in- telligence across diverse tasks (Brown et al., 2020; Kojima al., 2022; Ge al., 2023). However, this achievement is primarily attributed to some fea- tures of foundation models: overparameterization and the availability of vast, general-purpose
[Clue] Telephone Booely on the toble.
<Lorgo
<Telephone>
Figure 1. Working Pipeline. FXST represents solution rooted in system switching, demonstrating pronounced capabilities hierarchical reasoning and ad-hoc explainability.
Equal contribution and shared co-first authorship.
Corresponding author.

---

Published as a conference paper at ICLR 2025
datasets (Kaplan et al., 2020; OpenAI, 2024). It is imperative to note that while these models hibit human- like traits (e.g., inductive and deductive reasoning (Huang & Chang, 2023b; Dasgupta et al., 2022; Jin et al., 2024b)), these characteristics do not equate to the processes of System 1/2 thinking (Nye et al., 2021; Yao et al., 2023b) and are far less intelligent than human thinking.
In practice, visual agents often encounter challenges when moving from controlled, structured benchmarks to complex, real-world environments (Wu & Xie, 2024; Ge et al., 2023). This prob- lematic circumstance will result in spurious reasoning pathways, akin hallucinations, where they struggle to acknowledge their limitations or uncertainties (Gunjal et al., 2024; Chen et al., 2023c). Such an issue arises from the absence of explicit modeling of the fast and slow cognitive processes, reminiscent of human System 1 and System 2 thinking (Yao et al., 2023b; Kahneman, 201 I). Con- sequently, when faced with intricate inquiries, the Multimodal Large Language Model (MILM) frequently offers overly confident yet inaccurate responses (Wu & Xie, 2024; Chen et al., 2024b; Tong et al., 2024a). Addressing this problem entails reassessing MILM algorithms to incorporate insights from the interplay between fast and slow thinking (System I and System2) observed in hu- man cognition. Our design philosophy guides us to incorporate human qualities into our work.
In this study, we introduce the Fast and Slow Thinking (FAST) mechanism into visual agents. More concretely, we design a switch adapter to determine whether the encountered problems are best ad- dressed using which thinking mode. Simple tasks require only fast thinking (System 1) for straight- forward problem-solving pipeline, while complex tasks necessitate the slow, deliberate processing of System 2 (see Fig. 1). Specifically, System is triggered when encounter visual challenges that have: ① Uncertainty: When the model has low confidence in directly identifying the object to which the complex query referring. For example, the query asks "the appliance for storing and cooling food" instead of "refrigerator," and @ Invisibility: When dealing with minuscule-sized objects that evade detection by standard visual encoders, where normal visual agents cannot tell what it is. This switch adapter is achieved by designing negative contextual data to re-adjust the model's 's confidence and ignite world knowledge (as detailed analysis in $3.2). Subsequently, a proposal adapter is en- gaged to outline regions that are related to the questions. This allows visual agents to leverage the newly acquired data, thereby facilitating a more detailed and precise response. Further, if the inquiry necessitates detailed insights into particular instances, seg adapter provides segmentation masks, offering additional contextual information for deeper analysis (as detailed analysis in $3.3).
FAST enjoys few attractive qualities. Flexible system: Building foundation that explicitly models System 1/2 thinking, our proposed method adeptly handles complex visual tasks, demon- strating competitive performance in streamlined pipeline (see 12. FAST's core epistemology combines an intuitive mechanism for straightforward cases with deliberate analytics for more intri- cate scenarios, therehy enhancing the development of human-like visual agent. @ Hierarchical reasoning: FAST perceives visual tasks with a top-down granularity, encompassing image-level cues, box-level candidates, and pixel-level targets (see Fig. 2). This progressive approach facili- tates a sensible understanding of visual content, starting from global concepts, progressing through region-speciff candidate assessment, and culminating in precise target identification. Each stage in- volves developing concrete ideas and establishing coherent "chuin of evidence" to support the final inference. @ Transparent pipeline: FAST's ppcision-makkng process embodies a neuro-symbolli essence in System 2 mode, yielding intermediate step outputs as interpretable symbols (e.g., bound- ing boxes or masks), facilitating direct visual inspection by humans. This inherent reasoning mech- anism enables ad-hoc explainability of the model's behavior (see Fig. 3), distinguishing FAST from prior approaches (Liu et al., 2024a) that lack precise explication of their operational mechanisms.
We conducted series of experiments to validate the efficacy of our proposed method. In $3.1.1, we apply FAST to visual question answering and multimodal benchmarks. FAST demonstrates significantly improved performance over baselines such LLaVA-v (Liu et al., 2024a), achiev- ing performance gains on benchmarks like TexiVQA (Singh et al., 2019) with a 2.5% increase in accuracy and total score improvement of 6.7 on MME (Fu et al., 2024). In 13.1.2, we explore the versatility of our approach through its application to tasks such as referring and reasoning seg- mentation, with performance gains including an increase of 4.1% CloU with LLaVA- and improvements of 3.2% CToU and 2.7% GIoU on the ReasonSeg dataset over LISA-7B (Lai et al., 2024). The robustness and effectiveness of the core components of our FAST framework are further substantiated through series of ablation studies, as elaborated in $3.3.
2

---

Published as a conference paper at ICLR 2025
2 METHODS
Notation. The integration of components in visual agents F (based on the Large Language Model) typically involves visual encoder, denoted as Ey, nature language encoder, represented by &L. and a Language Language Model such as Vicuna (Chiang et al., 2023). Initially, the visual agent is presented with an image I and accompanying textual prompt Q. which could be question or instruction. Then the visual agent combines these multimodal tokens into a united space. Finally, the visual agent outputs textual response R given the textual and image input. The generation process can be expressed as Equation 1:
R F(J))v()),
Definition 1 (System and System 2) System I and 2 are two different systems of thinking proposed by Nobel Laureate Daniel Kahneman in his book Thinking. Fast and Slow (Kahneman, 2011).
System (Fast Thinking): Unconscious. automated thinking processes, fast, intuitive. effortless responsible for automatic responses and basic cognitive operations in daily activities, vulnerable to heuristic biases and errors, 6.g., recognizing familiar faces, and knowing the location of objects.
System (Slow Thinking): Conscious, energetic thinking processes, slow, effortful, logical, and analytical, responsible for complex calculations, reasoning, and decision-makingg can monitor and control System 1 processes, e-llfilling out a lax form, finding the position of a word in a sentence.
2.1 FAST
We present FAST (see Fig. 2), a novel framework designed efficiently handle both simple and complex visual queries. FAST features a dynamic system switch mechanism that enables rapid re- sponses straightforward questions (System I) and accommodates deliberate reasoning for intricate scenarios (System 2). During slow thinking. the system uses contextual clues to identify relevant region, facilitated by a proposal adapter. The adapter generates bounding box around the target object, and if needed, pixel-level mask adapter refines the proposal for further details. Finally, we summarize the gathered information from the whole system to provide a comprehensive answer.
System Switch. Current works on visual agents mostly rely on visual question- guestion-answering answering data, which gives direct answers (System 7) after inquiry as Eq. 1. However, attempting to answer ques- tions directly in this way can compromise the reliability of the responses. Agents tend to hallucinate over questions that require more deliberate reasoning and visual details. To reduce hallucination and make the model reliable, information. Specifically, for a question Q and an image I, we define a MILLM with switch adapter S and formulate the fast Faast and slow thinking process Fstow- When the query is easy, the frame does not need the switch adapter Sodapter and only output result R by Fjast as Equation 2.
utilize system switch trigger to tell when to require more visual
R= F (
Remark 2.1 Swritching-rrennde dataset) A Negative Data for Target Objects Reasoning Dataset D of 100,000 (image, question, answer) triples was constructed to facilitate the identification of target regions or objects required to answer a question. The dataset constructs questions about the absence or details of certain objects, deliberately made too small be perceived by the visual encoder. Section A for more details.
Remark 2.2 (Switch Adapter) A liiht- adapter that is fine-tuned with both positive fast- thinking data and negative data (Remark. 2.1) acquire system switching capability. When the adapter encounters harder questions, the switch mechanism will be triggered for later slow think- ing.
Note that a slow thinking process is not always activated. The system switch adapter as Remark. 2.2 Sadapter will determine whether the question for the particular image is sufficient to give direct answer. If so, the fast mode Fjost will give a quick and direct response as Equation 2. If there is any missing information about the question that current agent cannot solve, the switch adapter will be activated and find the pattern to elicit all the possible missing objects O Omissing related to question and context clues Catue which the possible location of the missing objects as Equation 3.
O Omissings Cotve Sadapter (
3

---

Published as a conference paper at ICLR 2025
Image
Query
Slow Answer
Switch Adapter
Is the umbrella on the left or right side the traffic light?
The umbrella is on the right side of the traffic light.
Vision-Language Model
Chain of Evidence (Remark 3.5)
Seg Instances
Switch Adapter (Remark 3.2)
Proposal Adapter (Remark 3.3)
Slow Thinking Template
Object Missing: Sorry, the current visual information not enough. need to find the [umbrella, traffic light] first.
Context Clue
Traffic Light
Umbrella
Context Clue: [umbrella held by people traffic light beside the street]
Seg Adapter (Remark 3.4)
Slow Thinking Mode of FAST. Adapter, which selectively activates a slow and analytical thinking mode when encountering complex visual queries, supplementing with extensive world knowledge to provide missing objects and contextual clues; Pro- posol Adapter, which identifies and emphasizes regions of interest within the visual inputs; Seg Adapter, which delivers precise pixel-level segmentation, enhancing depth of the visual analysis. The outputs from each mod- ule are integrated into chain of evidence (see Fig. 3), providing methodical and accurate response. FAST represents a neural-symbolic approach that combines the strengths of symbolic reasoning, ensuring that our system is effective and interpretable.
Figure 2.
thinking mode comprises core modules: Switch
Specifically, we use negative data that contain missing objects mieeeing and context clues Colue for training the system switch adapter for triggering the slow mode Remark. 2.2. The slow mode should deal with question-image pairs that are 1) uncertain in pinpointing the specific object in question, and 2) small perceive for the standard visual encoder. So we utilize triplet data in dataset Remark. 2.1 (image, question, answer) as where the question requires objects that are not in the image or too small to be perceived by the visual encoder. The threshold is require the model to tell that certain objects are missing instead of a direct answer, and we utilize the world knowledge list all the objects and also the context information for later deliberate reasoning.
be 20 20. We
2.2 PRELIMINARY
Hierarchical Reasoning, We use a top-down scheme to reason over multi-scale granularity images effectively in order reason and take advantage of world knowledge progressively. Similar to humans would look for some context clue to find specific objects relating to questions and zoom in if they think the answer lies in particular region, we model this process with system switch adapters as Remark. 2.2 to focus on the context clue Cotue generated from the switch adapter as Equation 3.
proposal adapter Padapter (visual agent). In System FaST many visual agents to accomplish hicrarchical reasoning. The frame tries to narrow down the search space by using the question and the previously obtained clue Cotue let the proposal adapter output region Region that aligns with the question and the context clue Equation 4.
We denote the MILM
Region = P Padapter (E), Ey EQ)), Catuel
(4)
After getting the region, the visual agents Pedepter will be asked focus on more specific target with bounding box [Bbores] complemented by the context clue Cetue and region Region get from Equation 4. This process can reveal the step-by-step reasoning and be modeled as Equation 5.
[Bbozes] P Padapter (Q), Region,
(5)
Remark 2.3 (Proposal Adapter) A lightweight adapter that is fine-tuned with proposal data to ac- quire the capability of finding the corresponding region given the context clue object name.
4

---

Published as a conference paper at ICLR 2025
Remark 2.4 (Pixel-level mask decoder) The Pixel-level mask decoder is the decoder of segment ( (Kirillov et al., 2023)). The pixel-level mask decoder is fine-tumed to produce target masks based on the hidden embeddings.
When we have more specific target proposalbbounding box [Bbozes)) FAST will apply a fine- grained pixel-level mask decoder Pseg Equation 6 to output the specific mask part [Mask] of the target proposal [Bbozes] to focus on as Equation 6. We name this whole process from Region to [Mask] chain of evidence as Remark. 2.5 similar to thinking more and more deeply by humans.
[Mask] Psg [Ex(Q) (Bbozes), O missingl
Remark 2.5 (Chain of Evidence) Chain of evidence is like the chain of thought in a Large Language Model. But define it as a deeper and deeper step of thinking based correct evidence in our frame FAST. The completion of the chain of evidence needs many visual agents to work together.
After getting the target proposal (bounding box [Bbozes)) from context clue Caiue with proposal adapter and specific mask part [Mask] by missing objects with seg adapter, a chain of evidence is con- structed as Remark 2.5 and Fig. 3. Our FAST frame- work then summarizes all this information (Z and Q) and the chain of evidence with switch adapter give the final correct reasoning answer Ans as Equation 7:
Chain of Evidence
CLIP
Image
Vision Encoder
Context Clue
Ans = Fstow (Q), Ev((), [Bbozes MaaaassMasshl Mask]] (7) The decision decision-making process in FAST is distinguished by its neuro-symbolic nature, which generates in- termediate outputs easily interpretable symbols, including region-of-interee (Rol) driven boxes and object-driven masks. This capability allows humans to perform direct visual inspections, thereby augmenting the transparency of the model's 's operations. Moreover, the intrinsic reasoning mechanism of FAST enhances the ad-hoe explainability of its behavior, see Fig.
Seg Instance
Query
Large Language Model
Slow Answer
Switch Adapter
Figure 3. Chain of Evidence. FAST represents solution rooted in switching. demonstrating pronounced capabilities in hierarchical reason- ing and ad-hoc explainability.
2.3 IMPLEMENTATION DETAILS
The framework of FAST(as Fig. 2)'s implementation details are shown in this section below.
Visual Agents. We choose the architecture and configuration LLa (Liu al., 2024a) as our visual agent. The most important component in visual agent is the visual encoder &y (f): A CLIP- ViT-L-336px model (Radford et al., 2021) used, where input images are resized or padded to 336 336 pixels, learning to associate visual features with corresponding textual descriptions. MLP projection with channels of [256, 4096, 4096] used for connecting image representations into the word embedding space.
Mask Decoder. The mask decoder Pseg architecture is identical to SAM. Besides, is fully fine-tuned with a collection of semantic segmentation (Caesar et al., 2018; Zhou et al., 2017; Ra- manathan et al., 2023; He ct al., 2022; Chen et al., 2014) and referring segmentation (Mao et al., 2016; Kazemzadeh et al., 2014) datasets to efficiently map the <seg>>ooken representations mask if the FAST need to segment.
Chain of Evidence. When apply the chain of evidence Remark 2.5 in the LLM to get the answer as the final step like Equation 7. The whole sequence the chain of evidence is too long load in the Fstow- So FAST needs visual sampler based cross-attention that trained to decrease the number of image tokens to a suitable length (from 256 to 32), apart from MLP projection.
3 EXPERIMENT
We utilize eight popular benchmarks to evaluate our framework FAST comprehensively. catego- rized into general visual question answering (VQA) datasets and multimodal benchmarks. The
5

---

Published as a conference paper at ICLR 2025
VQA benchmarks include VQA-v2 (Goyal al., 2017), GQA (Hudson & Manning, 2019), Sci- enceQA (Lu et al., 2022), and TextVQA (Singh et al., 2019) which focus on optical character recog- nition. For multimodal benchmarks evaluation,
use the hallucination benchmark POPE (Li et al., 2023c), along with comprehensive benchmarks such as MME (Fu et al., 2024), MM-Vet (Yu et al., 2024), and SEED (Li et al., 2024). We compare our model with the baseline VL- (Liu et al., 2023a), and other multimodal large language models. To thoroughly assess our model's understand- ing of pixel-level instances, we evaluate its performance on referring segmentation and grounding benchmarks, including refCOCO (Kazemzadeh et al., 2014), refCOCO+ (Kazemzadeh et al., 2014), and refCOCOg (Caesar et al., 2018). Further, examine the model's reasoning capabilities on FAST framework, we consider the Reasoning Segmentation benchmark (Lai et al., 2024).
3.1
MAIN RESULTS
3.1.1 EXPERIMENTS ON VQA AND MULTIMODAL BENCIMARKS
Training, In developing the Switch Adapter, we employed the VLa (Liu et al., 2024a) framework, conforming strictly to its established training protocols. We incorporated negative sam- ples from V* (Wu & Xie, 2024) with contextual cues to enhance system switching capability to amplify multimodal inferential and world knowledge. This augmented dataset was combined with LLaVA-v1.5's supervised dataset and trained for one epoch. For the Proposal Adapter, aug- mented the LLaVA-v1.5 dataset with region-specific bounding boxes based on contextual cues and queries, then fine-tuned for one epoch to optimize proposal generation. The Segmentation Adapter utilized the LISA (Lai et al., 2024) architecture integrated with the LLaVA-v1.5, 1.5, employing SAM as the mask decoder. The adapter was fine-tuned using the same datasets as Lisa, including se- mantic segmentation, referring segmentation, and reasoning segmentation. This fine-tuning process involved 10,000 steps to improve the model's s segmentation capabilities. Throughout developing the Switch Adapter. Proposal Adapter, and Segmentation Adapter, we employed the LoRA (Low-Rank Adaptation) technique (Hu et al., 2022). By leveraging LoRA, we introduce minimal additional parameters while preserving the original multimodal large language model's architecture and effi- ciency. All experiments used NVIDIA TESLA A100-80GB GPUs.
Metric. In model evaluation across diverse datasets, various performance metrics are utilized.
Accurucy. The primary evaluation metric utilized in the VQ4 GQA, TexIVQA ScienceQA, and SEED benchmarks is accuracy. Accuracy is a performance measure that quantifies the exact match percentage between predicted and acceptable ground truth answers, indicating a model's precision.
Fl Score. The POPE dataset uses the F1 Score to balance precision and recall, providing a compre- hensive assessment by harmonizing the trade-off between positive prediction accuracy and recall.
Total Score. The MME evaluation metrics include accuracy (based on individual questions) and ac- curacy+ (considering both questions per image), reflecting a stricter and more comprehensive model understanding. Random accuracies for these metrics are 50% and 25 respectively. Perception scores, calculated as the sum of these metrics across subtasks, total 2000 for perception.
GPT-Evvluation. In the MM-Vet dataset, perfor- mance is evaluated by GPT-4 through a compar- ative analysis of predicted and ground truth an- swers, generating score to quantify alignment.
Model Perfo Across Detasets
CaV.M CopCoM
17.00.1
PLST
50
21.225
Results. As depicted in Table 1, FAST demon- strates superior performance across multiple VQA datasets and multimodal benchmarks when com- pared to established methods. To ensure fair- ness in comparison, all methods in Table share the same visual encoder: basic CLIP (Radford et al., 2021). Remarkably, FAST consistently sur- passes the LLaVA-v1.5 model, achieving signifi- cant improvements in performance across all eval- uated datasets. Specifically, in VQA datasets, our model outperforms LLaVA-v1.5 by 2.3% in VQ4"2, 1.8% in GQA, and 2.5% in VQAT. Ad-
65.2
Palasct
Figure 4. The Comparison with CoVLM and CogCoM. These models use the more powerful vi- son encoder.

---

Published as a conference paper at ICLR 2025
VQA Datasets
Multimodal Benchmarks
Method
L.M
VQA42
GQA
VQAT
SQ4'
POPE
MME
SEED
MM Vet
BLIP-2 CLIP-22cMan 10MLL3
Vicuna-I 13B
65.0
32.3
42.5
61.0
85.3
1293.8
46.4
22.4
InstructBLIP Need1P524)
Vicuna-138
49.5
50.7
63.1
78.9
1212.8
53.4
25.6
Qwen- (L-Chataxxi23)
Qwen-7B
78.2
57.5
61.5
68.2
1487.5
mPLUG-Ow12 BLLUUG-Ow2:eee (cv99224
LLuMA-7B
79.4
56.1
58.2
68.7
1450.2
61.6
36.2
Monkey (CNPR24)
(CV9R2)
Qwen-7B
80.3
60.7
69.4
67.6
LLaVA-vI 1,5 (CNPRR24)
Vicuna-7B
78.5
62.0
58.2
66.8
85.9
1510.7
58.6
30,5
Chain of Spot (x00v244)
Vicuna-7B 7
80.7
63.7
60.9
68.2
86.4
1501.1
59.7
30.8
V (cV0R21)
Vicuna-7B
82,4
1128.9
41.7
27.7
Visual CoTorki
Vicuna-7B
63.1
77.5
FAST (Ours)
Vicuna-7B
80.8
63.88
60.7
68.9
86.4
1517.4
60.1
31.0
A (vs LLaVA-v1.55
Vicuna-7B
+2.3
+1.8
2.5
+2.1
+0.4
+6.7
0.5
Table 1. Main results on eight VQA and multimodal benchmarks. Our FAST consistently outperforms the baseline LLaVA1.5 model across all evaluated benchmarks, denoted with line A.
ditionally, FAST excels multimodal benchmarks, with notable increases of 6.7 in the MME score, 1.5 in the SEED score, and 0.5 in the Me-Vet score, highlighting its versatility and effectiveness in handling a broad range of domains. These results underscore the robustness of PAST, particularly in tackling complex visual and textual tasks. Moreover, Fig. 4 showcases direct comparison between FAST, CoVLM (Wang et al., 2023b), and CogCoM (Qi et al., 2024), both of which employ the more powerful EVA2-CLIP-E (Sun al., 2023) model as their visual encoder. As expected, these models exhibit stronger performance due to their enhanced encoder. To align with this, we replaced our original vision encoder with EVA2-CIIP- which resulted in further improved performance, ensuring a more rigorous and fair comparison with state-of-the-an methods.
3.1.2 EXPERIMENTS ON REFERRING AND REASONING SEGMENTATION
Training. The training settings for the Switch Adapter and Proposal Adapter remain consistent with those previously described as 1.1 During the training phase of the Segmentation Adapter, certain specific datasets are intentionally omitted to uphold an unbiased evaluation of referring and reason- ing segmentation datasets. This strategic exclusion a crucial measure implemented prevent any potential data leakage, thereby ensuring the integrity and reliability of the evaluation results.
Metric. Following prior research on segmentation (Kazemzadeh al., 2014; Mao et al., 2016), two evaluation metrics are employed: Generalized Intersection over Union (GIol) and complete Intersection over Union (Clol)).
CloU. The CloU is calculated based on the cumulative intersection over the cumulative union across all images in the dataset. This approach can introduce a significant bias towards larger objects images with more objects, as they contribute more to the cumulative union area.
GloU. The GIoU is computed as the average per image IoU, where the IoU is calculated for each image, and then the average is taken across all images in the dataset. This metrie provides balanced assessment by treating all images equally, regardless of their size or the number of objects.
Results. Table illustrates the performance of FAST compared to recent visual agents like LISA on refer- ring and reasoning segmen- tation benchmarks. notably outperforms LISA-7B on the refCOCO+ and ref- COCOg benchmarks by 2.0% and 0.6% CloU, respectively. For the more complex reason- ing segmentation task, FAST shows even stronger results, with 3.2% CloU gain and
Referring Segmentation
Reasoning Segmentation
Method
reCCCC
ReCOCC-
reCOCCO CloU
ReasoSog CloU
CloU
CloU
GloU
72.7
62.1
61.2
OVSegiemzzy
28.5
18.6
GRES
73.8
66.0
65.0
22,4
19.9
FAST
X-Deeodennveen
22.6
SEEM
65.7
25,5
21.2
LISA-7Biewees
74.1
62.4
66.4
44.4
46.0
LLaVA Seg Adapter
70.8
57.5
64.0
43.0
41.0
FAST (Ours) 73.3
64.4
67.0
47.6
48.7
Table 2.
Main results on referring and reasoning segmentation bench- marks. Our FAST exhibits competitive results in referring segmentation tasks like refCOCOg+ while showcasing superior performance in reason- ing segmentation, particularly when evaluated against LISA-7B.

---

Published as a conference paper at ICLR 2025
2.7% GloU improvement over LISA. The results highlight FAST's superior performance and its bustness in handling both straightforward and complex visual reasoning segmentation benchmarks.
3.2 ANALYSIS OF SYSTEM SWITCHING ADAPTER
GQA
MME
Our study investigates the efficacy of the switch adapter mechanism in balancing accuracy and com- putational efficiency. As depicted in Fig.5, our
52 32.2% ACC System
70.2% ACC
56 56.8% ACC System Answer 59% %
64.7% ACC System
analysis illustrates the system's adeptness in dis- cerning between the System and System cog- nitive modes triggered by query complexity. For queries requiring System 2 reasoning, the adapter dynamically combines System / reasoning for sim- pler subcomponents with System reasoning for more complex aspects. Consequently, the reported accuracy rates under System 2 mode (52.2% for MME and 56.8% for GQA) reflect combination of reasoning outcomes, emphasizing the adapter's s ability to differentiate query complexities and optimize task performance accordingly. This highlights the importance of maintaining System reasoning for prompt and confident responses while effectively utilizing System 2 reasoning for complex problem-solving.
%
System Answer 64%
Answer 41%
Figure 5. System Mode Analysis. We investigate the system switching ratio, along with fast thinking performance on easy or hard queries defined by the switch adapter.
Table compares runtime across system con- figurations. System 1 Only, using switch adapter, operates efficiently with one-time in- ference, while System 2 Only, which constructs chain of evidence for every query, is sig- nificantly more resource-intensive. In con- trast, FAST balances efficiency and perfor- mance, running 31% % faster than System 2 Only on MME and 50% faster on GQA, with comparable results. This highlights FAST's ability to opti-
MME
GQA
Method
Runtime
Result
Runtime
Result
System Only
734ms
1508.7
737ms
61.9
System Only
2938ms
1518.6
2937ms
64.0
OURS
2023ms
1517.4
1475ms
63.8
Table 3. Runtime Analysis and Comparison on only Syxtem 1 (fast), our FAST and only System (slow).
mize cognitive task processing while conserving computational resources.
3.3 ABLATION STUDY
Algorithm Component
GQA
POPE
MME
Output Component
MME
refCOCOg
BASELINE
62.1
85.7
1509.2
BASELINE*
1511.8
66.0
+ Proposal Adapter
63.2
86.0
1516.5
+ Missing Objects
1513.4
66.8
+ Seg Adpater
62.8
85.8
1514.4
+
Context Clue
1516.6
66.4
OURS (both)
63.8
86.2
1517.4
OURS (both)
1517.4
67.0
Table Key Component Analysis
Table Switch Adapter Output Analysis
Key Component Analysis. We undertake a detailed investigation into the core elements of our novel framework, FAST, with particular emphasis on the proposal adapter for contextual region localization and the seg adapter for pixel-level mask segmentation. To establish comparative base- line, we design model configuration that excludes both the proposal and seg adapters, instead relying solely on switch adapter to provide missing objects and context clues. This baseline model serves the foundation for evaluating the impact of the individual and combined components of the framework. As demonstrated in Table 4, the introduction of the proposal adapter, the seg adapter, or both, results progressive and substantial improvements in performance across various evaluation metrics. For instance, accuracy on the VQA"2 dataset improves from 62.1% to 63.8%, showcasing the considerable value these components add. This underscores the pivotal roles of the proposal and seg adapters in enhancing the model's 's overall capability, further affirming their importance within the FAST framework.
Further, evaluate the switch adapter's role in incorporating missing objects and context clues using a variant BASELINE*, which omits these features. Table shows that adding missing objects context clues improves metrics like MME and refCOCOg, with the best performance achieved
8

---

Published as a conference paper at ICLR 2025
when both are included. These results confirm the importance of all components in optimizing
FAST's effectiveness.
3.4
QUALITATIVE COMPARISONS OF FAST
Query
Quartyy
What are the jors sitting on top of?
Is the monitor on Top of person?
LLaVA.115
LLAVA The monitor is below the person.
Coffee
0.0
Choin of Exidance
Chain Evidence
Clue: The most likely the kinchen
Clue: The monitor is likely on the roof.
Proposol
Ans: Stove.
Proposal
What items should you find when you are hungry? Please output the seg mosk
Segment out the corioiiners that make me feel energized after drinking.
SWEDESE
Chein of Evidence LISA:78
Choin of Evidence:
LISA-78
item is likely Sure, It's sseg..
Clue: The container is
Sure, it's sseg.
likely on the desk.
Proposol
Proposal
(Sago
Figure 6. Qualitative Comparisons FAST. The top row shows the VQA results on FAST compared to LLaVA-v1.5. The bottom row presents the segmentation results compared to LISA-7B.
In 6ig.6, we present qualitative comparisons that highlight the enhancements introduced by FAST. The top row, above the dotted line, shows results from the VQA task, comparing FAST with LA- v1.5. LLaVA-v1.5 often fails to focus on key areas within the image, leading to incorrect or incom- plete responses. In contrast, FAST builds a chain of evidence by identifying key objects and elements (e.g., degectnnn a woman on the street or a monitor on the roof) and then applying object-level pixel masks to accurately determine the focus areas. This enables FAST to provide more precise and deliberate answers. The bottom row, below the dotted line, shows segmentation results, comparing FAST with LISA-7B. LISA-7B struggles with segmenting smaller objects or those requiring more complex reasoning, often causing confusion. In contrast, FAST excels at isolating relevant objects with greater accuracy and granularity, particularly with smaller or less obvious items. This demon- strates FAST's superior performance in both VQÅ and segmentation tasks, showcasing its ability to handle wide range of visual and reasoning challenges more effectively than its counterparts.
4 RELATED WORKS
LLM as Visual Agents. With the capabilities that LLMs have demonstrated in language under- standing and generation (Ouyang et al., 2022: OpenAI, 2024; Zheng et al., 2023; Touvron et al., 2023a;b; Wang et al., 2024a; Hua et al., 2024; Mei et al., 2024; Jin et al., 2024a; Chen et al., 2024c), the research community has progressed to explore how LLMs can be enhanced with vision input for multimodal tasks visual agents (Alayrac et al., 2022; Driess et al., 2023; Li et al., 2023a; Ge et al., 2023; Dai et al., 2024; Liu et al., 2023b; 2024a; Lin et al., 2024; Wang et al., 2024b). There are two paradigms for LLM-based visual agents: end-to-end based and tool-using visual agents. Fol- lowing the principle of instruction tuning, end-to-end visual agents are trained with a curated visual instruction tuning dataset to digest features from multi-modality, unlocking the capability to answer visual questions (Huang et al., 2023; Luo et al., 2023; Zhu et al., 2023a; Bai et al., 2023; Zhang al., 2023b;e; Chen al., 2024a; Ye al., 2023; Singh al., 2019). For other visual tasks (e.g.. Segmentation, Detection, etc), end-to-end trained tailored agents can further perform downstream tasks (Pi al., 2023; Peng et al., 2024: Lai et al., 2024; Chen al., 2023b; Wang al., 2023c; Dai et al., 2024; Wang et al., 2024c; 2023b; Jiang et al., 2023; Chen et al., 2023a; Zeng et al., 2024). Recent research has focused on leveraging improved vision encoders and fostering more detailed visual understanding, yielding promising results (Fan et al., 2024; Xu et al., 2024a; Shi et al., 2024). While these approaches can be implemented with direct instruction tuning data, they represent a 'System I' type of training. This type of training primarily relies on the dataset's 's quality and tends to provide direct answers that are prone to hallucinations, a consequence inherent to the nature of System I instruction tuning data. For the second paradigm, tool-using models are built top of frozen LLM with access to pretrained visual perception tools (Surís et al., 2023; Shen et al., 2023; Lu et al., 2023a). In this scenario, the LLM first selects visual tools and then decides by thoroughly

---

Published as a conference paper at ICLR 2025
analyzing the fine-graing information extracted by visual tools (Lu et al., 2023a; You et al., 2023; Wu et al., 2024). While external visual tools enhance the interpretability of the reasoning process, their complexity
introduce inaccuracies. Moreover, the abundance of information generated during reasoning may overshadow key details relevant to the query, resulting in incorrect answers.
Our research introduces novel and adaptable framework designed to enhance response accuracy by adopting distinct slow thinking cognitive modes. Unlike traditional end-to-end visual agents, our framework, FAST, systematically assesses information sufficiency, thereby mitigating the risk of overconfidence. When System 2 (slow, analytical thinking) is activated, FAST employs multiple experts to construct a coherent chain of evidence. This approach ensures the generation of accurate and interpretable responses, significantly advancing the reliabiliity and transparency of visual agents.
System 2 in AL. Recently LLMs have been engincered to produce text that mimics the step-by-step reasoning process characteristic of human cognition, akin to the analytical and deliberate thought processes associated with what is termed as System in the human cognition process (Qiao al., 2023; Huang & Chang, 2023a; Wang et al., 2023a; Shaikh et al., 2023; Shao et al., 2024). The sys- tematic approach to problem-solving is a hallmark across various domains, including mathematical word problems (Kojima et al., 2022; Wang et al., 2023d; Lightman et al., 2023; Cobbe et al., 2021; Liu et al., 2023c; Zhu et al., 2023b; Lu et al., 2023b). logical reasoning (Yao et al., 2023d;a; Besta et al., 2024; Wen et al., 2023; Lei et al., 2023; Cheng et al., 2024; Jin et al., 2024b), and multi- reasoning (Chen et al., 2024e; You et al., 2023; Wu & Xie, 2024). In Explainable AI, this system- atic method is emulated by the model as it generates text-based elucidation of its reasoning and decision-making process through step by step reasoning process (e.g.. chain of thought) (Han et al., 2024; Zhao al., 2024; Jacovi & Goldberg, 2020; Hua & Zhang, 2022). However, it is crucial to recognize that LLMs, while powerful, are not exempt from encountering challenges when facing complex problems. One such challenge is the issue of hallucination (Zhou et al., 2024; Cui et al,, 2023; Li et al., 2023b; Zhang et al., 2023a; Chen et al., 2024d; Guan et al., 2024a), which can dis- tort the model's reasoning process and lead to inaccuracies the explanations provided. Initially, LLM reasoning is seen as only linear chain of thoughts, where each step in the reasoning pro- cess is clearly articulated. As models evolve, they adopt more complex structures like hicrarchical trees (Geng et al., 2023; Yao et al., 2023b) and intricate graphs (Besta et al., 2024), which enable them to handle much more complex problems but also restrict their general applicability because of increased topological complexity (Yao et al., 2023b;a; Besta et al., 2024; Lei et al., 2023; Wen et al., 2023). Moreover, these complex structures can lead to errors propagating through the model's s reasoning, causing a cascade of mistakes (Xu al., 2024b). To counter this, incorporating feedback from intermediate reasoning steps and employing iterative refinement, which is similar to human re- flection, could help mitigate errors (Chu et al., 2023; Tong et al., 2024b; Guan et al., 2024b; Madaan et al., 2023; Yuan et al., 2024; Wu & Xie, 2024). In unsupervised scenarios, such feedback is vital for enhancing the reasoning capabilities of LLMS and reducing errors (Yao et al., 2023c).
Our key contribution is the introduction of the chain of evidence within multimodal reasoning frame- works. This methodology enriches each reasoning step with accurate, image-based cascading in- formation, effectively mirroring human visual and System 2 cognitive processes. Our approach enhances accuracy and significantly improves interpretability and generalization capabilities.
5 CONCLUSION
In this study, we introduced FAST, a framework that combines System I (which is fast and intu- itive) and System 2 (which is slow and deliberate) thinking to improve visual agents' reasoning and decision-making. FAST adapts to queries of varying complexity with a flexible system switch, deliv- ering quick responses for simple tasks and using hierarchical reasoning for more complex scenarios. The FAST leverages neuro-symbolic decision-making transparent pipeline delivering interpretable intermediate outputs that enable explainability. Our results show significant improvements across benchmarks, demonstrating the effectiveness of FAST's chain of evidence in reducing hallucina- tions and improving interpretability, Furthermore, ablation studies highlight the critical importance of contextual clues, symbolic reasoning, and pixel-level adapters in refining visual reasoning and understanding, marking step forward in creating more reliable and accurate AI cognition.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This research was supported by the National Science Foundation under Grant No.2242243. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of U.S. Naval Research Laboratory (NRL) the U.S. Govemment. The U.S. Govermment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
ETHICAL SAFEGUARDS
In our paper introducing a novel framework FAST, we implement rigorous ethical measures to prevent potential misuse and promote responsible application. These measures are delineated in comprehensive protocols accompanying the final release of models and datasets. Our protocols en- compass stringent usage guidelines, access controls, incorporation of safety filters, and monitoring systems. These concerted efforts reflect our steadfast dedication to upholding the utmost ethical standards scientific exploration. Our objective to protect the rights and privacy of all stakehold- involved, thereby fostering culture of responsible and ethical research within our community.
REPRODUCIBILITY
Our FAST framework is implemented in PyTorch (Paszke et al., 2019). All the experiments are conducted on eight NVIDIA A100-80GB GPUs. Our full implementation shall be publicly released upon paper acceptance to guarantee reproducibility. The codes are available at the anonymous link hwwp://www..ooo....cce////////// V for the review process.
All Experiments are conducted on eight NVIDIA A100-80GB SXM GPUs!. Reproducing the fine- tuning process would require approximately 15 A A100 GPU days.
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A Visual Lan- guage Model for Few-Shot Learning. In NeurIPS, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- VL: A Frontier Large Vision- LanguuLe Model with Versatile Abilities. arXiv preprint arXiv:2308. 12966, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. In AAAI, 2024.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language Models Few-Shot Learners. In NeurlPS, 2020.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and Stuff Classes in Con- text. In CVPR, 2018.
Hila Chefer, Shir Gur, and Lior Wolf. Generie Attention-Model Explainability for Interpreting Bi- Modal and Encoder-Decoder Transformers. In ICCV, 2021.
Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual Instruction Tuning with Polite Flamingo. In AAAI, 2024a.
Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Ziyong Feng, Yongle Zhao, and Yin Xie. Plug- and-Play Grounding of Reasoning in Multimodal Large Language Models. arXiv preprint arXiv:2403.11322,, 2024b.

---

Published as a conference paper at ICLR 2025
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-V2: Large Language Model Unified Interface for Vision-Language Multi-Task Learning. arXiv preprint arXiv:2310.09475. 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM's s Referential Dialogue Magic. arXiv preprint arXiv:2306. 15795, 2023b.
Mingkai Chen, Taowen Wang, Shihui Cao, James Chenhao Liang, Chuan Liu, Chunshu Wu, Qifan Wang, Ying Nian Wu, Michael Huang. Chuang Ren, Ang Li, Tong Geng, and Dongfang Liu. Iner- tial confinement fusion forecasting via large language models. arXiv preprinl arXiv:2407. 1098, 2024c.
Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Jinjie Gu, and Huajun Chen. Unified Hallucination Detection for Multimodal Large Language Models. arXiv preprint urXiv:2402.03/ /90, 2024d.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts. In CVPR, 2014.
Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models. In CIKM, 2023c.
Zhenfang Chen, Rui Sun, Wenjun Liu, Yining Hong, and Chuang Gan. GENOME: GenerativE Neuro-SymbOlic Visual Reasoning by Growing and Reusing ModulEs. In ICLR, 2024e.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. Self-Playing Adversarial Language Game Enhances LLM Reasoning. arXiv preprint arXiv:2404. 10642, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL https: /ttps//mmmsss////////s wwwooo////oooooooooo. bllp//2l322223233/ 03 30 vicura/.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future. arXiv preprint arXiv:2309. 15002,542. 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training Verifiers to Solve Math Word Problems. arXiv preprint vrXiv:2110.14166 2021.
Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic Analysis Hallucination in GPT-4V (ision): Bias and Interference Challenges. arXiv preprint arXiv:2311. 1.0328.......... 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards General-Purpor Vision- Language Models with Instruction Tuning. In NeurIPS, 2024.
Ishita Dasgupta, Andrew Lampinen, Stephanic CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language Models Show Human- Content Effects on Reasoning. arXiv preprint arXiv:2207.0705// 2022.
Danny Driess. Fei Xia, Mehdi SM Sajjadi, Corey Lynch. Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An Embodied Mul- timodal Language Model. arXiv preprint 1003:32300.03338, 2023.
Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li. Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. MouSi: Poly-Vin Sely-Visull-Ean Expeert Vision-Language Models. arXiv preprint arXiv:2401.1222 2024.
12

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A Comprchensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2024.
Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. OpenAGI: When LLM Meets Domain Experts. NeurIPS, 2023.
Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-image pretraining with hierarchy-aware attention. In ICLR, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Ster Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role Image Understanding in Visual Question Answering. In CVPR, 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination & Visual Illusion Large Vision-Language Models. In CVPR, 2024a.
Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting. In AAAI, 2024b.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and Preventing Hallucinations Large Vision Language Models. In AAAI, 2024.
Cheng Han, James C Liang, Qifan Wang, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Ying Nian Wu, and Dongfang Liu. Image Translation as Diffusion Visual Programmers. In ICIR, 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In EMNLP, 2023.
Ju He, Shuo Yang, Shaokang Yang, Adam Kortylessi, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. PartlmageNet: A Large, High-Quality Dataset of Parts. In ECCV, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.
Wenyue Hua and Yongfeng Zhang. System 1+ System 2= Better World: Neural- Saural-Symbolic Chain of Logic Reasoning. In EMNLP, 2022.
Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang. and Yongfeng Zhang. Trustagent: Towards safe and trustworthy llmebased agents. In Findings the for Computational Linguistics: EMNLP 2024, pp. 10000-10016, 2024.
the Association
Jie Huang and Kevin Chen-Chuan Chang. Towards Reasoning in Large Language Models: A Sur- vey. In ACL, 2023a.
Jie Huang and Kevin Chen-Chuan Chang. Towards Reasoning in Large Language Models: A Sur- vey. In ACI., 2023b.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is Not All You Need: Aligning Perception with Language Models. In NeurIPS, 2023.
Drew A Hudson and Christopher D Manning. GQA: A New Dataset for Korl-World Visual Reason- ing and Compositional Question Answering. In CVPR, 2019.
Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In ACL,, 2020.
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From CLIP to DINO: Visual Encoders Shout in Multi-Modal Large Language Models. arXiv preprint ar 200i 2023.
13

---

Published as a conference paper at ICLR 2025
Mingyu Jin, Qinkai Yu. Jingyuan Huang, Qingcheng Zeng, Zhenting Wang. Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge at different layers? arXiv preprint arXhv:2404.07066, 2024a.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng. Yongfeng Zhang, and Mengnan Du. The Impact of Reasoning Step Length on Large Language Models. In ACL, 2024b.
Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.0836/1 2020.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to Objects in Photographs of Natural Scenes. In EMNLP, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment Anything. In ICCV, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In NeurlPS, 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning Segmentation via Large Language Model. In CVPR, 2024.
Bin Lei, Chunhua Liao, Caiwen Ding, et al. Boosting Logical Reasoning in Large Language Models Through a New Framework: The Graph of Thought. arXiv preprint arXiv:2308. 08614, 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Bench-
marking Multimodal LLMs with Generative Comprehension. In CVPR, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre- Pre-Training for Unified Vision-Language Understanding and Generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICMI., 2023a.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian Yun Nie, and Ji-Rong Wen. HALUEVAL: A Large- Scale Hallucination Evaluation Benchmark for Large Language Models. EMNLP, 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. In ACL, 2023c.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, Schulman, llya Sutskever, and Karl Cobbe. Let's Verify Step by Step. arXiv preprint arXiv:2305.20055 2023.
Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, and Yongfeng Zhang. Battleagent: Multi-modal dynamic emulation historical battles to complement historical analysis. arXiv preprint 2tt.:24004155322 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurlPS, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. NeurlPS, 2023b.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved Basclines with Visual Instruc- tion Tuning, In CVPR, 2024a.
Yixin Liu, Avi Singh, C Daniel Freeman, John D Co-Reyes, and Peter J Liu. Improving Large Language Model Fine- Tinn-Tunig for Solving Math Problems. arXiv preprint arXiv:2310 10047, 2023c.
14

---

Published as a conference paper at ICLR 2025
Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of- Spot: Interactive Rea- soning Improves Large Vision-Language Models. arXiv preprint wrXiv:2403.12266, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In NeurIPS, 2022.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In NeurlPS, 2023a.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic Prompt Learning Policy Gradient for Semi-Structured Mathematical Reasoning. In ICML, 2023b.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and Quick: Efficient Vision-Langguae Instruction Tuning for Large Language Models. In NeurlPS, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative Refinement with Self-Feedback. In NeurIPS, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and Comprehension of Unambiguous Object Descriptions. In CVPR, 2016.
Kai Mei, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Llm agent operating system. arXiv preprint arXiv:2403.1699) 2024.
Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning. In NewrIPS, 2021.
OpenAL. GPT-4 Technical Report. arXiv preprint arXiv:2303.08772 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training Language Models Follow Instructions with Human Feedback. In NewrlPS, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS, 2019.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. In ICLR, 2024.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. DetGPT: Detect What You Need via Reasoning. In EMNLP, 2023.
Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, and Jie Tang. Cogcom: Train large vasion-language models diving into details through chain of manipulations. arXiv preprint arXi::2402.04236 2024.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuangi Tan, Fei Huang, and Huajun Chen. Reasoning with Language Model Prompting: A Survey. ACL, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML. 2021.
15

---

Published as a conference paper at ICLR 2025
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. PACO: Parts and Attributes of Common Objects. In CVPR, 2023.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of Textual Phrases in Images by Reconstruction. In ECCV, 2016.
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In ACL, 2023.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for cfain-of-thought reasoning. arXiv preprint 2rXvv:2403.16999 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- GPT: Solving AI Tasks with ChatGPT and Its Friends in Hugging Face. NeurlPS, 2023.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When Do We Not Need Larger Vision Models? arXiv preprint arXiv:2403.1 3043, 2024.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models that Can Read. In CVPR, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303./5388 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In ICCV, 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. In CVPR, 2024a.
Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning. arXiv preprint arXiv:2403.20046, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models. arXiv preprint 37X1V 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Founda- tion and Fine-Ture Tuned Chat Models. arXiv preprint 13Xi::2307.09288 2023b.
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards Understanding Chain-of-Thouge Prompting: An Empirical Study of What Matters. In ACL, 2023a.
Taowen Wang, Yiyang Liu, James Chenhao Liang, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng. Zenglin Xu, Cheng Han, et al. M Multimodal Prompt Tuning for Zero-shot Instruction Learning. In EMNLP, 2024a.
Taowen Wang, Yiyang Liu, James Chenhao Liang, junhan zhao, Yiming Cui, Yuning Mao, Shao- liang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, and Dongfang Liu. 2 pt: Multimodal prompt tuning for zero-shot instruction learning. In EMNLP, 2024b.
Weihan Wang. Qingsong Lv. Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song. et al. CogVLM: Visual Expert for Pretrained Language Models. arXiv preprint arXiv:2311 1003079 2023b.
16

---

Published as a conference paper at ICLR 2025
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World. In ICLR, 2024c.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. VisionLLM: Large Language Model is Also an Open-Ended Decoder for
Tasks. In NewrlPS, 2023c.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. Self- Ceff-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le. Denny Zhou, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022.
Yilin Wen, Zifeng Wang, and Jimeng Sun. MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models. arXiv preprint arXiv:2308.09729, 2023.
Penghao Wu and Saining Xie. Guided Visual Search as a Core Mechanism in Multimodal LLMs. In CVPR, 2024.
Yixuan Wu, Yizhou Wang. Shixiang Tang. Wenhao Wu, Tong He, Wanli Ouyang, Jian Wu, and Philip Torr. DetToolChain: New Prompting Paradigm to Unleash Detection Ability of MLLM. arXiv preprint arXhv:2403.12488 2024.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images. arXiv preprint urXiv:2403.11700 2024a.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv preprint arXiv:2401.11817, 2024b.
Fanglong Yao, Changyuan Tian, Jintao Liu, Zequn Zhang. Qing Liu, Li Jin, Shuchao Li, Xiaoyu Li, and Xian Sun. Thinking Like an Expert: Multimodal Sopergrapp-oo-Thughhh (HOT) Reasoning to Boost Foundation Modals. arXiv preprint arXiv:2308.06200 2023a.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurlPS, 2023b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023c.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond Chain-of-Tree Thouuhh Effective Graph-of-Thour Rea- soning in Large Language Models. arXiv preprint arXiv:2305. 16582, 2023d.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mPLUG-OWI: Modularization Empowers Large Language Models with Multimodality. arXiv preprint arXiv:2304. 14174, 2023.
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. Idealgpt: Iteratively Decomposing Vision and Language Reasoning via Large Language Models. In EMNLP, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet Vet: Evaluating Large Multimodal Models for Integrated Capabilities. In ICML, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-Rewarding Language Models. arXiv preprint arXiv:2401. 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Self-Taught Reasoner Boot- strapping Reasoning with Reasoning. In NewrIPS, 2022.
17

---

Published as a conference paper at ICLR 2025
Runjia Zeng, Cheng Han, Qifan Wang. Chunshu Wu, Tong Geng, Lifu Huang, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. In NeurIPS, 2024.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How Language Model Hallucinations Can Snowball. In arXiv preprint arXiv: 13534, 2023a.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. GPT4ROI: Instruction Tuning Large Language Model on Region-of-Interest arXiv preprint (XXiv:3377003601 2023b.
Yanzhe Zhang. Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLAVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding. arXiv preprint arXiv:2306.17107, 2023c.
Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. Meta Prompting for Al Systems. arXiv preprint drX11:2311.11482, 2023d.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic Chain of Thought Prompting in Large Language Models. In ICLR, 2023e.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du Explainability for Large Language Models: A Survey. ACM Transactions Intelligent Systems and Technology, 15(2):1-38, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judge with MTT-Bench and Chatbot Arena. In NeurlPS, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing through ADE20K Dataset. In CVPR, 2017.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In ICLR, 2023.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and Mitigating Object Hallucination in Large Vision- Language Models. In ICLR, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhanc- ing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304. 10592, 2023a.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving Math Word Problems via Cooperative Reasoning Induced Language Models. ACL, 2023b.
18

---

Published as a conference paper at ICLR 2025
SUMMARY OF THE APPENDIX
This appendix contains additional details for the ICLR 2025 submission, titled "Visual Agents as Fast and Slow Thinkers". The appendix is organized follows:
SA provides Implementation Details and Pseudo Code.
SB reports more Results for Different Thinking Modes.
$C reports more Quantitative Results for Visual Question Answering.
ID shows more Quantitative Results for Segmentation.
SE analyzes Failure Case.
F examines the Limitation and Future Work of our research.
$G discusses the Social Impact of our research.
SH offers Ethical Guard or our dataset.
SI claims Reproducibility of our approach.
SJ supplies Data License for the methods used for comparison.
A IMPLEMENTATION DETAILS AND PSEUDO-CODE OF FAST
Visual Resampler. The resampler (Alayrac et al., 2022) compresses nigh-dimensional visual fea- tures into a fixed-size latent space using a cross-attention mechanism. It begins with set of learn- able latent embeddings, which query the vision encoder's output features through scaled dot- dot-product attention. Each latent embedding attends selectively to the most relevant visual tokens, guided by attention weights computed via the query-key interaction. The process iterates across multiple lay- ers of cross-attention, followed by feedforward transformations, refining the latent representations at each step. This approach ensures efficient dimensionality reduction while retaining critical infor- mation, producing compact set of visual tokens for downstream tasks.
-lapee-parametrs. We follow established methodologies and utilize LLaVA-v1.5 (Liu et al., 2024a) the foundational visual agent. The image resolution is preprocessed 336 x 336 pixels to accommodate the http--ittarrg-pacch4--33333 14-366 vision encoder (Radford et al., 2021). The Adam optimizer (Loshchilov & Hutter, 2019) is employed with the DeepSpeed ZeRO 2 configuration for fine-tuning the switch, proposal, and segmentation adapters with LoRA (Hu et al., 2022). For the LoRA configuration, we the rank 128 and alpha to 256, consistent with the settings of ALaVA-v1.5 Additionally, we adjust the learning rate of the vision encoder projection layer to 2e-5 achieve better alignment. An MLP projection with channels of [256, 4096, 4096] is used to connect image representations into the word embedding space for the projection layer. An additional resampler projection layer is used to reduce the number of image tokens.
Training Data for Switch Adapter. Consistent with the pretraining stage of LLaVA-v1.5, we initially pretrain Vicuna-v1.5 1.5 as base frozen large language model and for the MLP projection layer and sampler layer of the CLIP vision encoder using a 558K subset of the LAION-CC-SBU dataset with BLIP (Li et al., 2022) captions. During the fine-tuning stage, we integrate the negative dataset acquired from V (Wu & Xie, 2024) and PixelLM (Ren et al., 2024) and with the original LLa LLaVA-v1.5 instruction tuning 665k data* for LoRA based finetuning.
Specifically, The dataset for fine-tuning the switch adapter was carefully constructed to emphasize scenarios requiring precise object recognition and complex reasoning. For the GQA subset of the 167k VQA data, specifically targeted questions where the annotated objects mentioned in the query were critical for deriving the correct answer. Initially, the InstructBLIP model was used to evaluate GQA questions with annotated objects. Only questions that the model could correctly an- swer were retained. To ensure the importance of these annotated objects, we applied the LaMa image inpainting model erase the mentioned objects from the corresponding images. The modi- fied images were re-evaluated using InstructBIIIP, and only questions that the model failed to answer
"https///itthbb.ccm http:///ghhbb.ooo//nnoooo///e/ss
"httss//hugginnffac. http::/hwugggggisscccooattttttnn cowdanaeet///uuannniinnlll/ waw.aaaaiiiiiiaaaaaaaaaaaaaaaaa VA--Instrutt-150K
httpps//htgggnggaccc.ooaaassss// coldutaseet/llluaaaiian/// LLVAA-retrainn
19

---

Published as a conference paper at ICLR 2025
after object removal were included. This process ensured that the curated subset focused exclusively on questions where the annotated objects were essential, forming a robust component of the VQA data.
For the VAW object attribution dataset, both open-ended and binary questions about object attributes were synthesized. Open-ended questions were formulated around attributes such "color," "mate- rial," and "pose," while binary questions incorporated additional attributes like "state" and "optical property." Answer formats adhered to predefined structures ensure consistency. The same ob- ject removal and re-evaluation strategy as used in the GQA subset was applied, filtering the data to include only questions where the absence of objects rendered the query unanswerable.
From the LLaVA-800 instruction tuning data, noun phrases were extracted from the text of questions or instructions and matched with object category names defined by COCO, augmented with common synonyms such as "man" and "woman" for the "person" category. Images were retained only if the identified categories had annotated instances with bounding boxes. These annotated instances, along with their bounding box coordinates, were used as target objects during training.
In addition, we incorporated data generation approach inspired by LISA, utilizing GPT-4 and GPT-4V to expand and diversify the dataset. Initially, LLAVA was used for image captioning, and GPT-4 generated questions about multiple regions in the image. While this approach utilized pre- existing mask annotations to reduce costs, its diversity was limited to the scope of the captions. To address these limitations, we refined the pipeline with GPT-4V, leveraging its advanced capabilities in visual understanding. Image captions, object names, and bounding box coordinates into GPT-4V, which, using dynamically crafted prompts, autonomously selected instances and gen- erated nuanced question-answer pairs tailored the image content. This refinement significantly improved the diversity and contextual relevance of the data. An illustrative example of such prompts provided below:
input
Prompt: Imagine you need to query machine agent about image. The image has a height of 720 pixels and a width of 1280 pixels. You are given several entities described by list, each identifying an object in the image along with its location. The class names and corresponding coordinates as follows:
Dog at 1350.12, 450.45, 480.89, 600.671 671;
Ball at [200.33, 300.22, 78, 350.56].
Grass at [0.0, 500.c, 1280.0, 720.0];
Coordinates are represented as (top-left x, top-left y, bottom-right x, bottom-right y) The question must incorporate at least two of these objects and require reasoning about the relationships or interactions between them. Additional requirements for the generated question are follows:
The answer to the question must explicitly reference each included object or its equivalent and avoid implying the presence of any other objects not listed.
2. The question must be precise, meaningful, and avoid being overly general.
The question should frame single cohesive activity or relationship rather than merely combining independent sub-queries.
4.When answering, the class names should be rephrased to indicate their position, role, or interaction in the image.
This multi-faceted dataset construction process ensured the generation of diverse and challenging samples, providing a robust foundation for fine-tuning the switch adapter on complex reasoning tasks.
20

---

Published as a conference paper at ICLR 2025
Training Data for Proposal Adapter To determine the corresponding region for query, LRP++ (Chefer et al., 2021) I) for data construction, similar to Chain of Spot (Liu et al., 2024b). Our
initial prompt is as follows:
<Image>
answer the question: [Q],
where is the region of interest in the image based on [C]?
Ans. str [wo, W1r ho, hi]
The question Q and the context clue formatted to get the answer in terms of bounding box. In this format, Wo. Wi represent the left and right boundaries, respectively, while hu, hı denote the upper and lower boundaries. To identify the correct region, we sampled one question per image from the LLaVA instruction tuning data, consisting of a total of 665k data for proposal finetuning.
Training Data for Seg Adapter. Adopting an approach similar to LISA Lai et al. (2024), the training data for our model comprises three distinct segments: semantic segmentation dataset, referring segmentation dataset, and a reasoning segmentation dataset. We deliberately exclude vi- sual question-ansseriin datasets to enhance the model's segmentation performance. The semantic segmentation segment includes the ADE20K (Zhou et al., 2017), COCO-Stufi (Caesar et al., 2018), and LVIS -ACCO (Ramanathan et al., 2023) part segmentation datasets. The referring segmentation datasets encompass refCOCOKazenzade fCOCOg (Caesar et al., 2018), and refCLEF (Rohrbach et al., 2016). The reasoning segmentation dataset includes ReasonSeg (Lai et al., 2024). It is important to note that the referring segmentation and reasoning segmentation datasets are carefully excluded during the evaluation of the segmenta- tion benchmarks to prevent any potential data leakage.
et al. (2014), refCOCO+ (Kazemzadeh et al., 4),4), re-
Pseudo-code Implementation. The pseudo-code FAST is given in Pseudo-code 1.
B MORE RESULTS FOR DIFFERENT THINKING MODES
As shown Table 6, FaST demonstrates strong performance in both System I and System rea- soning VQA datasets, outperforming baseline methods in most cases. Notably, FaST achieves highest accuracy challenging System tasks across GQA, VQA", and SQA', which require advanced reasoning capabilities. This highlights the effectiveness of the switch adapter mechanism in dynamically allocating tasks based on complexity. While maintaining competitive performance in simpler System I tasks, FaST leverages its adaptive architecture to excel in more complex scenarios, as evidenced by its superior System 2 results.
Further, in Table 7, FaST's robustness extends reasoning segmentation tasks, where it achieves significant improvements in System performance compared to baseline models such as LLaVA with segmentation and LISA-7B. For example, in the ReasonSeg dataset, FaST records remarkable 48.2 CloU in System 2 tasks, significantly outperforming L-SA-7B and LLaVA, which achieve 43.3 CloU and 42.4 CloU, respectively. This result underscores FaST's ability generalize effectively across diverse task families and reasoning paradigms.
Overall, the results validate the universal applicability and robustness of the FaST framework. By effectively utilizing the switch adapter to allocate tasks dynamically, FaST demonstrates strong capability balance performance across both simple and complex reasoning tasks, making it reliable solution for diverse real-world applications.
C MORE QUALITATIVE RESULTS FOR VISUAL QUESTION ANSWERING
Figure 7 presents additional qualitative results for Visual Question Answering (VQA). Our FAST framework consistently demonstrates remarkable performance across various challenging scenarios. Notably, in the bottom right corner of Figure 7, our FAST leverages extensive world knowledge to identify the keyboard, which subsequently aids in discovering the hidden computer mouse and pro- viding the correct answer. This ability integrate and utilize contextual information showcases the
21

---

Published as a conference paper at ICLR 2025
Algorithm 1: Pseudo-code of FAST in PyTorch-like style.
class FaST:
def init (self, switch proposal_llm, sig_llm)
self.awitch. 11m switch self self.proposal_1ae prropssal_la 11m
fel seq_iim
def get_contextual_clues (self, image, question);
# Get missing objects and context clues using switch adapter return self.
queation)
Construct Chain of Rvidence
def construct_coe (self, image, question, context_clues, missing_objects)) # Step 1: Get region proposals
region self.proposa 11m (image, question, context_clues)
Step 2: Get pixel-level mask for the missing objects wew..seg_llmmee.... (region, missing_cbjects!
return (context_clues, region, missing_objeets, mask)
Main Fuaction.
def forward (self, image, question):
initial answer
initial_answer wwlf..witchhllmleee
question)
# Check if slow thinking is needed based the initial answer if *sory,, can not answer" in initial_enswer lower (): $ Pe-form slow thinking
missing_objects, context_cluee anitial_ obbj',, initial_answer ['clue']
chain_of_evidence = self.construet_coe (image, question, context_clues, missing_objectss)
f Generate the final answer using the constructed chain of evidence final_answer self..sittctlllmiinge question, chain_of_errdence))
else:
Perform thinking
final_answer = initial_answa
return
VQΛ22
GQA
VQA
SQA'
Method
LLM
Sys 1
Sys 2
Sys
Sys 2
Sys
Sys 2
Sys 1
Sys 2
BLIP-2
Vicuna- 13B
67.3
53.1
37.8
22.4
44.3
39.7
63.4
59.2
LLaVA-v1.5
Vicuna-77 7B
81.2
68.0
70.3
47.0
61.1
53.7
68.4
65.7
Chain of Spot
Micuna-7B
82.1
74.5
70.9
50.7
62.1
59.0
68.6
67.8
FAST (Ours)
Vieuna-77 -B
81.1
75.5
70.2
52.3
61.2
60.2
68.2
70.2
Table 6.
System and System 2 performance on VQA datasets. FaST demonstrates superior performance
in both reasoning modes compared to baselines.
model's ' advanced capabilities and highlights its potential for practical applications. The qualitative results further underscore FAST's robustness and versatility in handling diverse VQA tasks.
D MORE QUALITATIVE RESULTS FOR SEGMENTATION
Figure 8 showcases further qualitative results for the Segmentation task. Our FAST model excels in various challenging scenarios, accurately locating difficult targets and performing complex reason- ing for more demanding queries. For instance, in the bottom right corner, the model successfully identifies an appliance that can be turned on when feeling hot by recognizing relevant contextual clues that suggest the appliance should probably appear the wall, thereby resulting in the correct answer. This example demonstrates the model's advanced understanding, adaptability, and preci- sion.
22

---

Published as a conference paper at ICLR 2025
refCOCOg
ReasonSeg
Method
Sys
Sys 2
Sys 1
Sys 2
LISA-7B
70.2
63.4
46.6
43.3
LLaVA Seg
68.4
60.2
44.2
42.4
FaST (Ours)
70.8
64.1
46.4
48.2
Table 7. System 1 and System 2 performance on reasoning segmentation tasks. FaST achieves strong per- formance across both tasks, demonstrating its robustness and effectiveness in dynamic task allocation between System and System 2.
Query
Query
Are the trousers the color
the below
Clue: The Chain Evidence
Chain Evidence
Clue: The trousers most lilely to appear
most likely to appear next refrigerator.
Proposal
Proposal <gag*
Anz: Yes.
Quany
Quany
Is the male person benind the catcher face mask?
Is the computer mouse on top of the mouseepod both block and small?
Chain of Evidence
Chain of Evidence
The mole person most likely
Clac: computer mouse most
to appaor the baseball foifor
likely to appear next to the keyboord.
Proposal
Proposol
Anst Yes.
Figure More qualitative results Visual Question Answering.
Query
Segment the appliances that be on
segment plant room.
Choin Evidence
Chain of Exidence
Clue: The plant is most likely to appear the toble.
Clue: The appliances that con be
Turned on when it's hot most likely to appear on the woll
Proposol
<pogo
Proposal
Querry Please segment the black white dog with pointy ears. Chain Evidence
Please segment the vehicle
eating.
Chain of Evidence
Clus: The block and white dog with pointy mostly likely appear
Clue: The vehicle take after eating
mostly likely to appear on the road.
Proposol
Proposai
Figure More qualitative results Visual Question Answering.
E
FAILURE CASE
In Figure 9, present overview of the most notable failure cases, providing insights into the distinct patterns that lead to suboptimal outputs in our FAST model. These challenges include dif- ficulty in triggering the System thinking mode, constructing adequate contextual clues, generating appropriate proposals, and providing accurate pixel masks. The model often fails to recognize the need for deliberate reasoning. relying instead on System I thinking, which leads to incorrect re- sponses, seen in Figure 9a. Inadequate contextual clues generated by the switch adapter impair model's focus on the correct region, resulting in vague or incorrect responses, as illustrated in Fig. 9b. The proposal adapter's 's inaccurate identification of regions of interest, shown in Figure 9c. leads to proposals that do not correspond to the query. Additionally, the segmentation adapter struggles with producing precise masks, particularly for small or occluded objects, causing erro- neous conclusions, as highlighted Figure 9d. These failure cases underscore the urgent need for refinement in FAST framework, emphasizing the importance of significantly enhancing the precision of the system switch adapter, improving contextual clue construction, and optimizing the proposal and segmentation adapters achieve more reliable and consistently accurate responses in complex visual and textual tasks.
23

---

Published as a conference paper at ICLR 2025
Is this picture Canelles Baix (la Binya]?
Does the flower look yellow?
Answer
Ard: No.
Ans: No.
Fast Answer
Fost Answor
X Ans: Yes.
Ans: Yes, the flower yellow.
(a) The model fails to trigger System thinking mode.
Is this artwork disployed in St. vitus's cathedral Progue?
What rope attcced to?
Ang: Coot.
Ars: No.
Choin of Evidence
Choin of Exidence
Clue: The most likely appear
Clue: The vitus cathedral, Proque likelly appear the cliff.
present image.
Proposal
<seg
Propossil
sseg
X Yes.
(b) The model fails to construct adequate contextual clues.
Quary
What the color the
blue luggage left right side
Ang: White.
Ans Right.
Choin of Evidence
Choin of Evidence
Clue: The
likely to appear
Clue: The blue luggage, most oppear Left.
next person.
Preposal
coego
X
Proposal csego
(c) The model fails generate appropriate proposals.
Query
Is lomp the right
Are there yellow shoes
to the left of the car?
GT Answer
ST Answer
Ars: Left.
Chain of Evidence
Zhain of Evidence
Clue:
The cars and lamp is most likely to appear on the street.
Clue: The yellow shoes likely appear
players.
Proposal
<sag>
Proposal <seg>
X Ans: Right.
Argi Yes.
(d) model fails provide accurate pixel masks.
Figure 9, Failure cases of Our FAST system.
24

---

Published as a conference paper at ICLR 2025
F LIMITATION AND FUTURE WORK
While the FAST framework has demonstrated sigmificant advancements emulating human-like cognitive processes in visual AI through its fast and slow thinking mechanisms, several limitations warrant attention. Firstly, the system's reliance on a predefined set negative data for training the switch adapter may not encapsulate the full spectrum of real-world complexities. Firstly, this could lead to suboptimal performance when faced with novel or unexpected scenarios. Secondly, despite its fine-grained analysis capability, the pixel-level mask decoder might struggle with highly textured or patterned images where segmentation becomes challenging. Lastly, the generalizability of FAST across various domains and tasks necessitates further validation to ensure its robustness and reliability in diverse applications. We plan to develop advanced learning mechanisms that will allow the model generalize more effectively beyond the predefined negative dataset. Additionally, we will focus optimizing it for real-time applications to reduce computational overhead and response times.
For the recent large reasoning model like OpenAl ol model, while these models leverage reinforce- ment learning and internal chains of thought to achieve scalability, they often require significant computational resources, making them less efficient.
In contrast, FaST is designed to prioritize multimodal reasoning with a clear focus on transparency and adaptability. The use of interpretable reasoning modes (System I and System 2) ensures that FaST provides insights into its fecisioo-makkng processes, which is critical for applications requir- ing explainability. Additionally, FaST's modular design allows it to balance computational effi- ciency and accuracy dynamically, making it suitable for diverse and resource-constraine environ- ments. These strengths highlight FaST's unique contributions and its complementary potential to scalable reasoning approaches.
G SOCIAL IMPACTS
The development and deployment of FAST significantly advance Al by enhancing the human- cognitive abilities of LLM-based visual agents. Positively, FAST enables sophisticated applications in areas like vision-ban based dialogues and security surveillance, while its transparent decision-making fosters trust and ethical AI practices. However, reliance large models and extensive datasets risks perpetuating biases, potentially leading to unjust or discriminatory outcomes. Addressing these ethical concerns and establishing responsible usage guidelines essential for the responsible de- ployment of such advanced AI systems.
H ETHICAL SAFEGUARDS
In our paper introducing a novel framework FAST, we implement rigorous ethical measures to prevent potential misuse and promote responsible application. These measures delineated in comprehensive protocols accompanying the final release of models and datasets. Our protocols en- compass stringent usage guidelines, access controls, incorporation of safety filters, and monitoring systems. These concerted efforts reflect our steadfast dedication to upholding the utmost ethical standards in scientific exploration. Our objective protect the rights and privacy of all stakehold- ers involved, thereby fostering a culture of responsible and ethical research within our community.
I REPRODUCIBILITY
Our FAST framework is implemented in Py PyTorch (Paszke et al., 2019). All the experiments are conducted on eight NVIDIA A100-80GB GPUs. Our full implementation shall be publicly released upon paper acceptance to guarantee reproducibility. The codes are available at the anonymous link for the review process.
25

---

Published as a conference paper at ICLR 2025
All Experiments (switch, proposal, and seg adapter) conducted on eight NVIDIA A100-80GB SXM GPUs?. Reproducing the finettning process would require approximately 15 A100 GPU days.
J LICENSES FOR EXISTING ASSETS
All the methods used for comparison are publicly available for academic usage. The switch adapter is implemented based on the released code (https: : /gi thub. com/ penghao-wu vstar) with an MIT license. The proposal adapter is implemented //github. com/ dongyh20 Chain-ann of- Spot) with an Apache-2.0 license. The seg adapter implemented on the released code pestps:/ggithee com/ /mmddv /dvlab- Research/LISM. with an Apache-2.0 license.
the released code (https:
26


Reviews:
[{"review_id": "vk8fB4vnUg", "reviewer": "Reviewer_M2ti", "summary": "Proposes FaST, a Fast-and-Slow-Thinking mechanism for vision-language models that models both “system 1” (fast) and “system 2” (slow) thinking, dynamically switching between the two based on query complexity. The method demonstrates improvements over baselines on standard benchmarks in addition to improved reasoning and transparency.", "questions": "Please see weaknesses above. I believe the paper in its current form already presents promising ideas+results but can be made still stronger.\n\n– It would be good include a discussion / conceptual comparison to recent “large reasoning models” that propose scaling test-time compute eg. the OpenAI o1 model. While not (yet) multimodal, such methods are speculated to be trained w/ reinforcement learning and to produce “internal” (i.e. not necessarily interpretable) chain of thought – what are some of the tradeoffs of this type of approach against FaST?\n\n– [Minor] As I understand, FaST does not provide increased transparency when it opts for “fast” thinking – it would be good to clearly demarcate this.\n\n– [Minor] L538 typo: extra \"A\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "– The paper presents a highly novel solution to an important problem\n\n– The paper is very well-written and easy to follow\n\n– The motivation for the approach is laid out very clearly and is compelling\n\n– The proposed approach  is designed intuitively, and shows convincing improvements over prior work\n\n– The neuro-symbolic nature of the approach directly translates to improved transparency\n\n– The paper presents a comprehensive set of results and ablation studies on visual question answering and referring and reasoning segmentation benchmarks. The qualitative results included in the paper are also very compelling\n\n– The paper includes a detailed appendix and code implementation which will aid in reproducibility", "weaknesses": "– [Complexity] One concern I have is the complexity of the approach, especially considering the large number of moving parts. Do the same set of method hyperparameters generalize across settings, or was a large amount of tuning required to achieve the results reported in the paper? \n\n– [Clarity] I found some of the method details to be lacking eg. L259-260 mentions “a fast sampler based on cross-attention” but I could not find any further details for this component.\n\n– [Performance] While FaST does seem to consistently outperform the LLaVA-1.5 base approach that it builds on top of, it is not exactly an apples-to-apples comparison considering the additional models /  datasets that are used (including large ones such as SAM). Meanwhile, its gains over methods reasoning-focused approaches like Visual CoT and Chain-of-Spot are modest (and sometimes lags considerably eg. behind VCoT on VQAT) – a deeper analysis of / comparison against these methods (eg. average runtime etc.) would be helpful to establish the pros/cons of using FaST. The comparison against CoVLM and CogCOM appears more significant (Fig 4) but is missing a comparison to the LLaVA-1.5 baseline w/ the stronger encoder. \n\n– [Analysis] The results presented in Figure 5 are very interesting, but are missing baseline performance of the base LLaVA-1.5 model: what is its accuracy on the subset of system 2 / system 1 questions (as judged by FaST) in GQA/MME? How much does “slow” thinking actually improve performance on such questions (assuming that the switch adapter is accurate – it would also be good to validate this eg. by collecting question complexity labels)?\n\n– [Analysis] It would be valuable to include an evaluation of the correctness/quality of the “chain-of-evidence” generated by the approach, perhaps via a human study and/or model interpretability methods. Does the chain-of-evidence appear faithful or is it mostly a post-hoc rationale? How often does a “correct” chain of evidence lead to the right conclusion, and vice versa? Is the chain-of-evidence used for the model “optimal”, or does it include unnecessary steps or logical leaps? This will help build confidence around the improved transparency claims made in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wViSWwFpR7", "reviewer": "Reviewer_GSxm", "summary": "The paper presents FaST, a framework for improving the cognitive capacities of visual agents.   \nInspired by System 1 and System 2 thinking in human’s cognition, FaST incorporates a switch adapter that dynamically toggle between a faster, intuitive response mode and a slower, analytical reasoning mode depending on the task complexity. \nFaST also features a high interpretability by integrating hierarchical reasoning strategies and symbolic intermediate outputs into the framework.\nExperiments show that the proposed outperforms baselines in various VQA and multimodal benchmarks.", "questions": "1. Can the authors provide more details in the synthesis of the dataset used for switch adapter finetuning? Specifically, how are the negative samples from V* augmented to cover the two cases that are expected to trigger a slow thinking mode switching (uncertainty in pinpointing objects or the targets are too small)?    \n2. Will the datasets collected by the authors be released? The reviewer believes that these datasets are crucial for reproducing the FAST framework proposed in the paper, and they will also be highly beneficial for future related work.\n3. Can the authors provide some experimental results for other reasoning tasks, like raven, bongard or physical reasoning tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. FaST strikes a balance between accuracy and efficiency for visual agents by dynamically integrating fast and slow reasoning. The motivation to incorporate different thinking modes based on the scenario is intuitive, and the primary concept is presented clearly and logically.\n2. The interpretability and transparency of the proposed framework are improved through the introduction of a chain-of-evidence mechanism and the generation of symbolic intermediate results as feedback.\n3. Experimental results across various tasks validate the effectiveness of the proposed framework.", "weaknesses": "1. The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks (e.g., the Bongard problem, physical reasoning tasks) to further validate the generalization capabilities of the proposed method.\n2. The efficacy of the switch adapter can be further analyzed in Section 3.2:\n  * No investigations have been conducted on the switching ratio and performance on reasoning segmentation and other tasks, which is pivotal to demonstrate the universal compatibility of the switch adapter mechanism across different families of tasks.\n  * The description of accuracy rates “under System 2 mode” contains some ambiguities. The authors should clarify in the text that accuracy rates reported under System 2 are also evaluated using fast thinking mode, otherwise, the claim regarding the adapter’s ability to differentiate problem complexities is not adequately supported. \n  * The accuracy rates mentioned in the passage do not appear to match those shown in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JDpIs41Sk1", "reviewer": "Reviewer_uXT1", "summary": "The paper proposes a slow-fast paradigm for MLLMs based on Kahneman's System 1/2 therory. The framework consists of a \"fast\" module built upon MLLM and vision foundation models. A switch adaptor will determine if it's necessary to use \"slow\" module, which will gather all detailed clues to find a more accurate result. The approach shows impressive results across multiple benchmarks and outperforms baselines models.", "questions": "The major concerns are listed in the weakness part.\n\nMinimal concern: It is definitely not a good idea to use \"fast\" as an acronym for \"fast\" and \"slow\".\n\nPotential Typos:\n- Lines 151-156: Format like \"Equation 1\" does not align with Sec.2.2 \"Eq.3\"\n- Line 160: \"Def.2.2\", only Remark 2.2 exists\n- Line 196: \"Eq.2.1\", only Remark 2.1 exists\n- Line 215: \"Eq.2.4\", only Remark 2.4 exists", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The method is built based on the idea of Kahneman's system 1/2 theory from cognitive science, which is clearly-motivated and reasonable.\n\n- The approach is clearly written and its implementation is reasonable.\n\n- The authors provide comprehensive evaluation across multiple benchmarks and tasks. And the method outperforms baselines.\n\n- The authors provide clear visualizations of the model performance.", "weaknesses": "- The analogy between human cognitive system 1-2 and the \"fast\" \"slow\" module in the proposed method lacks rigor justification, or even misaligned. In human cognition, System 1 usually handles low-level tasks and is much faster, while System 2 is slow but handles complexity. However, in FaST, both the fast and slow modules solves same problem, but only have a tradeoff between efficiency and performance. Considering that the \"slow\" module is only 4x slower, we can abandon the switch adaptor and use the slow module all the time for a maximal performance.\n- From another perspective, if the only purpose of two stage model is the computation efficiency, then the proposed approach is more like an early-exit trick of a complex model (referring to the \"slow\" module). THis diminish the novelty of the paper.\n- The framework essentially implements a manual Chain of Thought (CoT) approach rather than dual cognitive system. If reframing the method in to CoT paradigm, a progressive visual cue integration would be potentially developed (more than 2 layers). This approach would gradually add visual evidence into the reasoning process, which could be both more efficient and accurate.\n- The complexity burden is only **shifted** to the switch adaptor, rather than being alleviated or addressed. So what is the performance of the switch adaptor itself?\n- The details of the datasetin remark 2.1 is missing. It is critical to introduce the construction process of the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FL63LTDT8G", "reviewer": "Reviewer_KNeJ", "summary": "The paper introduces FaST, a framework based on human reasoning that incorporates fast (system 1) and slow (system 2) thinking mechanisms into visual agents. FaST involves a switch adapter to change between systems 1 and 2 based on the complexity of the query question. While System 1 reasons the question in a short or one-step approach, system 2 uses a switch adapter to provide clues and object names, which later inspires the proposal adapter to provide region boxes and seg adapter to provide seg instances. In the end, all additional information will be part of a chain of evidence and prompt LLMs to do \"slow\" reasoning to output an answer. The paper does a comprehensive experiment comparing FaST with current models and demonstrates the effectiveness of FaST on both visual question answering and segmentation reasoning. Further ablation studies prove the significance of each component within the Fast.", "questions": "Overall, I found the paper very interesting, my main concern is within the weak points. Another question I have is, if this FaST can outperform other models by using only System 2, what are the advantages of switching between System 1 and System 2 instead of just System 2?  I think time cost would be an answer, but I'd be curious if the author provides more answers.\n\n\n\n\ntypo: In Line 538, AFurthermore should be Furthermore", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for incorporating systems 1 and 2 in visual agents seems promising and convincing. Especially the design of system 2, the introduction of clue, seg instance and chain of evidence can assist the reasoning process based on experiment results. The construction of Negative Data for Target Objects Reasoning Dataset seems useful for training the adapter switcher. \n2. The hierarchical reasoning capabilities that start with global image cues and drill down to pixel-level analysis provide a well-rounded method of handling visual inputs. The symbolic approach can provide more explainability on the model reasoning process.\n2. A comprehensive experiment, including many current models, proves the effectiveness of FaST. The ablation study on pipeline structure and the time comparison highlight the advantage of FaST.\n3. Overall, I found the paper well written, with clear definitions and formulas to demonstrate the complex system. The motivation based on human cognitive science is well presented.", "weaknesses": "1. Result Demonstration: based on the paper, I think we can assume the previous model mainly uses fast thinking on reasoning questions. A better way to illustrate the effectiveness of FaST is to split Table 1 with the model's system 1 and system 2 performance. In other words, for all those questions in which FaST use system 1 reasoning, compare the performance in one table and compare all questions with system 2 in another table.  I expect the improvement of FaST mainly comes from system 2, which means FaST will show similar performance with other models in questions that require system 1 reasoning.\n2. Dependence on Specific Datasets: I appreciate the work constructing the “Negative Data for Target Objects Reasoning Dataset\" to train the switch adapter. However, I am concerning the training data can not reflect all real-world scenario, which may limit generalizability. Is it possible for us to gain the signal from the model's inner representation to decide the switch between systems 1 and 2?\n3. Failure case: the paper should also explore scenarios where it fails or underperforms compared to baseline models. As the FaST use a hierarchical reasoning pipeline, it is possible the error of some component can make the whole system collapse or lead to a cascade of issues. An analysis of a failure case can provide insight into the model's weaknesses and future directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)