Caption This, Reason That: VLMs Caught in the Middle
Authors: Zihan Weng
Lucas Gomez
Taylor Whittington Webb
Pouya Bashivan
Venue: NeurIPS 2025 spotlight ()
Keywords: vlm
lmm
mllm
cognition
visual reasoning
reasoning
evaluation
finetuning
Primary area: neuroscience_and_cognitive_science
TL;DR: Novel systematic cognitive reasoning evaluation of VLMs, providing clues for the sources of reasoning bottlenecks, as well as simple and effective solutions.

Abstract: Vision-Language Models (VLMs) have shown remarkable progress in visual understanding in recent years. Yet, they still lag behind human capabilities in specific visual tasks such as counting or relational reasoning. To understand the underlying limitations, we adopt methodologies from cognitive science, analyzing VLM performance along core cognitive axes: Perception, Attention, and Memory. Using a suite of tasks targeting these abilities, we evaluate state-of-the-art VLMs, including GPT-4o. Our analysis reveals distinct cognitive profiles: while advanced models approach ceiling performance on some tasks (e.g. category identification), a significant gap persists, particularly in tasks requiring spatial understanding or selective attention. Investigating the source of these failures and potential methods for improvement, we employ a vision-text decoupling analysis, finding that models struggling with direct visual reasoning show marked improvement when reasoning over their own generated text captions. These experiments reveal a strong need for improved VLM Chain-of-Thought (CoT) abilities, even in models that consistently exceed human performance. Furthermore, we demonstrate the potential of targeted fine-tuning on composite visual reasoning tasks and show that fine-tuning smaller VLMs moderately improves core cognitive abilities. While this improvement does not translate to large enhancements on challenging, out-of-distribution benchmarks, we show broadly that VLM performance on our datasets strongly correlates with performance on established benchmarks like MMMU-Pro and VQAv2. Our work provides a detailed analysis of VLM cognitive strengths and weaknesses and identifies key bottlenecks in simultaneous perception and reasoning while also providing an effective and simple solution.

Caption This, Reason That: VLMs Caught in the Middle
Zihan Weng*
Lucas Gomez"
Integrated Program in Neuroscience (IPN)
Integrated Program in Neuroscience (IPN)
McGill University
McGill University
Mila, University of Montreal Canada
Mila, University of Montreal
Canada
z1 zihan. www.weein..me.eeeeeee.ee mca.ll..lacaa.
lucas mucas.gomezzmaale
www.e/mmmammmmmm.ll.lll......... ca
ca/
Taylor Whittington Webb
Pouya Bashivan
Microsoft Research
Department of Physiology
USA taylor www.ebbggail.com
McGill University
Mila, University of Montreal Canada
pouya. bws.aashhhaai www.baainnnnngglllcccommm ca
Abstract
Vision-Language Models (VLMs) have shown remarkable progress in visual un- derstanding in recent years. Yet, they still lag behind human capabilities in specific visual tasks such as counting relational reasoning. To understand the underlying limitations, we adopt methodologies from cognitive science, analyzing VLM per- formance along core cognitive axes: Perception, Attention, and Memory. Using suite of tasks targeting these abilities, we evaluate state-of-the-art VLMs, cluding GPT--o.. Our analysis reveals distinct cognitive profiles: while advanced models approach ceiling performance on some tasks (e.g. category identification), significant gap persists, particularly in tasks requiring spatial understanding selective attention. Investigating the source of these failures and potential methods for improvement, employ vision-text decoupling analysis, finding that mod- els struggling with direct visual reasoning show marked improvement when rea- soning over their generated text captions. These experiments reveal strong need for improved VLM Chain-of-Thie Thought (CoT) abilitics, even in models that consistently exceed human performance. Furthermore. demonstrate the po- tential of targeted fine-tuning on composite visual reasoning tasks and show that fine-tuning smaller VLMs moderately improves core cognitive abilities. While this improvement does not translate to large enhancements on challenging, out-of- distribution benchmarks, strongly correlates with performance on established benchmarks like MMMU-Pro and VQAv2. Our work provides detailed analysis of VLM cognitive strengths and weaknesses and identifies key bottlenecks in simultaneous perception and rea- soning while also providing an effective and simple solution.
show broadly that VLM performance on our datasets
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
A hallmark of human intelligence is the capacity for logical reasoning to solve problems and make decisions, Replicating these abilities in artificial beings has been a longstanding goal in the field of artificial intelligence. Recent advancements have demonstrated that large language models can sometimes exhibit reasoning capacity deemed comparable to humans, functioning effectively as few- shot or even zero-shot reasoners [μ--]]. Innovations such as chain-of-thouglt (CoT) prompting and majority voting have further enhanced these models, enabling them to approach, and in some cases rival, human-level reasoning capabilities, evidenced by their performance tasks like coding challenges and the ARC benchmark [5--7]]
Building the success of large language models (LLMs), vision-language models (VLMs) have emerged to address vision-language tasks. These large-scale models integrate pre-trained LLMs with vision models, such as Vision Transformers (ViTs), enabling them to proficiently handle tasks like visual question answering and scene description
Percspibc
Alteetoo
Corsocate Movel Reeectiyg
Atterani
Money
OUL
ONT
Pocostion
Maual Pesssoning
Ohil
(AAee Caree
Gogntthe Palaoring
Tapprto equul
Figure 1: PAM Dataset. a) Relationship between different cognitive abilities underlying reason- ing. b) Different components of the PAM dataset and example tasks from each. Green and blue correspond to Category and Location tasks respectively. A complete list of examples is provided in Appendix Figures A.
State-of-the-art (SOTA) vision-language models (VLMs) have been evaluated across wide range of benchmarks spanning diverse task domains such as simple object recognition, document understand- ing, and general reasoning tasks [0S-18]. Prominent benchmarks like MMMU-Pro MMBennh, and MME primarily assess cognitive reasoning at a high level, focusing on specific tasks such as coding and mathematical cooblem-solving [14-22]]
Despite their strengths, VLMs continue to struggle with visual Decision-making and reasoning tasks [22-25]] Campbell et al. [26] show that even tfate-of-the-an models perform poorly multi-object reasoning, such as counting or identifying objects, highlighting persistent deficits in visual binding, key perceptual ability [ Spatial reasoning remains another unsolved VLM challenge [28-
30, 23 61]]
Evidently, most of these VLM evaluations ultimately deviate from human cognitive science, where researchers break intelligence down into core facultiesPerception, Attention, and Memorythat under- lie higher-level functions like reasoning, decision-makin planning, and, more broadly, intelligence [32--55]] In humans, Perception refers to fine-grained sensory encoding [G6]], Memory to maintain- ing information despite distraction [37]], and Attention to selecting goal-relevant inputs [38]. By reframing VLM evaluation around these core abilities, we can pinpoint which cognitive mecha- nisms fail and whether deficits in tasks like spatial reasoning arise from encoding, maintenance, or selection. Here, we adopt that perspective and probe various VLMs through the lens of cognitive science.
Our contributions are as follows:
1. Systematically analyzing the cognitive profiles state-of-the-art VLMs using the procedurally-genente
Dicception-Attention-Meen (PAM) and Composite Visual Rea- soning (CVR) datasets.
2. Identifying specific weaknesses, particularly the pervasive difficulty with spatial informa- tion (localization) and selective attention, even in top-performing models.

---

3. Investigating the nature of the reasoning bottlenecks using a VLM-only vision-text decou- pling paradigm, revealing that limitations often stem from the integration of visual infor- mation rather than solely from perceptual encoding or language-based reasoning.
4. Demonstrating that targeted fine-tuning on diverse cognitive reasoning tasks can signifi- cantly enhance core cognitive abilities in VLMs, while also generally validating that im- provements on PAM and CVR tasks translate to broader gains in visual reasoning perfor- mance.
2 Related Works
The rapid development and success large transformer-base language models have inspired re- searchers extend these architectures and scaling frameworks to other multi-task domains. Many of these domains involve multi-modal data, including visual imagery, video, audio, real or virtual sensory inputs, and language. Integrating large language models (LLMs) directly into VLMs has shown major success across various multi-task vision-language applications [39 42 14, [2]]
Shortly after the release of early VLMs such GPT-4 Vision, Gemini, Flamingo, and the original LLaVA [TTO 8, 83, B2], researchers began assessing their vision-language capabilities. Commonly evaluated skills include image classification, captioning, scene description, and visual question an- swering (VQA). Datasets like IQA and VQAv2 focus on visual understanding grounded in general world knowledge [LS)
3 35. ES while others target specialized domains, testing models on scientific reasoning, mathematics, and document comprehension [TG A6, 17 47]
While some of these datasets involve reasoning about visual information, more complex reasoning and cognitive evaluations have since emerged to measure these features more extensively. Prominent examples include MMMU, MMBench, and MME [19-2T]. These benchmarks adopt comprehen- sive approach to evaluating reasoning and cognitive skills in state-of-the-art VLMs. MME and MMBench categorize their tasks into reasoning and perception, with both groups sampling diverse contextual domains. However, these benchmarks fail rigorously evaluate low-level cognitive abil- ities necessary for visual reasoning.
Beyond these broad vision-language benchmarks, recent studies have focused on testing VLMs on more narrow cognitive domains. BlindTTest [48] measures the abilities of VLMs to solve visual geo- metric tasks. VCog-Bent -CCo-BBncch [ takes inspiration from early nuuo--eeellommenall cognitive tests and evaluates VLMs on their ability to solve abstract logic puzzles and pattern recognition tasks. VisFac- tor from [S0] measures similar highly abstract visual reasoning abilities. Schulze Buschoff al. [_]]] focus their cognitive evaluations on intuitive physics, causal reasoning, and intuitive psychology. Vi- sual abductive reasoning is another related area of research[522 (53). focuses on the inference of hidden cause for a give observation, which differs from our focusing on logical reasoning. However,
by far the most commonly measured cognitive domain spatial reasoning [2X
BT].
While all of these studies highlight important weaknesses within specific domains, they fall short of providing comprehensive evaluation of VLMs' cognitive visual reasoning. Specifically, they (1) fail to assess core cognitive abilities and their interrelations; (2) focus mainly on single- or few-image tasks; and (3) rely heavily on abstract stimuli that are likely underrepresented in VLM training data. Our work addresses these weaknesses through a systematic evaluation of Perception, Memory, and Attention, their integration in complex visual reasoning tasks, and introduces methods improve VLM visual reasoning abilities overall.
3 Methods
3.1
Data Source
We utilize the iWISDM task environment [2>]] to generate all cognitive tasks and fine-tuning data in this study. This environment enables the procedural generation of an effectively limitless number of vasion-language decision decision-making tasks. In this study, we leverage iWISDM to generate tasks with varying levels of complexity, aligning with the requirements of each cognitive axis in the PAM dataset. These tasks range from simple single-object localization to more complex ones requiring logical reasoning and object comparisons across image sequences. The PAM dataset consists of tasks manually designed to isolate core cognitive abilities. The CVR dataset was generated using
3

---

iWISDM to produce complex, composite reasoning problems. Together they provide a wholistic set of tasks that allow us systematically analyzing the cognitive profiles of sfate-of-the-an VLMs.
We used ShapeNet objects [544], which include images of 3DDrendered everyday objects taken at vari- viewing angles. There are 8 object categories and 8 unique objects for each category, and objects are placed in one of four possible locations: top left, top right, bottom left, and bottom right. While not fully naturalistic, these familiar objects enable the measurement low-level cognitive abilities in a controlled setting, while largely avoiding the issue of limited training-data representation that affects more abstract stimuli such as simple geometric shapes used in prior work [99, 50].
3.2 Cognitive tasks for Perception, Attention, & Memory (PAM)
The PAM dataset includes three categories of tasks which measure individual cognitive abilities: Perception (Perc), Attention (Att), and Memory (Mem) (Figure
Perception. The Perception tasks assess model's immediate access to specific visual object properties according to the task instruction. These tasks contain one or more object frames and require the agent to be able to identify visual object properties, such as spatial location.
Attention. The Attention tasks assess a model's ability select the task-relevant ob- ject from multiple distractors within individual image frames. There are two variants of attentionnbassd tasks: (1) Spatial attention tasks, where the target is specified by its lo- cation (c.g. "top-right"); (2) Feature attention tasks where the target is specified by its category (e.g. "chair"). In both cases, the model must ignore irrelevant objects and report or compare the cued object's property.
Memory. The Memory tasks assess model's ability retain and recall visual object information across irrelevant image inputs. Specifically, each trial contains one or more object frames followed by blank distractor frames. The agent must encode the target's properties on the initial frame and accurately report them when prompted after the interrup- tion.
For each type of task. there are two task variants: Report (R) and Compare (C). These refer to whether the task requires the agent to report an object property or requires the agent to compare two objects by their properties. The type of property which is required to report or compare is either object location (Loc) or object category (Cat). See Figure II and Appendix Figures M 5 for example task trials.
To probe how VLMs' PAM scores relate to their more general visual reasoning capabilities, we ad- ditionally evaluate each model on a of composite visual reasoning tasks (CVR) that each involve various combinations of the different cognitive abilities. The CVR tasks were randomly generated using the iWISDM AutoTask framework. The three levels of complexity. Low (L), Medium (M), and High (H), were generated following the AutoTask parameters outlined in Lei et al. [22] and can be found in Appendix Table A45 Altogether, we tested each model on 22 vision- langua-an tasks (see Appendix Table A.4.3 for the full list).
3.3 Vision Language Models
We tested seven different vision language models: Intern [55], VLaVa--OnViision77 [566], MiniCPM- 2-MM-V-2.6-0 6B [57]], Qwen2.5-VL-7B [[=]], GPT-4o-Mini, and GPT-4o [IO].. These mod- els were chosen as a representative state-of-the-art set of open- (MiniCPM-V. Intern VI.2.5, LLaVe-One Vision, and Qwen2.5-VL) and proprictary (GPT-40 and GPT-4o-Mini) model series. We focused on evaluating smaller open-source models, they often achieve performance comparable to their larger counterparts while offering greater computational efficiency and broader practical us- ability. However, to gain more insight into whether our evaluations open-source models revealed properties that were due to their smaller size rather than other factors, we performed further tests on Qwen2.5-VL-72B, the largest size of this model.
GPT-4o and GPT-4o-Mini were evaluated with the official API. All variations of Qwen2.5-VI and LLL Vision were hosted with llama-factory (58] and evaluated with the OpenAl-style API. The Intern VI.2.5 and MiniCPM-V 2.6 were deployed using Hugging Face Transformers. All mod- els were evaluated with (near- aidentica task prompt template: the prompt describes the tasks and

---

possible answers, includes the trial's instructions and images, and suggests Chain-of Thought rea- soning (see Appendix Figure A.TII for the full prompt and OpenAI API code). After the VLMs are run on the benchmark task, the responses are passed to Dwen2.5-72B L.M,, which is prompted to extract the final answers given a list of possible answers. From these final answers, accuracy scores are calculated for all benchmarks. The GPT--o-Mini and GPT-4o results were collected using gpt- snapshots provided by OpenAl.
and gpt-4o-2024-055
3.4 Decoupling Vision & Text via Captioning
Aside from GPT-4o, our findings show notable disparities in cognitive performance between open- source models and humans on visual reasoning tasks. To address this, we performed a set of experi- ments designed enhance the cognitive and visual reasoning abilities of Qwwen.55VLL-7B via vision-text decoupling. Implemented through prompt modifications, we vary the required processing of visual information as well reasoning load through three methods:
PC (Pre-captioned): All images in the task prompt are replaced with ground truth captions containing both category and location object information. This method allows us to confirm whether any visual reasoning weaknesses on our datasets result from the LLM not being able to sufficiently understand the semantics of the prompt instructions and questions. !
SC (Self-captioning): In separate conversations, models are instructed to caption an image with information on object category and location. These model self-captions are then used to replace images within given task prompt. This method is related to the decoupling method from [5%] however, unlike the Prism framework, our method does not rely on a separate LLM to perform the text-only reasoning.
Instead of simply replacing images with model self- captions, we interleave the captions between the images such that all task images are fol- lowed by their corresponding self-caption.
SC-I wwwpraappiiiiinnnnnnreeereeeeeee
Qwen2.5-VL-7B was selected as the primary VLM for these experiments due to its substantial performance gap relative to both GPT-4o and humans, while also holding its status as the most recent open-source model in our evaluation set. To ensure the generality of our findings, we also applied the key vision-text decoupling experiments to GPT-4o and to the larger 72B size of Qwen2.5-VV VL.
Example prompts and Python code for our captioning methods can be found in Appendix A.
4 Results
4.1
Cognitive Evaluation of VLMS with PAM Dataset
We first evaluated selected VLMs and humans (see Section A. for details) on the PAM dataset, split by Location and Category subtask types. The results listed in Table 1I present the average accuracies on each task group (see Appendix Table A.4.I for more granular performance scores). Overall, GPT-40 performed best, with scores nearing human levels, while GPT-4o-Mini closely followed, matching human levels on category tasks. Between the four compact open-source models, performances were extremely dependent on the cognitive axis being probed. Both spercsource and proprietary models showed the pre-established weakness for spatial-based visual reasoning, with open-source models and GPT-4o-Mini expressing large performance gaps between location and category task variants.
In addition to task accuracy, we recorded human response times to provide context on the perceived difficulty of these tasks (see Appendix Table A.2.I).. The response times clearly correlate with task complexity; for instance, the average time for a high-complesy more than ten times longer than for simple perception task (7.39s). Interestingly, humans con- sistently took longer to solve category-based tasks than their location-based counterparts, a pattern that contrasts with the performance of most VI.Ms, which struggle more with spatial reasoning. difference likely reflecting the abundance of object category based task training data in comparison to spatial task data.
category CVR task (77.21s) was
"The PC method's ground ground-truth captions are made from the iWISDM task trial meta-data.
5

---

Table 1: Average PAM and CVR scores across all tasks for various open-source models, closed- source models, and human performance. Scores are presented as mean percentage accuracy (#: standard deviation). Best performance for open-source models for each task is bolded. Best perfor- mance overall for each task underlined.
Task
LLaVa
MiniCPMV InternVL. Qwen-7B
4o-Mini
40
Human
Percep. (Cat)
71.00=5.11
82.67=4..5
72.67=5.02
83.67#11118
89.00 90.33=3.36
93.75=#5555
Percep. (Loc)
67.00-5.29
39.00-1549
75.00
44.33 57.67 15.56 89.00
97.50 1:398
Feature Attn.
62.00
54.30
64.67
55.85
71.85
87.19
98. 75 13.27
Spatial Attn.
54.52-2.65
58.67#2.62
60.89
57.484263
68.81 1:2.47 75,93= =2222
98.75 3.227
Memory (Cat)
66.83=1.51
73.68=1.41
61.20*1.56
78.59+1.31
88.96=1.00
91.47=0.89
96.88 9 2.88
Memory (Loc)
50.83-1.68
39.11
42.22 42..54116
57.28 83,47=1.21
96.25 96.25+3.110 1.
CVR-Cat-L.
46.00=7.88
62.00
52.00 01.30
91.33=1.50
82.50
CVR-Loc-L.
66.00-7.55 50.00
46.00- 56.00 51.33 3,300
66.00
92.50
CVR-Cat-M
44.00=7.85
49.33
"2,00 72.00 96.67=3.u77
95.00 7.56
CVR-Loc-M
88.00=7.82 59.33
49.003
51.33 3799
=.44
70.00=11..6
CVR-Cat-H
24.00=4.81
37.00-45.43
39.333
63.004444
83.67
72.50
CVR-Loc-H
20.00 44511
31.00-15.21
36.33
36.33
29.33 39.67
64.67
5.38
75.00 133.00
4.1.1 VLM Perception
Consistent with prior literature, our perception tasks reveal that VLMMs much better object cat- egorization than they are at object localization. GPT-4o, representing the top proprietary VLM,s, had the smallest gap between these subtask types. However, its miniature variant (GPT-4o-Ming) as well as all tested open-source models, showed significant weaknesses on tasks requiring the perception of spatial object information. Interestingly, taking look at the granular results of Table A4T we see significant variations in performance between Perception comparison (C) and report (R) task types. For example, LLa Vision achieves 88% accuracy and exceeds humans on Pere-Loc-R, but achieves only 46% on Perc-Loc-C. While an opposite bias can be seen for other models and for category variants. These results reveal that VLMs not only have pervasive perception limitations but also harbor granular biases that seem to vary greatly depending on specific task requirements and structure. The fact that many VLMS find report tasks more difficult than comparison tasks is surprising, and as shown below, these peculiar biases also appear to be prevalent in VLM Attention and Memory abilities.
4.1.2 VLM Attention
The Attention tasks proved more of a challenge for the proprietary models. While still the best model tested, here we observed GPT-4o's 's first major gap with human performance. Specifically, GPT-4o struggles with attending to task-relevant objects while ignoring irrelevant ones. This was especially true for spatial attention, where relevant objects are specified by their location. All open-source models performed far below human levels. We had expected the differences between Feature and Spatial Attention performances be smaller than those between the Location and Category variants of the other task types. This is because both of these Attention task variants require simultaneous understanding of location and category. contrast to GPT-4o, the compact models largely met that expectation of similar performance between Feature and Spatial Attention task variants.
4.1.3 VLM Memory
Finally, for Memory, most models perform similarly to their evaluation on Perception, suggesting robustness to additional irrelevant frames. GPT-4o outperforms all models once again, reaching human-level accuracy on the Category variant. However, like Spatial Attention, location- based Memory tasks revealed another gap remaining between humans and GPT-4o. GPT-4o-Minn follows a similar pattern, with exceptional category- Memory task performance and weak location- based Memory task performance. Similar biases are present for all other models. A granular look with Appendix Table A.4.1 also reveals GPT-40 struggles more with Memory comparison tasks when the delay frames contain irrelevant objects.

---

Average Accuracyy (CVR) (%6
MancPHay
LLaVa-OV
Average Accuracy ayy (CVR) (9)
MeiicW--/2. 2.6
LLava-OV
Swenng 5-V
Dwwend 5-A
Owend 5-1
75
Oweeng 5-L ILoRA 3kI
75
Owen2 5AL (LORA 1et
75
Owen2: 5 (LORA 140
Qwer2 5-V (LORA 101)
Owen2 5- (LORA 15k)
Owen2.5 5. (LIRA 19dd
GPT Ninl
Swerg 3-1 (LoRA 100c)
Oween 5-1 dietA 100k) do Mini
Average Accuracy
Owend 5-t (LARA 180c]
4F
(R
-5-
25
25
15
35 75
35
75
15
55
75
Average Accuracy (Perception) (%)
Average Accuracy (Attention) (%)
Average Accuracy (Memory) (%)
Figure 2: Scatter plots comparing average PAM task performance against average CVR task per- formance across all models. Each point represents different model or Qwen2.5-VL-7B LoRA versions. The x-axis shows the average accuracy on the specified PAM task category (averaging Loc
and Cat), and the y-axis shows the CVR accuracy score.
4.2 Evaluating VLMS Composite Visual Reasoning
With their core cognitive abilities established, we turned to assessing how well the models perform composite visual reasoning (CVR) tasks. For this, we tested all models on the set of procedurally generated CVR tasks of varying levels of complexity (Table M].. We find that GPT-4o was consis- tently the strongest model across all complexities, even exceeding humans on most tasks. GPT- 4o-Mini managed to perform near this level on the low-complexity category variant. However, significant gap emerged all other CVR tasks. The open-source models demonstrate pronounced drops in performance with increasing complexity of the tasks. Interestingly, GPT-4o seems to be better at solving CVR tasks that have Medium-leve complexity than Low-level complexity. This difference could be due to CoT bias towards th-then-else operations, which are excluded from the Low complexity task generation. However, a detailed analysis inspecting the outputs of GPT-4o across these two complexities would be needed confirm this.
Relative to the PAM evaluation, GPT-4o performed much worse on the location-basse variants com- pared to their category-based counterparts. This may stem from small performance differences in PAM tasks compounding within the more demanding CVR setting, leading to larger gaps. The result highlights the importance of addressing even minor biases in core cognitive abilities.
4.2.1 Relationship Between PAM & CVR
Intuitively, models with stronger memory, attention, and perception abilities should also perform better on tasks that require combination of these skills. Since our CVR tasks designed to engage exactly these abilities in combination, we plot each model's 's PAM performance against its CVR performance, shown in Figure D. All performances on all three core cognitive axes showed strong correlation to CVR task performance. Memory task performance stands out with a correlation of 0.94. This is likely due to the abundance of Memory comparison-basse subtasks (i.e comparing two objects with irrelevant images in between), contained within CVR tasks. Furthermore, models' PAM and CVR accuracies are significantly correlated their performance on widely used benchmarks such as MMMU-Pro (Figures I.).I,, which further validates the effectiveness of the evaluations presented here.
4.3
What It Takes for VLM to Reason
Our results reveal substantial gaps in cognitive performance between open-source models and hu- mans on visual reasoning tasks. In this section, we explore ways to enhance the core reasoning abilities of VLMs. We start by testing series of prompt modifications to assess how different levels of vision-text decoupling affect Qwen2.5-VL--7B. These experiments introduce a simple yet effective strategy to boost visual reasoning while also highlighting key architectural bottlenecks. We then evaluate the impact supervised LoRA fine-tuning on Qwen2.5-VIL-7. using a separate set of CVR tasks.
7

---

4.3.1 Decoupling Vision & Text via Captioning
Table PAM and CVR performance of Qwen2.5-VL-7B using different captioning methods. The Base column shows absolute percentage accuracies (# standard deviation). Subsequent columns show the percentage change in accuracy from the Base method (# standard deviation). Positive changes are shown in green, and negative changes in red. Asterisks denote statistical significance: p < 0.05, ** p < 0.01, *** p < 0.001.
Task
Base
SC
SC-1
PC
Perception (Cat)
83.67=#4.18
+1.33#4400
-6.1725.77
-2.00
Perception (Loc)
44.33=:5.59
+28.67425.00
+10.17#6.84
+55.00=#1.11
Feature Attention
55.85
+22.00*##:12.21
-1.7413.22
+22.224**.12.21
Spatial Attention
57.48=22.62
+18.15#*nt22.29
-1.70=#3.24
+8.52=2.52
Memory (Cat)
78.59-t1.3
+4.72*m==1.19
-.227MM=
+6.27*oH
Memory (Loc)
1.62
+26.89
+17.07464t11.96 1.96
+47.39
CVR-Cat-L
60.00+7.75
+2.67+7.65
+1.00+9.39
+1.33=7.70
CVR-Loc-L
56.00-4.885
+8.6747.56
+0.00
+12.6747.34
CVR-Cat-M
54.00
+3.3337.82
+7.00
+12.6747.46
CVR-Loc-M
49.33#47.90
+10.00#7.77
+2.6749.61
+10.67
CVR-Cat-H
39.33#5.49
+6.00
+6.67#6.84
+13.00
CVR-Loc-H
29.33
+17.33##5.61
+7.1746.61
+32.00#15.44
Considering the well-established reasoning abilities of LLMs, used self-captioning and pre- captioning methods to investigate the source of poor visual reasoning in VLMs through vision-text decoupling. The results for these experiments are displayed in Table D: for fully granular look, please see Appendix Table A.4.2. We also evaluated one-shot prompting, however this approach did yield significant performance boost compared other methods, finding that aligns well with previous literature [60 61]]
Our pre-captioning method led to the largest and most consistent performance gains across the PAM tasks, with these performance gains also reflected in CVR tasks. Most importantly, the SC method also led significant improvements all task sets across PAM and CVR datasets.
The PC results suggest that the underlying language model components of Qwen2.5-VL-7BB are capable of interpreting the semantics of cognitive task instructions and performing the required rea- soning. Replacing images with self-generated captions (SC) of those same images proved to a simple yet effective strategy for enhancing (wen2.5-VL-7BB's cognitive performance. The compa- rable performance between PC and SC indicates that the model can caption least single images with sufficient accuracy to support PAM and CVR tasks. Therefore, any performance differences between PC and SC are likely attributable to the core language model's ability to translate visual tokens into accurate captions.
Notably, the largest performance gains by far for the SC method compared to the baseline were for location tasks. Across both PAM and CVR results, applying SC to location-only tasks saw an average accuracy improvement of 18.31% 31% compared 3.54% for category tasks. This stark differ- ence suggests that the bottleneck of VLM spatial reasoning may stem from inadequate CoT training. Two key observations support this: (1) Qwen2.5-VL-T VL-7BB can accurately describe object locations from visual inputs, and use those descriptions to answer location-based Perception questions, and (2) Qwen2.5-VV V-BBBBB fails at the same questions when no text-based location information is provided. limitation in current CoT strategies, for single-image
These simple observations point to spatial reasoning, and likely explain the near-chance performance on multi-image CVR and PAM location tasks. By contrast, it is unclear why this bottleneck affects category tasks less. A likely reason is the abundance of object recognition data in VLM training.
To assess the broader applicability of our self-captioning (SC) method for improving VLM perfor- mance, we conducted additional experiments on GPT-4o and the largest variant of Qwen2.5-VI The results, presented in Table A.5. and A.5. demonstrate that the effectiveness of the SC method extends beyond Qwen2.5-VI.-7B. These findings further confirm our CoT diagnosis, showing that comparable improvements can be gained in larger and SOTA proprictary VLMs.

---

Finally, we used the SC-I method on Qwen2.5-VL-7B to test the effect of including images in the model's input on its performance. In this setup, the model is provided with its self-caption descrip- tions interleaved between the corresponding images. This method saw significant improvements on location-only PAM tasks and almost all CVR tasks. However, on category-only PAM tasks, the ad- dition of images decreased performance relative to the baseline. Furthermore, SC-I improvements were significantly smaller than those of the image-free SC method. These results suggest the inclu- sion of images alone leads to diminished reasoning abilities, likely resulting from attention capacity (see Attention Analysis in Appendix A or interference issues. These results align with recent mechanistic interpretability findings from [2]] that show attention limitations a contributing fac- tor to spatial reasoning weaknesses in VLMs.
4.3.2 Fine-tuning VLMs
Table 3: PAM and CVR performance of Qwen2.5-VL-7B base model and fine-tuned variants using LoRA with different amounts of training data. The Base column shows absolute percentage accu- racies (# standard deviation). Subsequent columns show the percentage change in accuracy from the Base method (# standard deviation). Positive changes are shown green, and negative in red.
Asterisks denote statistical significance: p 0.05, ** 0.01, *** < 0.001.
Task
Base
LoRA 1k
LoRA 10k
LoRA 100k
Perception (Cat)
83.67=4.11
+2.00
+2.67=3.89
+6.00*
Perception (Loc)
44.33
+17.00
+26.33#5.13
+34.33#14.622
Feature Attention
55.85+2.65
+20.74***+2.20
+21.41*2x422.2
+16.96****t22.33
Spatial Attention
57.48+2.63
+24.59**ut2..05
+22.52
+17.93
Memory (Cat)
78.59:41.31
+2.35+a21.26
-3,20
+3.68*ad11.22
Memory (Loc)
42.22+11.61
+10.78*##*1.66
+22.72***+1..66
+27.69*8*#:11.00
CVR-Cat-L
60,00
-11.3347.90
-9.3347.00
+8.00
CVR-Loc-L
56.00
+0.00
-5.33
+10.00
CVR-Cat-M
54.00=7.88
+1.33+7.86
+6.67#7.72
+8.67=7.65
CVR-Loc-M
49.33#7.90
+7.33+7.83
-8.67=7.77
+4.67+7.88
CVR-Cat-H
39.33=55.49
+9.334#5.62
+26.67##5.39
+28.67*45.25
CVR-Loc-H
29.33#5.13
+9.67#5.19
436.67e25.33
+44.33=#11.66
We investigated whether fine-tuning Qwen2.5-VI.-7B on random CVR tasks leads to improvements on the core cognitive tasks of the PAM dataset (Perception, Attention, and Memory; Table B). To prove the generalization of the fine-tuning, we also performed LoRA fine-tuning 32B model on the same data (see Table II. We used supervised fine-tuning with LoRA on ran- domly generated CVR tasks (please see Section A for method details), We found that fine-tuning on even 1,000 trials yields large gains in performance on core cognitive tasks, and the performance gains further increase with additional data. Similar to our method of self-captioning fine-tuning ap- pears to disproportionately improve location-basee PAM task performance. Unsurprisingly. training on 100,000 held-out CVR task trials improved CVR task performance the most.
Qwen2.5-VL-
While these results are promising, standard LoRA methods are known to be prone to overfitting [62-64]] To mitigate this risk and make sure no severe overfitting has occurred in the Qwen2.5-VL. 7B model during fine-tuning, we experimented with several dropout rates. Our training runs with lower dropout ratios (0.0 and 0.1) experienced overfitting, resulting in rise in the validation loss during training. Consequently, we selected a dropout rate of 0.2 for our final models. To further assess generalization, we benchmark the LoRA fine-tuned models along with the base model on MMBench, MMMU-Pro and VQAv2. Table A A shows that supervised LoRA fine-tuning on CVR tasks can provide modest improvements to MMBench and VQAv2. However, performance on MMMU-Pro slightly decreases when fine-tuned on smaller dataset sizes.
5 Discussion
This work provides detailed analysis of the cognitive capabilities of modern VLMs by systemat- ically evaluating them along the core axes of Perception, Attention, and Memory, using the PAM

---

dataset, as well as on combinations of these abilities via the CVR dataset. Our findings reveal dis- tinct cognitive profiles: SOTA models like GPT-4o demonstrate strong perceptual abilities for object categories and reasonable memory, but exhibit significant weaknesses in processing spatial informa- tion and performing comparative judgments across frames or objects. The spatial reasoning deficit in particular is pervasive issue across most VLMM tested.
Our analyses strongly suggest that primary bottleneck lies not in the fundamental reasoning ca- pacity of the underlying LLM, nor purely in low-level visual perception, but rather in the effective integration of visual features into the reasoning process, Models often possess the visual information (as shown by self- captionni success) but struggle to utilize it correctly when solving tasks directly from images. limitation that a sufficiently robust CoT strategy should be able overcome. These same analyses provided simple rision-text decoupling method for which this bottleneck can be easily mitigated. We demonstrate that this method leads to consistent visual reasoning performance gains, even for a SOTA VLM that already rivals human-level performance.
Fine-tuning VLMs on a small dataset of random visual reasoning tasks also led substantial im- provements along all cognitive axes while yielding slight gains on out-of-distribution benchmarks. We believe it is likely that more sophisticated methods of fine-tuning, such as Reinforcement Learn- ing with CoT reward [6S], could lead to improved generalization. This hypothesis is supported by high correlations when comparing model PAM and CVR accuracy against established benchmarks like MMMU-Pro and VQAv2, which can be found in Figures A. and A 8. This is an important finding that further validates our evaluation datasets and approach.
6 Limitations
Our study has several limitations that future work should address: 1) The stimulus set used to gener- ate test samples consisted of synthetic objects from only eight categories everyday items. Future evaluations could incorporate wider variety of objects, including more diverse and natural visual categories. 2) Visual frames in our benchmarks featured uniformly colored backgrounds, consistent with practices in cognitive science and neuroscience. However, future studies could adopt more naturalistic settings where objects are embedded in complex scenes. Leveraging generative image models may offer promising avenue for creating such stimuli [66 18]. 3) For feasibility, we re- stricted our fine-tuning experiments to the Qwen2.5-VL model and used up to 100k trials. Expand- ing these experiments to larger models and datasets could yield additional insights. Generative task environments, such as those proposed by Lei et al. [22], provide a convenient framework for scaling data to arbitrarily large sizes. 4) Our fine-tuming experiments consisted of tasks with random level of complexity. While fine-tuning on these tasks already resulted in improvements on the cognitive benchmarks, expect that more balanced data distribution would be even more helpful. Future studies could perform fine-tuning on more complexity-balancee (SC) method, while effective at improving performance, introduces computational trade-off. Our analysis shows that this method has roughly double the inference runtime compared to the standard direct-inpus approach, which could be critical factor in real- world applications. A breakdown of this analysis is provided in Appendix A.
task datasets. 5) The self-captioning
Acknowledgments
This research was supported by the Healthy-Braans--eallle Heilt startup supplement grant, the NSERC Discovery grant RGPIN-2021-00033, and CIHR Project Grant PJT-191957. P.B. was sup- ported by FRQ-S Research Scholars Junior grant 3 310924, and the William Dawson Scholar award. M.P. was supported by the UNIQUE Masters and PhD Fellowships, and the Stichting Formation Award. All analyses were executed using resources provided by the Digital Research Alliance of Canada (Compute Canada) and funding from Canada Foundation for Innovation project number 42730. Z.W. was supported by China Scholarship Council Grant 202506070003. L.G was supported by the UNIQUE MSc Excellence Scholarship 2024-2025.
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar-
10

---

wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child. Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, July 2020. doi: 10.48550/ 11Xi..0005.44155. URL
14165, arXiv:2005. [cs].
[2] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe V. Le, and Denny Zhou. Thoughh Prompting Elicits Reasoning in Large Lan- guage Models, Advances in Neural Information Processing Systems, 35:24824-24823 Decem- ber 2022. URL
www.eeedddiinnnnnnnn...e....eeee cc/ wwwpppprpp.ppepeeppeeeeeppeeeeee files /wwes/paper/222220 paper/2022/hash) 1att////aaaa.. http:///aaaaaaa----aatat-a----oa tonttrrre.ttttteeeee..... htm
[3] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. hRLps: www.ooggg//bbbbeooooooo 2205. 11916.
[4] Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1226 1541, September 2023. ISSN 2397-
3374. doi: 10.1038/s41562-02 -22-622-2--------- URL https: : //www. wwwuraatuee.... com/ articles/ Publisher: Nature Publishing Group.
[5] DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang. Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li. Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. I.. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng arg/abs/ htgggabs/2412... 2412 19437,
Pan. Deepseck-v3 v3 technical report, 2024. URL /ttps://arxiit
[6] Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li. Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang. Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang. Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan
11

---

Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, and Tianming Liu. Evaluation of OpenAl ol: Opportunities and Challenges of AGI, September 2024. URI.ttp: http::/aaxivvorrt wrtp./////bbb/.... 18486. arXiv:2409.18448 [es].
[7] François Chollet. Openai o3 breakthrough high score on arc-agi-pub, December 2024. URL
org/ /log/
wwclog/gaiiiio oai -a---3---000 -0--033pppp- pu--pp--i-- breakthrough.
[8] Google. Introducing gemini: Our largest and most capable ai model. https://arxiv.. abs/2312.0 11805, 2023. Accessed: 2024-11-24.
[9] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved basclines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296-26306 June 2024.
[10] OpenAI. GPT-4o System Card, October 2024. URL http: httgabbb/410..12//1 arXiv:2410.21276 [cs].
[11] Pan Zhang. Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang. Linke Ouyang. Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang. Wei Li, Jingwen Li, Wenhai Wang. Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM- XComposer-2.55 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output, 2024. URL http::/arxi: org/ hbwpg/abb/2447.. 03320,
[12] MetaAI. The Llama Herd of Models, August 2024. URL (utp://arxiie www.ai.wgw...////////// 21783. arXiv:2407.21783.
[13] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https://arxix: www.oog.////.oooooooo.o.oo...e// http//bb/2200....o 0308.
[14] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jian- qiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https:://axx:: org/ abs/ /5022. 13923.
[15] Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[16] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Pro- cessing Systems (NeurIPS), 2022.
[17] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Doevqa: A dataset for vqa on document images, 2021. URL https://arxiie org/ ab8/2007.003003
[18] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. URL. https://arxiv org/ /bs/ 1902. 09506.
[19] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[20] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URL https: http://awww...oo//////3000 06281.
12

---

[21] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https://arxivv. org/abs/2306. 13394
[22] Xiaoxuan Lei, Lucas Gomez, Hao Yuan Bai, and Pouya Bashivan. IWISDM: Assessing in- struction following in multimodal models at scale, 2024. URL http://arxiv www.:org/aaao /bss/ 2406. 14343
[23] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? m investigating their struggle with spatial reasoning, 2023. URL https:/arxiv.rr//aan
2310. 19785
[24] Taylor Webb, Shanka Subhra Mondal, and Jonathan D. Cohen. Systematic
Visual Reasoning through Object-Centric Relational Abstraction.
Advances
in Neural Information Processing Systems, 36:72030-72043, December 2023.
URL
https://proceeddii htw.:ecceedinng................e wwwrinurrricoo cc/ ccppaer paper www.ffillesapapeee wttps//pppr//22//aeeeee 2023/hashhh hhmml
83cdcsars7/Sd
[25] Aishik Nagar, Shantanu Jaiswal, and Cheston Tan. Zero-shot visual reasoning by vision-
language models: Benchmarking and analysis. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2024. doi: 10.1 N0.11000/CNNN009992222222
[26] Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M. Frankland, Thomas L. Griffiths, Jonathan D. Cohen, and Taylor W. Webb. Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem, October 2024. URL http: org/ hbt//bb///00 2tab///11.... 00238. arXiv:2411 1.t::2211.00038 [cs].
[27] A Treisman and G Gelade. A feature-integration theory of attention. Cognitive Psycholo, 1980.
[28] Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vlms? an attention mechanism perspective on focus areas, 2025. URL https://arxiv www. /ba/
2173.
[29] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024. URL https://arxiie org/ aba/2406. 14852.
[30] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, Transactions of the Association for Computational Linguistiies, 11:635651, 2023. ISSN 2307-387X. doi: 10.1162/ tacl a a_00566. URL http://dx. http://wxxddo.ooro..o/ooo wttp:o////............... 1162/
00566,
[31] Fatemch Shiri, Xiao Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models, 2024. URL https: wwwpa/xwwioorog/baaae
[32] Jay Friedenberg, Gordon Silverman, and Michael J Spivey. Cognitive science: an introduction to the study of mind. Sage Publications, 2021.
[33] Richard P. Heitz, Nash Unsworth, and Randall W. Engle. Working Memory Capacity, Attention Control, and Fluid Intelligence, page 6178. SAGE Publications, Inc., 2005. doi: 10.4135/ 9781452233522 n5....9..........m URL http://dx doi org /13.//11115522222
.5.
[34] Timothy A. Salthouse and Jeffrey E. Pink. Why is working memory related to fluid intelli- gence? Psychonomic Bulletin & Review, 15(2):364-37, April 2008. ISSN 1531-5320. doi: 10.3758/PBR.15.2.33 URL https: wrw.:o.g/oooooo 3758///R 15. 15.2.364
[35] Zach Shipstead, Tyler L. Harrison, and Randall W. Engle. Working memory capacity and fluid intelligence: Maintenance and disengagement. Perspectives on Psychologicul Science, 1 1761797,779,, November 2016. ISSN 1745-6924. doi: 10.1177/7445911616666666 URL.
http://ds //dx doi. org/ 10. 100111771145591116655666
13

---

[36] David Marr. Vision: A Computational Investigation into the Human Representation and
Processing of Visual Information. The MIT Press, 07 2010. ISBN 9780262289610. doi: 10.755 (0.7551|/mipprsss/78822263 0121:/226621440000.....00 htt https::/doo.i https:://bb..rr//t http:/ooo///...///mmpppr...... 7551/mttrrsmmm/
#780262514620.000
[37] Alan Baddeley. Working memory. Science, 255(5044):556-559, 1992.
[38]
Michael Posner, Charles R Snyder, and Brian Davidson. Attention and the detection of signals. Journal of experimental psychology: General, 109(2):160, 1980.
[39] OpenAl. www. . www.aaatteeeciila.ooooorrrrrrrea - System-Card/
GPT-4V(ision)
System Card, 2023.
URL
https://
[40]
Anthropic.
Claude 3.5 Sonnet, 2024.
URL
Claude- sonnet..
[41]
Gemini Team Google, Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL
05530,
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
URL https://arxiv.org//Ba 2304 08485.
[43] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Scbastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022. URL attps://arxiv org/ abs/2204 14198.
[44] Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021. URL https://arxiv. abs///4. 12756.
[45] Jinggun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, and Can Huang. Mtvqa: Benchmarking multilingual text-centric visual question answering. 2024. URL attps://arzia org/ htte//bb/244055 2405. 11985.
[46] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical rea- soning of foundation models in visual contexts, 2024. https:://///xxit. org/ /ttp///rxii..orr/b////2/1 abs///1b//11. 02255
[47] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehen- sion on document images, 2021. URL https: htttps://arxiv.orpeee wttp://///aa......///// 11272.
[48] Pooyan Rahmanzadehgervi. Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind: Failing to translate detailed visual features into words, 2025.
URL attps://arxis org/ abs/ 2407 06581
[49] Xu Cao, Bolin Lai. Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M. Rehg. What the visual cognition gap between humans and multimodal 1lms?, 2024.
URL https: https://arxii //arxiv...vvv//// www./rgg//////ooooro/o.oo...eeee 2206 10424
[50] Jen-Tse Huang, Dasen Dai. Jen- Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, and Zhaopeng Tu. Visfactor: Benchmarking fundamental visual cognition in multimodal large language models, 2025. URI. htttttsss///xxiiv org/ abs/2502.
16435.
[51] Luca Schulze Buschoff, Elif Akata, Matthias Bethge, and Erie Schulz. Visual cognition in multimodal large language models. Nature Machine Intelligence, 7(1)96106, January 2025. ISSN 2522-5839. doi: 10. 101033844255--4440009..y URL http: http://ddx.ooo..o///// 0722560024-00663-
14

---

[52] Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction ofäsherlock holmes: A dataset forăvisual abductive reasoning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Gio- vanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, pages 558- 575, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20055
[53] Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual abductive reasoning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15565-15575, June 2022.
[54] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
[55] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiy- ong Wu, Huipeng Deng. Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL https: http:/aaiiiv....aaaa.... abs/ 2412105211
[56] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-One Vision: Easy Visual Task Transfer, 2024. URL
arxiv. org/ abs/2408, 03326.
[57] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng. Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. MiniCPM-V: A GPT-4V Level MILM on Your Phone, 2024. URL attp://arxinn wwwpxiii.oorraaa abs abs/ 2408. 01800,
[58] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo. Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http::/aaxii> org/ abs/ //2///2402 2403. 13372.
[59] Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang. Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms, 2024. URL https://aaxiie http://aaxxiv...gabbbba wttp:///ggaabbs/////0/ abs/ 14544.
[60] Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VL-ICL bench: The devil in the de- tails of multimodal in-context learning. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL
net/ http://rrmmmidpppp/ppte//
[61] Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. Beyond task per- formance: evaluating and reducing the flaws of large multimodal models with in-context- learning. In The Twelfth International Conference on Learning Representations, 2024. URL /topsr///perreiexx heth/forrmmmidcmmaaaaakbb
[62] Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei. Lora dropout as a sparsity regularizer for overfitting control, 2024. URL https://arxiv.or
abs/2404.092
[63] Yupeng Chang, Chenlu Guo, Yi Chang, and Yuan Wu. Lora-ggpo: Mitigating double descent in lora fine-tuning via pradient-guides perturbation optimization, 2025. URL https://aaaxx. 14538.
org/
[64] Yufei Chen, Wenhao Yu, Xin Jin, Caiming Xiong, Yining Shi, Shuangjun Wang, and Zhiting Hu. Bilora: bi-level optimization framework for overfit low-rank adaptation of large pre-trained models. arXiv preprint arXiv:2403. 2024.
15

---

[65] AL. Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, 7. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan. Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https://arxis: org/ abs/ 12948.
[66] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxiv. 10752, arXiv:21 [cs].
org/
[67] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Imane Image Generation, February 2021. URL http:
/ //arxiv. wwtgg/bs///i/ 12092, arXiv:2102. [cs].
[68] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour. Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text- to-Image Diffusion Models with Deep Language Understanding, May 2022. URL http:// arxiv www.vvorr//ooo abs/ 11487. arXiv:2205.1 1487 [cs].
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Empirical results in support of the claims are provided in the Results section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
16

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations of our work extensively discussed in the Limitations section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
17

---

4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details of all analyses are clearly explained in the methods section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper does not provide open access to the code or data. However, includes sufficiently detailed methodological descriptions in the Methods section and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
18

---

reproducible, they The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
//nips.
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The data generation procedure for both train and validation are clearly plained the Methods section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Each modeling experiment was performed three times and all results are re- ported with standard deviation. However the number of samples was insufficient per- form statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
19

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The hardware used for training the model is specified in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research fully complies with the Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We extensively discuss the importance of evaluating VLMs and the relevance our contribution the Introduction section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
20

---

If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No new data or model is released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
allow for controlled use of the model, for example by re-
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in our research are properly cited in the manuscript.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- thcode.
age should be provided. For popular datasets, paper phcode com/ dataseta has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
21

---

Researchers should communicate the details of the dataset/codeemmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Experiment details and screenshots depicting the provided instructions as well as trial example can be found in Section A21l
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The study was carried out under an existing IRB approved protocol.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
22

---

Please refer to our LLM policy http:://hoearppsoccoooooorrrooo
for what should or should not be described.
23

---

A Appendix
A.1
Task Examples
Perception Category
Perception Location
Report
observe object
frame 1, category object Answer:
observe object in frame 1, location of object 1? Answer: bottom right
Cumpare
Compan
observe object frame 1,
observe object 1 frame 1,
observe object 2 frame 2,
observe object 2 in frame 2, location of object equals location of object 1?
category of object equals category object Answer: false
Answer:
Figure A.1.1: Example trials from Perception Category (Pere-Cate Pare-Cat-Re & Perc- Parc-Cat-CC and Localization (Pere-Loc-R Pere-Loc-O CCc tasks. Each task consists of two variations: Report where the agent is tasked with reporting the object's property; and Compare where the agent is tasked with comparing that property between two objects on separate frames.
Feature Attention
Spatial Attention
Report Diemattnn
Ropont Dionseoo
observe object frame category: planes,
observe object frame with location: top right,
location object 1?
category of object 1?
Answer: bottom left
Answer:
Compon Diuemcttoss
Compore
observe object in frame with Incarion: top left,
observe object in frame with category: cars, observe object frame with category: boats, location of object equals location of object 17 Answer: false
observe object in frame with location: bottom left, category of object equals category object Answer: false
Figure A.1.2: Example trials from Spatial (Att-Spa-R & Att-Spa-C) and Feature Attention (Att-Feat-
Att-Feat--) tasks. Each task consists two variations: Report, where the agent is tasked with reporting the object's property, and Compare, where the agent is tasked with comparing that property between two objects on separate frames.

---

Memory w/ Object Distractions Location
observe
object frame observe objert frame 3, observe objject in frame observe objectt frome 5, location of objject 1?
left
olserve oljject frame 1, olserve uljiect in frame observe objeett frame object frame object 5, location object 5 equals location of object Answer:
Memory w/ Object Distractions Category
Report 2 Distractions
observe object in frame 1, observe object 2 in frame 2,
observe object 3 in frame category of object 1? Answer: cars
Compare Datrctiin
observe object 1 in frame 1, observe object in frame 2,
observe object 3 in frame 3, location of object 3 equals location of object 1?
Answer: true
Figure A.1.3: Example trials from Memory with Distractors Category (Mem-Dis-Cat-R & Mem- Dis-Cat-C) and Memory with Distractors Location (Mem-Dis-Loc-- Mem-Dis-Loc-C) Memory tasks. Each task consists of two variations: Report, where the agent is tasked with reporting the object's property, and Compare, where the agent is tasked with comparing that property between two objects separate frames.
25

---

Memory Category
Regert Delege
observe object
frame, delay, category object 1?
Answer: couches
Compun Dalee
observe object frame delay, observe object 2 frame category of object equals category of object 2? Answer: false
Memory Location
Repont Dalay
observe object in frame location of object Answer: top left
Compers Salay
observe object frame 1, delay, observe object 2 frame 5,
location of object equals location of object 2?
Answer: true
Figure A.1.4: Example trials from Memory Category (Mem-Cat-R & Mem-Cat-C) and Location (Mem-Loc-R & Mem-Loc-CC tasks. Each task consists of two variations: Report where the agent is tasked with reporting the object's property, and Compare where the agent is tasked with comparing that property between two objects separate frames.
26

---

Composite Beasuming (Caeegory)
deay, object delax, obseree object objoct
catepoy objoct equals category objoct coregory otject equals canapogy otjeeett
Asswer falie
delax dijes oboerve object 2 objoc: саисроо cacopory of objjct equals casegory object
object
observe category categiry
caregory ohjen repals estegery
dday ebserve dbjec in fatte aiserve object in Eaaee observe objec: is fiune 4, obazve object in frunc 5, dbserve okieat frumc
chernce object framee 7. sterrw onject 7 cheerw objec trmme rargggr of objer equals rateggry of mejert estegory sbject 2 expalll bears, then category of akyect espale eategory skject eveggey olject e-egul cueggey congray olject categery object
Answer: ciee
Compostte
obeerve dbject In Dame 1, observe objec: 2 Baane dony, ebserse object 3 is frome 5, deserv-olject la franc 6,
location of object 4 not equale Jocation of object 3 location of otjeen 2 equals locaiion onject Answen fale
observe obiect in frazze 1, observe objoa in fame 2 observe object 3 Eame delag obaerve obiect 4 frunc 5, observe obsect framc
delay, whourv objaet é loceiin viject repils loeation of oljeet then luzeiun aljer equsk top vise object not aqual top location cljeet equads sbjecn
Answer: Filee
abeence object in frame 1, obeerve eijer 3 framte 2, cherrv otject 3 trame observe objen 4 is fimm 4, delyy, onsemv ofject 5 frome churroo abjecr 6 fimm 8, abourv object 7 fome i'loorito of akyeet is segonic lccation olgere 5 Incotio of chyect not equals iscerion of anjeer then lsceion of objec 2 epule incernn of obyect 17 wler Incation of olrect Answer: fike
Figure A.1.5: Example trials from Composite Visual Reasoning Category CCVR-Ctt-Hil and Location (CVR-Loc-H/MiP tasks. These tasks were randomly generated into three sets of varying complexities: High (H), Medium (M), and Low (L).
27

---

A.2 Human Baseline Details
To establish human baselines, eight people were tasked with performing all 22 different task types across the PAM and CVR datasets. Each subject solved 5 trials of every task, providing total of 880 human accuracy data points. No compensation was given to subjects.
Welcomd!
you
und
xxp-b-oottom PLEAZE injepral tham thi orewr.
Your ubwerv the nead Intruction, and ruppont commityy to the queetion
imagee,
posdiiie displaedd the Please ONLY respond with one the provited poesitle
caloure Seadt HOUn comoo s0ec
callgary
hoood' bogin expoiment
Figure A.2.1: Screenshots of human subject instructions (left) and a task trial example (right).
Table A.2.1: Average human response time across all task types.
Task
Mean Response Time (s)
Perception (Cat)
7.39
Perception (Loc)
5.77#1.12
Feature Attention
11.34+1.78
Spatial Attention
10.79+1..9
Memory (Cat)
9.67411.0
Memory (Loc)
8. 8.
CVR-Cat-L
30.68-16.47
CVR-Loc-L
23.39 1.683
CVR-Cat-M
117.41
CVR-Loc-M
34.36+9974
CVR-Cat-H
77.21 25.90
CVR-Loc-H
41.44=10.81
A.3 Supervised VLM Fine-tuning on Composite Visual Reasoning
As an additional attempt improve visual reasoning, we performed fine-tuning experiments on Qwen2.5-VL-7B. The model was fine-tuned with a supervised objective to output only the correct final answer. This meant the model would likely learn to solve the tasks without the use of any CoT. This presents an interesting contrast to our captioning experiments. Qwen2.5 was selected as the VLM for these experiments for the same reasons provided in Section 544
To achieve this, we used iWISDM [22] to generate task sets of varying sizes: 100, 1,000, and 10,000 tasks. For each task, 10 trials generated, resulting training sets comprising 1, 1,000, 10,000, and 100,000 trials, respectively.
The CVR tasks generated for LoRA fine-tuning have similar AutoTask + to those used for the CVR evaluation tasks. However, while the distributions should similar, the fine-tuning tasks have lower bound of complexity avoid overfitting to complex instructions. It also important to note that there is no overlap between the training and CVR evaluation datasets.
generation parameters
We chose to keep the ViT vision encoder of Qwen2.5-VL-7B frozen during fine-tuning This deci- sion was based on prior studies showing that the vision encoder and vision projection components already capture sufficient visual information [[К]. However, the vision projector was unfrozen dur- ing fine-tuning as offers lightweight yet potentially impactful set of vision-text alignment weights to adapt. As such, the nonlinear MLP vision token projection in Qwen2.5-VL. VLB was unfrozen dur-
28

---

ing training along with the core LLM. All fine-tuning experiments were performed on 4 NVIDIA A5000 GPUs. Please see Appendix Table A.3.4 for training hyperparameter details.
Table A.3.1: The iWISDM AutoTask parameters used to generate the fine-tuning task sets.
# of allowed and/or operators in task
# of switch operators
# of trial frames
root operators
boolean operators
IsSame, And, Or, NotSame, GetLoc, GetCategory
IsSame, And, Or, NotSame
0-2
0-1
3-9
Table A.3.2: MMBench & MMMU-Pro benchmarking scores (%) before and after LoRA fine- tuning.
N fasks
(base)
100
1k
10k
N trials
(base)
1000
10k
100k
MMBench
82.85
83.41
83,63
83.30
MMMU-Pro
34.36
33.70
33.83
34.94
Table A.3.3:
Benchmarking scores for Qwen2.5-VI-7B and Qwen2.5-VL-72B on the VQAv2 dataset with different caption techniques and finetuning configurations.
7B Models
72B Models
Base
SC
LoRA 1k
LoRA 10k
LoRA 100k
Base
SC
Accuracy (%)
60.07
65.07
63.96
64.44
65.84
71.09
72.02
Table A.3.4: Overview of the hyperparameters for -0--5---V-7----strrue LoRA Fine-tuning.
N_tasks
100
1000
10000
N trials
1000
10000
100000
N. Tepoccs
10
Batch size
1
Gradient accum
32
Scheduler
Cosine
Peak_LR
4e-05
Warmup
7
70
Mixed _ereciiireeoonn
b
Optimizer
AdamW(0.01)
LoRA_rank
8
LoRA_ __alpha
16
LoRA dropout
0.2
LoRA targets
vision projector, LLM
29

---

A.4
Granular Results & Details
Table A.4.1: Average granular accuracy (%) comparison across all tasks and models. Scores
presented as mean percentage accuracy (#: standard deviation). LLaVa: VLaVa-OneVision-27
MiniCPM: MiniCPM-V-2.6-88; InterVLL IntermVL2.5-8B; Qwen 7B: Qwen2.5-VL-7B: Qwen
72B: Qwen2.5-VL-72B; 4o-Mini: GPT-4o-Mini; 4o: GPT-44;; human: Human subjects.
Task
LLaVa
MiniCPM IniemVL Qwen 7B Qwen 72B 4o-Mini
40
human
Mem-Cat-R Cat-
87,47
87.87#2.34
100.00 -..00....
Mem-Cat-C
80
99 99.60=2..52
97.50 97.50=6.22
Mem-Loe-R
96.40=1.35
95.00#7.66
Mem-Loe-C
87.33 +2.33
100.00 #4.38
Pere-Cat-R
6.
97.50
Perc-Cat-C
100.00000 90.00 #-9.55
Pere-Loc-R
93.333-4.00 95. 95.00#7.666
Perc-Loc-C
100.00=4.35
CVR-Cat-H
83.67 12.50
CVR-Loc-H
64.67 75.00=1:3.00
CVR-Cat-M
96.67+3.77 95.00#7.66
CVR-Loc-M
82.6776..44 70.00+133.68
CVR-Cat-L
94.33 82.50
CVR-Loc-L
66.00-
28.60
Att-Feat-R
97.50)26.22
",,.,
Att-Feat-C
82.67
100.00 100.00
Att-Spa-R
+..44
97.50 6.
Att-Spa-C
100.00 74.67
Mem-Dis Man--Di-Cat--R Cat-R
92.50 64
Mem- Mem-Dis-Cal-C
01.00 91.04=1.58
97. 97.50=#0.22
|4g_333ee8..aa..0000000000011111 63.6
+1.33
100.00 #4.38
77.63 42.22 90.00 +9.55
30

---

Table A.4.2: Average granular accuracy (%) comparison of Qwen2.5-VL-7B using different caption-
ing methods. Scores are presented mean percentage accuracy (# standard deviation).
Task
Base
SC
SC-I
PC
One Shot
Mem-Cate 76.53
80.00
69.60-1.022
85.87 2.49
78.53
Mem-Cat-C
82.13
91.0742.05
76.40
93.8741.78
86.80
Mem-Loc-R
33.20
72.00
63.40
97.47=1.15
32.27
Mem-Loc-C
52.13=23.57
69.73#3.28
59.20
96.00=1142
50.80+33.57
Perc- Pere-Cat-R
74.67-16.90
76.67=6.72
65.00+99.19
80.67-6.29
76.6746.72
Pere-Cat-C
92.67-4..25
93.33 1.009
90.00
82.67-6.044
90.00
Pere-Loc--R
33.33=7.446
73.33=7.01
58.00+9.50
99.33=1.78
27.33
Pere-Loc-C
55.33+7.86
72.67=7.07
51.00+9.61
99.33+1.78
44.00+7785
CVR-Cat-H
39.33=t5.49
45.33
46.00=t6.84
52.33
31.33
CVR-Loc-H
29.33=t5.13
46.67=5.61
36.50
61.33=55.88
30.00
C
54.00+#7.88
57.33=7.82
61.00+9939
66.67#7.16
58.00
CVR-Loc-M
49.33-1.900
59.33 7.77
52.00
60.00
44.00
CVR-Cat-L
60.00
62.67=7.65
61.00
61.33=7.77
61.33
CVR-Loc-L
56.00
64.67=7.50
56.00
68.67#7.84
58.00
Att-Feat-R
58.67:44.53
80.44+3.66
46.00
99.78=00.00
58.89
Att-Feat-C
54.44 1:3.55
76.56 2.
58.17433.9
67.22443.06 4:306
67.22
54.00
Att-Spa-R
49.33++1.60
72.22=1.13
40.67+5.62
83.78=3.40
51.11 +1160
Att-Spa-C
61.56++3.7
77.33+2.73
63.33
57.11
61.89+33117
Mem- Mem-Dis-Can
78.56++2.08
79.56 2..63
66.00
75.44+2.81
77.67#2.72
Mem- Mem-Dis-Cate Dis-Cat-C
77.78++2.22
83.33#1.99
78.56+2.68
85.56+1.88
70.89+22.42
Mem-Dis-Loce
34.1 +4.11++3.09
75.56+2.80
65.67+33799
94.44+1.51
29.56+2.98
Mem-Dis-Loc-C
51.41 1:2.66
70.81 12242
59.89 13.20
89.93 11.61
52.22
Table A.4.3: Cognitive benchmark tasks, their abbreviations, and chance levels.
Chance
Task
Abbreviation
Level (%
Perception Category Report
Perc-Cat-R
13
Perception Category Compare
Perc-Cat-C
Perception Location Report
Pere-Loc
25
Perception Location Compare
Perc-Loc-C
50
Feature Attention Report
Att-Feat-R
25
Feature Attention Compare
Att-Feat-C
50
Spatial Attention Report
Att-Spa-R
13
Spatial Attention Compare
Att-Spa-C
50
Memory Category Report
Mem-Cat-R
Memory Category Compare
Mem-Cat-C
50
Memory Location Report
Mem-Loc-R
25
Memory Location Compare
Mem-Loc-C
50
Memory with Distractors Category Report
Mem-Dis-Cau
Memory with Distractors Category Compare
Mem-Dis-Cat-C
50
Memory with Distractors Location Report
Mem-Dis-Loc-P
Memory with Distractors Location Compare
Mem-Dis-Loo--C
50
Composite Visual Reasoning Category High Composite Visual Reasoning Location High
CVR-Cat-H
CVR-Loc-H
Composite Visual Reasoning Category Medium Composite Visual Reasoning Location Medium
CVR-Cat-M
CVR-Loc-M
50
Composite Visual Reasoning Category Low
CVR-Cat-L
50
Composite Visual Reasoning Location Low
CVR-Loc-L
50
31

---

Table A.4.4: The iWISDM AutoTask parameters used to generate the PAM tasks. No and/or opera-
tors, boolean operators and switch operators are allowed in these tasks.
Complexity
of trial frames of distractors root operators
Perception Category Report
1
GetCategory
Perception Category Compare
2
IsSame
Perception Location Report
1
GetLoc
Perception Location Compare
2
IsSame
Feature Attention Report
1
1-3
GetLoc
Feature Attention Compare
2
1-7
IsSame
Spatial Attention Report
1
1-3
GetCategory
Spatial Attention Compare
2
1-7
IsSame
Memory Category Report
2-6
GetCategory
Memory Category Compare
3-7
IsSame
Memory Location Report
2-6
GetLoc
Memory Location Compare
3-7
IsSame
Memory with Distractors Category Report
2-7
1-6
GetCategory
Memory with Distractors Category Compare
3-11
1-9
IsSame
Memory with Distractors Location Report
2-7
1-6
GetLoc
Memory with Distractors Location Compare
3-11
1-9
IsSame
32

---

Table A.4.5:
The iWISDM AutoTask parameters used to generate the CVR tasks. The Cat and Loc task variants are obtained by allowing object feature selection of either Category or Location.
# of allowed and/or
# of switch # of trial # of operators frames distractors
root operators
boolean operators
Complexity
operators in task
IsSame, And, Or, NotSame IsSame, And, Or, NotSame IsSame, And, Or, NotSame, GetLoc, GetCategory IsSame, And, Or. NotSame, GetLoc. GetCategory
IsSame, And, Or, NotSame IsSame, And, Or, NotSame
Low
6
Medium
8
IsSame, And, Or, NotSame
High
1-2
9
IsSame, And, Or, NotSame
High-Distractor
1-2
12
A.5 Captioning Method Validation
Table A.5.1: PAM and CVR performance Qwen2.5-VL-T using different captioning methods. The Base column shows absolute percentage accuracies (# standard deviation). Subsequent columns show the percentage change from the Base method. Positive changes are shown green, negative in red. SC: Self-captioning, PC: Pre-captioned. Asterisks denote statistical significance: p 0.05,
*** p 0.01, *** p < 0.001.
Task
Qwen 72B Base
Qwen SC
Qwen PC
Perception (Cat)
89.00
-2.67=3.89
+2.00
Perception (Loc)
70.50
+9.17#4.11
+29.50=#0.91
Feature Attention
10.54
+8.15*#11.70
+19.224#+:10.43
Spatial Attention
80.56 1:2.88
+5,44*#11.85
+15,44=s=111.29
Memory (Cat)
87.69 1.29
-1.42*d1.10
+6.59*w#ł00.91
Memory (Loc)
58.44 1.97
+17.78***41..00
+36.47*#+:00.88
CVR-Cat-L
80.00+7.77
-4.67=6.81
+20.00=1.85
CVR-Loc-L
52.00)9.61
+30.67#6.01
+47.00=2.64
CVR-Cat-M
88.00
-11.33-1.722
+6,00
CVR-Loc-M
49.60
428,6746.72
+42.00
CVR-Cat-H
46.33
-7,83=5.46
+8.17
CVR-Loc-EE
40.77
+24.3645..35
+50.03
33

---

Table A.5.2: PAM and CVR performance of GPT-4o using different captioning methods. The Base column shows absolute percentage accuracies (# standard deviation). Subsequent columns show the percentage change from the Base method. Positive changes are shown in green, negative in red. SC: Self-captioning. PC: Pre-captioned. Asterisks denote statistical significance: p 0.05, ** p 0.01, *** p < 0.001.
Task
4o Base
4o SC
4o PC
Perception (Cat)
90.33+3.36
-2.00+3.61
+0.6773.26
Perception (Loc)
89.00)3.555
+8.33#1.91
+11.00*#0.63
Feature Attention
87.19:11.78
18.89**=d1100
+12.81 +12.81#**:10.14
Spatial Attention
75.93+22.28
+15.33#*ss:1..1
+19.85=s*
Memory (Cat)
91.47=
+1.73=00.81
+3.44 +3.44####0070
Memory (Loc)
83.47-11.21
+9.89****110.88
+12.22#84:10.66
CVR-Cat-L.
91.33++1.66
+2.00
+8.67=1.25
CVR-Loc-L
66.00+7.50
+27.33+4.09
+34.00=1.25
CVR-Cat-M
96.67+3.07
-7.33#4.98
+0.67+2.81
CVR-Loc-M
82.67.66.04
+7.33+4481
+14.67=2.81
CVR-Cat-H
83.67#1.18
-2,00
+2.67#3.89
CVR-Loc-H
64.67
+18.67#14.22
+30.33
A.6 Additional Fine-tuning Results
Table A.6.1: PAM and CVR performance of Qwen2.5-VL-322B base model and fine-tuned variants using LoRA with different amounts of training data. The Base column shows absolute percentage accuracies (# standard deviation). Subsequent columns show the percentage change in accuracy from the Base method (#: standard deviation). Positive changes are shown in green, and negative red. Asterisks denote statistical significance: p 0.05, ** < 0.01, p 0.001.
Task
32B Base
32B LoRA 1k
32B LoRA 100k
Memory (Cat)
85.38=1.13
-1.91
+2.61 1.04
Memory (Loc)
48,80 48,89=144 1.63
+8.44*8*t1.00
+16.80+##41.66
Perception (Cat)
87.67=3.73
-1.33
-1.00=3.85
Perception (Loc)
49.67=5.62
+19.33#5.21
+29.67*=1.57
Feature Attention
76.13=2.27
-11.13#=2.64
110.31
Spatial Attention
22.0
0.544
+1.52=22.08
CVR-Cat-H
59.39=5.52
+.066
+16.944=4.79
CVR-Loc-H
35.3345.35 5.38
35,33
45.33
+26.33#
CVR-Cat-M
72.67 1.00
-9.333
-4.6747.38
CVR-Loc-M
46.00=7.88
+6.00#7.80
+15.33+7.70
CVR-Cat-L
83.33 5.995
-2..677777
-18.00
CVR-Loc-L
48.67=7.90
+10.00+7.78
+15.33
A.7 PAM CVR vs MMMU-Pro Performance
34

---

Mato Ey
Neeernge Accuracs IMNNNN mro
Nuvenng Accurary (ANONN PrO
Onart Cark
Auerape accurse (MMIN
Oneer wit wee 181
Mucrrape Accurscy INNONU
Average Accurscy (Perception) 1961
Avarage Ac scouracy (Attentoo) (%)
Avctage Accuracy (Mamery) 151
Average Accuracy (CVR) (%)
Figure A.7.I: Scatter plots comparing average PAM task (Perception, Attention, Memory) perfor- mance and CVR task performance against MMMU-Pro performance across all models. Each point represents a different model configuration. The x-axis shows the average accuracy on the specified PAM task category (averaging Loe and Cat), and the y-axis shows the overall MMMU-Pr accuracy.
A.8
PAM & CVR vs VQAv2 Performance
(V0A (80
(50)
CASSOA POOOO (VOR 198)
74 180
78 1-M USE
78
24 1-M 1X0
138
130.5
Average areyrary (Pereprionn (54)
Average Accuracy (Attené un) 1861
Average Accurary (Nemoryl 1961
Pverage Accurwty (CVRL (50
Figure A.8.1: Scatter plots comparing average PAM task (Perception, Attention, Memory) perfor- mance and CVR task performance against VQAv2 performance across different variants of Qwen2.5 VL models. Each point represents a different configuration. The x-axis shows the average accu- racy on the specified PAM task category (averaging Loc and Cat), and the y-axis shows the overall VQAv2 accuracy.
A.9 Attention Image Capacity Analysis
To investigate potential attention capacity limitations introduced by the presence of images during reasoning, compared the average attention allocated to ground-truth caption tokens across the chain-of-thought (CoT) tokens generated by Qwen2.5-VL-7B both PC (pre-captioned) and PC-I (pre-captioned with images) trials. Appendix Figure AТT shows the inclusion of images signif- icantly reduces the attention a given ground-truth caption token receives while the model outputs COT. These findings may reflect potential attention capacity issue imposed by images during reasoning, which improvements during CoT training could remedy.
35

---

Average Caption Token Attention Across CoT Tokens
0.045
PC
0.040
PC-1
Attention
Perception
Memory
Spatial Attn
Temporal Attn
Task
Figure A.9.1:
Comparison of average Qwen2.5-VL-7b attention scores for ground-truth caption tokens across CoT tokens during PC and PC-I prompts of PAM tasks.
A.10 Inference Runtime Analysis
To assess the computational overhead of our proposed Self-Captioning (SC) method, measured the average inference runtime and compared it against the baseline direct-input method. We con- ducted these analyses timed 10 trials for each task. The results, detailed Table A. show that the SC method incurs significant runtime cost, taking approximately twice long the base method for both simple and complex tasks.
Qwen2.5-VL-7B model running single NVIDIA A5000 GPU. We
Table A.10.1: Inference runtime comparison for the Self-Captioning (SC) vs. Base method on Qwen2.5-VL-7B. Runtimes are reported in seconds (# standard deviation).
Task
Base
SC
Mem-Cat-E
1.81 + 1.46
5.55 + 2.13
Mem-Cat-C
2.06 + 1.86
5.37 + 1.59
Mem-Loc-R
1.60 + 1.20
5.27 + 1.99
Mem-Loc-C
2.20 + 1.72
5.40 + 1.29
Perc-Cat-R
1.54 + 1.16
3,21 + 0.59
Perc-Cat-C
1.96 1.43 1.43
4.61 0.60
Perc-Loc-R
1.54
2.66 1.04
Perc-Loc-C
2.07
3.39
CVR-Cat-H
6.58 4.67
14.66
CVR-Loc-H
5.12
14.62
CVR-Cat-M
4.05
12.59 # 1.94
CVR-Loc-M
6.70
13.06 + 2.10
CVR-Cat-L
+
10.54 2.17
CVR-Loc-L.
10.76 + 1.55
Att-Feat-R
2.74
4.06 0.89
Att-Fcat-C
1.59
6.09 + 1.80
Att-Spa-R
3.06
4.66 + 1.27
Att-Spa-C
2.78 1.52
5.89 1.46
Mem-Dis Dis-Cat-R
3.31 +
6.56
Mem- Miem-Dis-a Dis-Cat-CC
3.21
2.89
Mie--is-Looc
2.77 2.45
2.51
Mem-Dis-Loc-CC
3.28 +: 2.92
8.35 + 2.25
A.11 Prompt Script Examples
36

---

"messages": [
"role": "user",
"content"; "In this task, we will show you a series of frame images. Each frame will either be blank (delay frame) or contain one or more 3D objects. The objects will always be from lighting, planes, and tables. For each category, there are eight unique objects that could be used the task. Any object sampled will displayed as an image taken from a random viewing angle. The objects will be placed in one of four locations: top left, top right, bottom left, and bottom right. If there are multiple objects a single frame, only by either its location or its category. written instruction will be provided. Your goal is to follow the instructions and answer the question contained in the instructions. Anawers will always be one of the following: true, false, botton right, bottom left, top left, top right, benches, boats, cars, chairs, couches, lighting, planes, tables.
of eight categories: benches, boats,
chairs, couches,
of them would be specified in the task instruction
Please solve the following task:
Task instruction; observe object in frame 1, observe object 2 in frame 2, observe object in frame 3, observe object in frame observe object frame 5, delay, observe object in frame 7, delay, observe object frame 9, if identity of object 3 equals identity of object 2, then location of object not equals location of object 6 and identity of object 5 equals identity of object 4? else location of object 1?
Here are the corresponding frames: <image><image>
What is the correct answer this task? (bottom right, bottom left, top left, top right). Provide your answer here:
}, "role": "assistant", "content"; "top right"
"images": [
wt./rriall//trame//pooooo png",
httrriall//trmmee//ppctt png",
/triall/ftramee//ppocht png",
/tt.aall/rraaeeeeppptt......p//p png"
/ttraall/traaess/pocc....ppp png",
/tt.aall/tramme/pppctt...pp png"
/trpiall//tramme/pppctoooppp png",
/ttp:/aal//.ramee//ppctoo.to png"
]
}
Figure A.11.1: Supervised LoRA Fine-tuning Prompt Example
"Frame chairs located at the top left",
"Frame chairs located at top right".
"Frame 3: benches located at the top left",
"Frame 4: A benches located at the bottom right",
"Frame 5: boats located the bottom left".
"Frame benches located the bottom right",
"Frame delay frame",
"Frame 8:
delay frame", "Frame 9: A planes located at the top left"
Figure A.11.2: Pre-caption example
37

---

def caption, image (image_ __tth, model, config): Prepare prompt asking the model to caption the smage: paption_prompt
"role": "user", "content":
"type": "text",
"text": "Please provide a concise caption for the given image, including what the location of each the object in the images are and what the category of each object is. Each image either blank delay frame) or contains one or more 3D objects from \
one of eight categories: benches, boats, cara, chairs, couches,
lighting, planes, and tables. The object is placed in
four
locations: top left, top right, bottom left, or bottom right." }. {
"type": "image_url", " iimage_ur"* url": {"url":
+ "'data: ddata:image/png.
baseee,6,,, base64, (encode_imess (image 3ataagee_pah)"
]
}.
]
Call. the model to get caption
reaponse chat..chat... completions create(
model=configg get g"oaii-modell"" "gpt-4o-mini"), messages-captin__ee _proppt, max 10x_tokens=1024
The model 's caption is expected in the response. caption response. choices [0] message. wontsatteecoonttt.oommm strip( return caption
Figure A. 1.3: Code for Self-captioning
38

---

Think def evaluate model mtaee base, trial_num, model, config):
crial_base ""[task_base) (tria http:/riialttriaaamammmrrrreeee / task_info_path os. path. join
"new ton_task_inne insis_innoojooo""
with open info inth__pthh 'r') file: data json. load(file)
Extract the instruction and answers instruction datal' new instruction' '] data answers
Find the images pattern "eooc* png'
matchingg files [f for listdiirttriaall base)) if fnmatch, fnmatch + patteru)]
images = los.path. join file) for file in matching_ files]
NEW PART: First, caption each image
captions [
for image image_path in images:
caption caption, (imgg model, config)
captions appiinna.pppeddcccc appendccaption (caption)
Now have list of captions, Incorporate these captions into the final prompt
per frame
Instead of sending images to the model in the final task, we send their captions:
Prepare the final evaluation prompts
The prompt should now descride that we have tentual descriptions of each 4 frame:
nossible_ansvert answers = reduce (1ambda x, f"[x), {y)", get_ get_all_ppat pow.allpppbbeeeeeeeeeeaaeaeeelle ans (answers[-1]]
prompts
g"""In this task, will show you series of frames described by captions instead of direct images. Each frame either is blank (a delay frame) or contains 3D object from of eight categories: benches, boats, cars, chairs, couches, lighting, planes, and tables. The object placed one of four locations: top left, top right, bottom left, or bottom right.
written instruction provided, and you must follow that instruction + find the correct answer. The valid answers are alvays one of the following: true, false, botton right, bottom left, top left, top right, benches, boats, cars, chairs, couches, lighting, planes, tables.
Task instruction: (instruction)
Here are the frame captions:
aun
"\n" [a".joon([[ {i+1): (cap)" for i, cap (captionnc
4 "\n'n" +
f" What is the correct ansver to this task?
step-by-step, analyze each frame and provide your answer Narnswere AnLet's think step by step." "
]
Figure A.11.4: Example for Evaluation (Part 1/2).
39

---

measages
"role": "user", "content";
"type": "text", "text": prompts[0],
"type": "text", "text": prompts[1],
},
]
}
]
print (messages) instruction instruction. (oinn(f"Frame {i+1): {cap)" for i, cap in enumerate (captions)])
Get the response for the final evaluation reaponse = model.chat. www....attt...e..o
completions create( messages-nessages
monfl-config... get
"gpt-4o-mini")
max_ mox_tookes=ccofigg get (wmam" hewk_new_ttkkeeet 50),
return instruction, answers [-1], response... choices [o] message content
Figure A.11.5: Example for Evaluation (Part 2/2).
40


Reviews:
[{"review_id": "V0XT2v3LB4", "reviewer": "Reviewer_o9hF", "summary": "The paper analyses the VLM performances across three core cognitive axis - perception, attention and memory. They perform rigorous experiments across seven models spanning from open-source models to GPT-4o. The authors analyse several gaps pertains these models in tasks such as spatial understanding and selective attention. The authors state the need for improved chain-of-thought capabilities. This works presents a holistic understanding of the VLM cognitive strengths, weakness to explore the bottlenecks in simultaneous perception and reasoning.", "questions": "The authors might refer to the weakness and answer the question presented.", "limitations": "Yes the authors provided a separate section for the Limitations", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. VLM evalaution on Perception, Attention and Memory based on human faculties presents a well cognitive grounded framework.\n2. The authors have well explored the VLM bottlenecks showing a consistent gaps in the spatial localisation and selective attention in almost all the models such as perception-loc, memory-loc and spatial-attention.\n3. The authors also replaced images with gournd truth captions  to improve Qwen2.5-VL by 55pp on perception-loc and +22pp on feature attention validating that lan can reason when the visual information is provided in a text form. \n\n\nWeakness:\n1. The generalization of the fine-tuning of LoRA is limited to Qwen2.5-VL-7B., why not other larger models?\n2. One of the major questions is relying on a single chain of thought template, the authors night explore few-shot prompting etc.\n3. The authors could have tested on unseen real-image dataset VQAv2, as there is marginal change on MMBench and MMMU-Pro for LoRA finetuned model.\n4. The authors do mention of LoRA overfitting and add dropout but might add a detailed ablation such as different dropout rates to quantify better.\n5. The paper could have presented attention heat maps for attention capacity issues.", "was_revised": true, "final_justification": "The authors did address most of the points raised and thus the rating is revised.", "rebuttal": "**The generalization of the fine-tuning of LoRA is limited to Qwen2.5-VL-7B., why not other larger models?**\n\nWe performed new fine-tuning experiments on Qwen2.5-VL-32b to address this request. The table below shows how fine-tuning on 1k and 10k datasets affected PAM and CVR performance. \n\n| Task               | 32B Base        | 32B LoRA 1k    | 32B LoRA 10k  |\n|--------------------|------------------|----------------|----------------|\n| Memory (Cat)       | 85.38 ± 1.13     | 83.47 ± 1.19   | 87.99 ± 1.04   |\n| Memory (Loc)       | 48.89 ± 1.63     | 57.33 ± 1.61   | 65.69 ± 1.55   |\n| Perception (Cat)   | 87.67 ± 3.73     | 86.33 ± 3.89   | 86.67 ± 3.85   |\n| Perception (Loc)   | 49.67 ± 5.62     | 69.00 ± 5.21   | 79.33 ± 4.57   |\n| Feature Attention  | 76.13 ± 2.27     | 65.00 ± 2.54   | 65.81 ± 2.53   |\n| Spatial Attention  | 80.98 ± 2.09     | 80.44 ± 2.12   | 82.51 ± 2.03   |\n| CVR-Cat-H          | 59.39 ± 5.52     | 53.33 ± 5.61   | 76.33 ± 4.79   |\n| CVR-Loc-H          | 35.33 ± 5.38     | 40.67 ± 5.52   | 61.67 ± 5.47   |\n| CVR-Cat-M          | 72.67 ± 7.07     | 53.33 ± 7.88   | 68.00 ± 7.38   |\n| CVR-Loc-M          | 46.00 ± 7.88     | 52.00 ± 7.89   | 61.33 ± 7.70   |\n| CVR-Cat-L          | 83.33 ± 5.95     | 58.67 ± 7.78   | 65.33 ± 7.53   |\n| CVR-Loc-L          | 48.67 ± 7.90     | 58.67 ± 7.78   | 64.00 ± 7.59   |\n\nApart from a few outliers like CVR-Cat-M/H and Feature Attention, fine-tuning improved the model’s performance. Compared to the 7b variant, fine-tuning 32b had a similar but lesser effect. Unfortunately, due to time and computational constraints, we were unable to fully test our 32b models. The results for their performance on held-out iWISDM tasks, MMMU-Pro, MMBench, as well as a 100k LoRA model, will be added to the manuscript. \n\n\n**One of the major questions is relying on a single chain of thought template, the authors might explore few-shot prompting etc.**\n\nWe had not included few- shot prompting in our experiments because of two reasons: 1) The context window of VLMs, as well as computational resource limits, severely restricted the number of images that could be included in the prompt. For this reason, we were unable to include more images as examples, especially for the complex tasks which already included up to 9 images; 2) Prior work had reported limited improvements in VLM reasoning capacity in few-shot settings [1,2]. \n\nNevertheless, to address the reviewer’s request more directly, we performed new one-shot experiments with Qwen2.5-VL-7B.\n\n| Task              | Base       | SC          | One Shot    | PC          |\n| ----------------- | ---------- | ----------- | ----------- | ----------- |\n| Memory (Cat)      | 78.59      | +4.72       | -1.55       | +6.27       |\n| Memory (Loc)      | 42.22      | +26.89      | -1.14       | +47.39      |\n| Perception (Cat)  | 83.67      | +1.33       | +2.33       | -2.00       |\n| Perception (Loc)  | 44.33      | +28.67      | -3.33       | +55.00      |\n| Feature Attention | 55.85      | +22.00      | -3.63       | +22.22      |\n| Spatial Attention | 57.48      | +18.15      | +0.74       | +8.52       |\n| CVR-Cat-H         | 39.33      | +6.00       | -10.33      | +13.00      |\n| CVR-Loc-H         | 29.33      | +17.33      | -2.33       | +32.00      |\n| CVR-Cat-M         | 54.00      | +3.33       | +8.00       | +12.67      |\n| CVR-Loc-M         | 49.33      | +10.00      | +6.67       | +10.67      |\n| CVR-Cat-L         | 60.00      | +2.67       | -2.00       | +1.33       |\n| CVR-Loc-L         | 56.00      | +8.67       | +2.00       | +12.67      |\n\nCompared with SC, one-shot didn't provide a strong performance boost. We will include these results as an additional table in the appendix. \n\n[1] Zong, Y., Bohdal, O., & Hospedales, T. (2025). VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning. ICLR 2025\n\n[2] Shukor, M., Rame, A., Dancette, C., & Cord, M. (2024). Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning. ICLR 2025\n\n\n**The authors could have tested on unseen real-image dataset VQAv2, as there is marginal change on MMBench and MMMU-Pro for LoRA finetuned model.**\n\nThat’s an excellent suggestion! Following the reviewer’s advice, we performed new experiments where we additionally tested the fine-tuned Qwen2.5-VL-7B and Qwen2.5-VL-72B on the VQAv2 dataset. The results for these tests: \n\n| Model/Configuration   | VQAv2 (5k subset) Accuracy (%) |\n|------------------------|--------------|\n| 7B Base                  | 60.07        |\n| 7B SC                    | 65.07        |\n| 7B LoRA 1k            | 63.96        |\n| 7B LoRA 10k           | 64.44        |\n| 7B LoRA 100k          | 65.84        |\n|72B Base| 71.09 |\n|72B SC| 72.02  |\n\nThis indicates an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model. These results further confirm that the observed improvements are robust and generalize to broader settings involving natural images. \n\nWe will include these results along other benchmarks in Table A3.2. \n\n**The authors do mention of LoRA overfitting and add dropout but might add a detailed ablation such as different dropout rates to quantify better.**\n\nTo find the best fine-tuning procedure, we ran a number of experiments during which we changed several hyperparameters, including the LR, LR scheduler decay ratio, dropout ratio, and optimizer parameters. Due to the sheer number of possible combinations, the high computational cost, and our limited academic-level resources, we were unable to try all possible combinations and therefore adopted the best setting for our smallest-scale experiments. Regarding dropout, we had briefly tried 0.0 and 0.1 before settling on 0.2. Our training runs with lower dropout ratios (0.0 and 0.1) experienced overfitting, resulting in a rise in the validation loss during training. These performance values will be added to the appendix to address the reviewer’s concern. Finally, we would like to note that the efficiency of the fine-tuning experiments was not directly tied to any of our main contributions. \n\n**The paper could have presented attention heat maps for attention capacity issues.**\n\nWe thank the reviewer for their thoughtful suggestion. We believe examining the attention scores would be an effective way to probe the role of attention capacity in our observations. To address the reviewer’s suggestion, we will compare the total attention scores on the relevant caption tokens between SC and SC-I experiments, visualize individual samples and also present the average values across many samples. We hypothesize that if the textual attention scores in SC-I are significantly lower than those in SC, that would confirm our hypothesis that the inclusion of images in the context distracts the model from the more reliable textual information. Our results show that the caption text tokens received a total of 0.022 attention averaged over the model's output when the SC method is used, and 0.017 when SC-I is used. This difference in attention provides strong evidence in support of our attention capacity hypothesis. \n\n**Note:** we cannot provide visuals of these attention scores due to Neurips guidelines; however, we will add the results of these and further analyses to the Appendix."}, {"review_id": "cuItzpjm2H", "reviewer": "Reviewer_Dy9Q", "summary": "The paper introduces PAM (Perception-Attention-Memory) and CVR (Composite Visual Reasoning)––two procedurally-generated suites that probe VLMs along cognitive-science axes. Seven state-of-the-art models (GPT-4o family, Qwen2.5-VL, MiniCPM-V, InternVL-2.5, LLaVA-OneVision) plus humans are evaluated. Results reveal strong object-category perception but large gaps in spatial reasoning and selective attention. A simple vision-text decoupling trick (self-captioning) substantially narrows the gap, and LoRA fine-tuning on 1k–100k CVR samples further boosts performance. PAM/CVR scores correlate tightly with MMMU-Pro, highlighting their diagnostic value.", "questions": "Questions for the Authors\n\n- Human study details. How many annotators? Were images shuffled to reduce practice effects?\n- Statistical significance. Are accuracy gaps tested (e.g., bootstrap CIs) or visually inspected only?\n- Caption quality. For self-captioning, do hallucinated captions ever hurt reasoning, and how is that measured?\n- Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?", "limitations": "Limitations (some acknowledged, some additional)\n\n- Synthetic bias. Only eight ShapeNet categories on plain backgrounds limit ecological validity. \n- Single-frame captioning. SC assumes each frame can be described independently; coherence across complex videos remains untested.\n- Overfitting risk. LoRA boosts CVR but slightly hurts MMMU-Pro at small data sizes; larger sets might magnify this trade-off. \n- Attention-capacity hypothesis unproven. The paper infers attention interference from SC-I drop, but no probing of cross-modal token limits is shown.\n- Missing energy/runtime analysis. Procedural tasks are cheap, yet large-scale SC + LoRA incurs extra inference cost.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Paper Strengths\n\n- Cognitive framing. Mapping VLM skills to classic P-A-M faculties is fresh and helps pinpoint bottlenecks. \n- Procedurally generated benchmarks. Tasks of unlimited size avoid test-set saturation and enable controlled ablations.\n- Comprehensive evaluation. Mix of open-source and proprietary models + human baseline yields a clear landscape.\n- Actionable insight. Self-captioning is a one-line prompt change yet delivers 18–55 pp gains on spatial tasks. \n- Practical fine-tuning study. Shows that even 1 k synthetic examples meaningfully lift location, attention, and memory scores. \n- Correlation to real-world benchmarks. Strong R²≈0.9 between PAM/CVR and MMMU-Pro argues external validity.", "was_revised": true, "final_justification": "The authors clarifications have addressed my questions well and have helped to further refine the paper. After careful consideration, I am pleased to increase my score by +1.", "rebuttal": "We would first like to thank the reviewer for their thoughtful feedback and suggestions! Our responses to your comments are as follows:\n\n**Questions:**\n\n**Human study details. How many annotators? Were images shuffled to reduce practice effects?**\n\nThe details of the human study were included in Appendix A.2. To answer your question briefly, the study included 8 subjects and trials were all randomly generated to include random selection of object images. \n\n**Statistical significance. Are accuracy gaps tested (e.g., bootstrap CIs) or visually inspected only?**\n\nThe sample size in our studies was limited (N=3 per model per subtask), resulting in low statistical power for some tasks, especially CVR and Perception tasks, they have 3 or 6 samples in total per task. However, in response to the reviewer’s concern, we performed significance tests for all of our main results. \n\nBriefly, we found that for Qwen 2.5 VL 7B, SC, SC-I PC contrasts with Base were 5/6 tasks, 2/6 tasks, 4/6 significant with at least p<0.05 on PAM, both SC and PC has a significant different (p < 0.05) on CVR-Loc-H with Base.\n\nFor larger Qwen 2.5 VL 72B SC and PC contrast with Base in all Memory and Attention tasks with at least p < 0.05 significant.\n\nFor closed model, GPT 4o, SC and PC contrast with base in 3/6 and 5/6 tasks, with p < 0.05 in PAM and both in CVR-Loc-H.\n\nFinally, the LoRA fine-tuned Qwen 2.5 VL 7B models, LoRA 1k, 10k, 100k demonstrated p < 0.05 differences in 4/6, 4/6, 6/6 tasks in PAM with the base model and all LoRA models have a p < 0.05 difference in CVR-CAR-H, 10k and 100k in CVR-Loc-H too.\n\nWe will include the results in all tables in the revised manuscript. \n\n**Caption quality. For self-captioning, do hallucinated captions ever hurt reasoning, and how is that measured?**\n\nThe answer is yes, and some results in the manuscript already address this. Specifically, the differences between PC and SC values reported in Tables 2 and A.5 capture the accuracy of VLMs when using ground-truth image captions (PC) versus VLM-generated captions (SC). We interpret the drop in performance between these conditions as an effect of captioning hallucinations. This is an important point that was not clearly stated, and we will clarify it in the revised manuscript.\n\n**Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?**\n\nThat’s an excellent suggestion! Following the reviewer’s advice, we performed new experiments where we additionally tested the fine-tuned Qwen2.5-VL-7B and Qwen2.5-VL-72B on the VQAv2 dataset. The results for these tests: \n\n| Model/Configuration   | VQAv2 (5k subset) Accuracy (%) |\n|------------------------|--------------|\n| 7B Base                  | 60.07        |\n| 7B SC                    | 65.07        |\n| 7B LoRA 1k            | 63.96        |\n| 7B LoRA 10k           | 64.44        |\n| 7B LoRA 100k          | 65.84        |\n|72B Base| 71.09 |\n|72B SC| 72.02  |\n\nAs evident, an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model, which suggests that both of these approaches lead to general improvements in the VLMs’ ability to perform general visual reasoning. We will include these results along with other benchmarks in Table A3.2. \n\n**Limitations**\n\n**Synthetic bias. Only eight ShapeNet categories on plain backgrounds limit ecological validity.**\n\nWe chose iWISDM and synthetic objects because of their strong controllability, which allowed us to precisely control the object properties in the context of diverse cognitive tasks. With regard to the reviewer’s concern about ecological validity, we had already shown that the performance of different models on our PAM and CVR task sets is strongly correlated with those on benchmarks with natural images (MMMU-Pro). This suggests that the low performance of these models on our tasks is not incidental nor a result of poor data diversity. \n\nFurther, as shown previously in response to the reviewer’s question about generalization to natural photographs, we validated our methods on the naturalistic benchmark VQAv2, and found a 5-6% gain in performance from both SC and LoRA. \n\n**Single-frame captioning. SC assumes each frame can be described independently; coherence across complex videos remains untested.**\n\nWe acknowledge that this is a limitation of our current tests, as our experiments only involve video and do not consider more complex media formats such as videos. We will add this point to the limitations section of our paper to address the reviewer’s comment. \n\n**Overfitting risk. LoRA boosts CVR but slightly hurts MMMU-Pro at small data sizes; larger sets might magnify this trade-off.**\n\nWe agree that LoRA fine-tuning comes with the risk of overfitting. However, on our largest training dataset size of 100k, LoRA actually led to marginal improvements on both MMBench and MMMU-Pro. Even more substantial were our additional results on VQAv2, probed from the reviewer’s comments on generalization, which show a ~5.8% improvement. With that said, fine-tuning via other methods, such as reinforcement learning or full SFT, would be interesting avenues for future experiments. \n\n**Attention-capacity hypothesis unproven. The paper infers attention interference from SC-I drop, but no probing of cross-modal token limits is shown.**\n\nWe agree with the reviewer that we did not offer any evidence to support this statement and only speculated that this is likely due to this hypothesis. Following reviewer **o9hF**’s suggestion we examined the attention scores to probe the role of attention capacity in our observations. To address the reviewer’s suggestion, we compared the total attention scores on the relevant caption tokens between SC and SC-I experiments, visualized individual samples and also calculated the average values across many samples. We hypothesize that if the textual attention scores in SC-I are significantly lower than those in SC, that would confirm our hypothesis that the inclusion of images in the context distracts the model from the more reliable textual information. Our results show that the caption text tokens received a total of 0.022 attention averaged over the model's output when the SC method is used, and 0.017 when SC-I is used. This difference in attention supports our attention capacity hypothesis. \n\n*Note:* Unfortunately, we cannot provide visuals of these attention scores due to Neurips guidelines; however, we will add the results of these and further analyses to the Appendix.\n\n**Missing energy/runtime analysis. Procedural tasks are cheap, yet large-scale SC + LoRA incurs extra inference cost.**\n\nWe thank the reviewer for suggesting this critical analysis we missed. We have now performed runtime analyses on 10 trials of a 1-frame task (Perc-Loc-R) and 10 trials of a 9-frame task (CVR-Loc-H) given to Qwen2.5-VL-7b. We find that the SC method is indeed slower than the standard method of input on average for both trial lengths. SC performed the 1-frame trials in an average of 5.731 seconds and the 9-frame trials in an average of 35.259 seconds. The base method performed the 1-frame trials in an average of 2.841 seconds and the 9-frame trials in an average of 13.410 seconds. These analyses revealed an important limitation to our method; despite the significant performance gains of SC, it comes at the trade-off of roughly a 2x slower run-time. We will be sure to include this limitation and the details of our run-time analysis in the manuscript. \n\n**Formatting Concerns** \n\nThank you for pointing out these issues. They have been corrected in the updated manuscript!"}, {"review_id": "MGi2cnm9jt", "reviewer": "Reviewer_ptot", "summary": "This paper systematically investigates Vision-Language Models (VLMs) through the lens of cognitive science, breaking down their abilities into Perception, Attention, and Memory (PAM) and assessing how these core capabilities impact composite reasoning. It introduces:\n\n1. PAM and Composite Visual Reasoning (CVR) datasets, procedurally generated with iWISDM, to test fine-grained cognitive abilities.\n\n2. A vision-text decoupling analysis, using self-captioning and pre-captioning to understand if failures come from perception or integration.\n\n3. Experiments showing targeted fine-tuning significantly improves performance, especially on spatial reasoning.", "questions": "As mentioned in the weakness.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. **Novel Analytical Framework:**\n   The paper reframes VLM evaluation around core cognitive primitives—**Perception, Attention, and Memory**—rather than simply task types. This perspective, inspired by cognitive science, provides clear, testable hypotheses about where and why models fail.\n\n2. **Interesting Experimental Observations:**\n   The work conducts several insightful experiments to probe specific deficiencies of current VLMs, such as the vision-text decoupling analyses that help disentangle perceptual encoding from reasoning limitations.\n\n---\n\n### **Weaknesses**\n\n1. **Limited Evaluation Data Diversity:**\n   All tasks are generated from a synthetic dataset with only 8 object categories and uniform backgrounds. This significantly limits ecological validity, as real-world images involve clutter, occlusion, and more diverse visual content.\n\n2. **Domain-Specific Testing Setup:**\n   The reliance on the iWISDM environment creates a highly specialized evaluation scenario. This makes it difficult to disentangle whether observed errors reflect true perception and reasoning deficits, or simply failures of out-of-distribution generalization.\n\n3. **Interpretation of Fine-tuning Gains:**\n   Regarding the claimed contribution (line 65), it is not surprising that fine-tuning on the target domain improves performance on tasks from that same domain. Therefore, it is hard to attribute these gains specifically to improved *diverse cognitive reasoning abilities*, rather than to domain memorization or overfitting.", "was_revised": true, "final_justification": "The response has addressed most of my previous concerns and questions. I increase my score by to 4.", "rebuttal": "We would first like to thank the reviewer for their thoughtful feedback and suggestions! Our responses to your comments are as follows:\n\n**Limited Evaluation Data Diversity and Domain-Specific Testing Setup:** \n\nWe chose iWISDM and synthetic objects because of their strong controllability, which allowed us to precisely control the object properties in the context of diverse cognitive tasks. With regard to the reviewer’s concern about ecological validity, we had already shown that the performance of different models on our PAM and CVR task sets is strongly correlated with those on benchmarks with natural images (MMMU-Pro). This suggests that the low performance of these models on our tasks is not incidental nor a result of poor data diversity. \n\nRegardless, to directly address the reviewer’s concern, we ran new experiments in which we tested our methods on the VQAv2 dataset. The results are as follows:\n\n| Model/Configuration   | VQAv2 (5k subset) Accuracy (%) |\n|------------------------|--------------|\n| 7B Base                  | 60.07        |\n| 7B SC                    | 65.07        |\n| 7B LoRA 1k            | 63.96        |\n| 7B LoRA 10k           | 64.44        |\n| 7B LoRA 100k          | 65.84        |\n|72B Base| 71.09 |\n|72B SC| 72.02  |\n\nAs evident, an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model, which suggests that both of these approaches lead to general improvements in the VLMs’ ability to perform general visual reasoning. We will include these results along with other benchmarks in Table A3.2. \n\n**Interpretation of Fine-tuning Gains:** \n\nWe included this claim based on two key results in the paper: 1) In Table 3, we showed that the fine-tuned QWEN2.5-VL-7b model performed substantially better on unseen tasks from PAM; 2) In Table 3 (reported on lines 312-315), we show that fine-tuning the QWEN2.5-VL-7b model led to slight improvements on MMBench and MMUPro benchmarks, both of which contain realistic images from different domains. In addition, the newly performed VQAv2 experiments and results show significant improvements from fine-tuning. Together, these results indicate that the gain in performance is large in tasks that are conceptually close to the training distribution, but somewhat remains in natural domains with relatively little overlap with the training distribution. To address the reviewer’s concern we will revise the claims (and related statements) to the following: “Demonstrating that targeted fine-tuning on diverse cognitive reasoning tasks can moderately enhance core cognitive abilities in VLMs *when tested with paradigms similar to those used to test humans*, while also generally validating that improvements on PAM and CVR tasks translate to broader gains in visual reasoning performance.”\n\nFinally, we will revise the abstract and the text and tone down our claim about the effect of finetuning by changing the claim from “substantially improving core cognitive abilities” to  “moderately improving core cognitive abilities”."}, {"review_id": "ixwk0CfGAH", "reviewer": "Reviewer_n5pe", "summary": "This paper explores the visual cognitive abilities of Vision-Language Models (VLMs), focusing on perception, attention, and memory. The authors follow a four-step approach. First, they build two datasets, PAM and CVT, using a toolkit called iWISDOM. Second, they test top-performing VLMs on these datasets to find their weaknesses. Third, they analyze the reasons behind these weaknesses. Finally, they fine-tune the models to improve their cognitive abilities. Overall, the paper builds custom datasets, uses them to test VLMs, and draws conclusions based on the results.", "questions": "1. The PAM and CVR datasets are built using iWISDOM, but the paper does not explain how they differ from the original iWISDOM dataset. It remains unclear why the existing dataset cannot be used directly. More explanation is needed to justify the creation of these new datasets.\n\n2. The perception ability of VLMs has already been extensively studied. The PAM and CVR datasets do not seem to provide significant new insights in this area. As for attention and memory, the tasks mostly involve referring to objects across single or multiple frames, which are still closely tied to perception rather than true reasoning. Therefore, the connection to reasoning ability is weak.\n\n3. There are already several works that explore the reasoning capabilities of VLMs, such as [1] and [2]. The paper should discuss these related works, clarify how its approach differs, and explain why it is necessary to create new datasets or tasks focused on reasoning.\n\n4. In Table 1, the performance of GPT-4o and human participants on CVR-Cat is confusing. Despite being labeled as a low-difficulty task, the results show performance worse than the medium-difficulty tasks. The paper lacks a detailed analysis and fails to provide convincing explanations for this inconsistency.\n\n[1] Visual Abductive Reasoning\n\n[2] The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning", "limitations": "The paper aims to study the cognitive abilities of VLMs, but the problem definition remains more focused on perception rather than reasoning. It does not specifically address key reasoning challenges in existing VLMs, such as counting. In addition, the PAM and CVT datasets are not described in sufficient detail, and the paper lacks essential information such as dataset statistics, creation process, and sample examples. The differences between these datasets and the original iWISDOM dataset should also be clearly explained.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper attempts to study the cognitive abilities of Vision-Language Models (VLMs), which is an exploratory and meaningful direction.\n\nWeaknesses:\n\n1. The motivation is not clearly presented. In the introduction, after the background section (Lines 25–41), the authors claim that \"VLMs struggle with multi-object reasoning tasks such as counting or identifying objects...\". However, the rest of the paper does not clearly explain how their method addresses the counting problem.\n\n2. The paper lacks a clear and detailed description of the PAM and CVR datasets in Section 3.2. There is no information available about the dataset statistics, the creation process of the datasets, or example cases. It’s also unclear why two separate datasets are needed. \n\n3. The description of the CVR dataset (Lines 139–145) is too brief.\n\n4. The paper claims it can \"identify specific weaknesses\" of VLMs (Line 60), but this is not clearly described.\n\n5. The claim that the paper reveals the reasoning bottleneck of VLMs (Line 62) is also not clearly described.\n\n6. Fine-tuning on a small dataset does not necessarily address cognitive reasoning issues if the training and testing follow the same distribution. To prove that fine-tuning improves general reasoning rather than just helping the model memorize the data, the train/test should be out-of-distribution data.", "was_revised": true, "final_justification": "The author's response tackles my concern, and agrees with the other reviewers that this paper is more complete with newly added contents in rebuttal. I would increase my rating by +1.", "rebuttal": "We would like to first thank the reviewer for their thoughtful feedback and suggestions. Our responses to your comments are as follows:\n\n**The motivation is not clearly presented. In the introduction, after the background section (Lines 25–41), the authors claim that \"VLMs struggle with multi-object reasoning tasks such as counting or identifying objects...\". However, the rest of the paper does not clearly explain how their method addresses the counting problem.**\n\nTo clarify, our aim for that section of the introduction was to introduce weaknesses in VLM reasoning that other studies have found. Reasoning is a broad term that encompasses many cognitive abilities, including counting objects. We did not intend to imply that counting would be directly addressed in our study, as this work focused on evaluating core cognitive abilities (PAM) as well as high-level decision making where those core abilities are used (CVR). We will be sure to update the introduction section to make this clearer to the reader. \n\n**The paper lacks a clear and detailed description of the PAM and CVR datasets in Section 3.2. There is no information available about the dataset statistics, the creation process of the datasets, or example cases. It’s also unclear why two separate datasets are needed.**\n\nThank you for pointing this out. We agree that Section 3.2 would benefit from a more detailed presentation of both the PAM and CVR datasets. In the revised manuscript, we will add dataset statistics and clarify the creation process. We had already included examples for all PAM and CVR task types in Figures A.1.1-5. We also address the differences between PAM and CVR and why they are both necessary below in response to the reviewer's question of *The PAM and CVR datasets are built using iWISDOM…*.\n\n**The description of the CVR dataset (Lines 139–145) is too brief.**\n\nIn the revised manuscript we will provide further details on the description of the CVR dataset, similar to how we describe it below in response to the reviewer's question of \"*The PAM and CVR datasets are built using iWISDM…*\". However, as CVR is a replication of the data used in the benchmark section of the iWISDM paper, we feel it is sufficient to have a brief explanation and list of details, and point to the iWISDM paper, if the readers are interested in further details. \n\n**The paper claims it can \"identify specific weaknesses\" of VLMs (Line 60), but this is not clearly described.**\n\nWe already mention the two specific weaknesses revealed by our analyses (spatial localization and selective attention) in the list of contributions at the end of the Introduction (lines 60-61). Furthermore, sections 4.1 and 4.2 of our manuscript provide detailed descriptions of the VLM weaknesses exposed by the PAM and CVR datasets. \n\n**The claim that the paper reveals the reasoning bottleneck of VLMs (Line 62) is also not clearly described.**\n\nWe appreciate the reviewer’s feedback, and we will update the manuscript to include a clearer description of the reasoning bottleneck we find. We will amend the end of Section 4.3 (lines 275-292), where we have our current descriptions, to more directly describe that the reasoning bottleneck is caused by an inability of SOTA VLMs to effectively use CoT to first caption images and then subsequently reason with the extracted information. \n\n**Fine-tuning on a small dataset does not necessarily address cognitive reasoning issues if the training and testing follow the same distribution. To prove that fine-tuning improves general reasoning rather than just helping the model memorize the data, the train/test should be out-of-distribution data.**\n\nWe agree entirely, and this is precisely why we made sure to evaluate the finetuned models on other cognitive benchmarks like MMMU-Pro and MMBench. You can see in the Appendix Table A.3.2 that slight improvements to these benchmarks were made. Moreover, following suggestions from reviewers *o9hF*, *Dy9Q*, and *ptot*, we further tested the models on the VQAv2 dataset consisting of diverse questions about natural images, completely different from those used during fine-tuning. A gist of these results can be seen in our response to reviewer *Dy9Q*’s comment on \"*Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?*\". Altogether, these results confirm that fine-tuning improves the model’s performance on multiple out-of-distribution datasets. We will be sure to make these results more evident in the main text. \n\n**Questions:**\n\n**The PAM and CVR datasets are built using iWISDOM, but the paper does not explain how they differ from the original iWISDOM dataset. It remains unclear why the existing dataset cannot be used directly. More explanation is needed to justify the creation of these new datasets.**\n\nTo clarify, iWISDM is an environment where datasets for visual decision-making tasks can be generated randomly or constructed manually, rather than a set of fixed datasets. To perform the experiments and analyses that our paper focuses on, we created PAM and CVR using iWISDM:\n\n- PAM is manually designed, explicitly targeting core cognitive abilities: Perception, Attention, and Memory. The Attention dataset also included the use of distractor objects, a feature of iWISDM that was not explicitly used in the iWISDM paper.\n\n\n- CVR employs the iWISDM *AutoTask* environment to dynamically generate complex tasks via random compositions of basic task operators. These tasks are specifically generated to *assess the general visual reasoning capacity of different models* in our work. The tasks in CVR were randomly generated following the same parameters as used in the iWISDM paper for its section on benchmarking.\n\nWhile necessary in achieving the main contributions of this study, PAM and CVR were not the main contributions in themselves. We will clarify this and provide further details on the dataset distinctions in the revised manuscript.\n\n**The perception ability of VLMs has already been extensively studied. The PAM and CVR datasets do not seem to provide significant new insights in this area. As for attention and memory, the tasks mostly involve referring to objects across single or multiple frames, which are still closely tied to perception rather than true reasoning. Therefore, the connection to reasoning ability is weak.**\n\nWe acknowledge that the perceptual abilities of VLMs have been studied previously. However, our work offers a new perspective in several key ways: 1) it reveals substantial failures in object localization, even in simple scenes with few objects and no background; 2) it demonstrates that a simple self-captioning strategy can meaningfully address this limitation across multiple visual reasoning tasks; and 3) it grounds evaluation in a cognitively inspired framework, using tasks from classic studies of perception, memory, and attention. Together, we believe these contributions offer a fresh lens on the cognitive capabilities of VLMs that, to our knowledge, has not been explored in prior work.\n\n**There are already several works that explore the reasoning capabilities of VLMs, such as [1] and [2]. The paper should discuss these related works, clarify how its approach differs, and explain why it is necessary to create new datasets or tasks focused on reasoning.**\n\nWe thank the reviewer for pointing us to these papers and will include them in our discussion of reasoning in VLMs in the revised manuscript. To clarify the differences: both papers focus on visual *abductive* reasoning, which differs from our work in key ways. (1) Abductive reasoning involves inferring the most plausible explanation—often a hidden cause—for a given observation. In contrast, our work focuses on *logical* reasoning, where VLMs are required to follow explicit instructions with unambiguous answers, leaving no room for alternative interpretations. (2) Unlike those works, our tasks involve *multiple images* and require *explicit relational reasoning* across them. Existing multi-image datasets are limited in number and typically involve only two images per task, whereas our tasks operate over larger sets.\n\n**In Table 1, the performance of GPT-4o and human participants on CVR-Cat is confusing. Despite being labeled as a low-difficulty task, the results show performance worse than the medium-difficulty tasks. The paper lacks a detailed analysis and fails to provide convincing explanations for this inconsistency.**\n\nWe acknowledge the reviewer's observation regarding the unexpected performance results in Table 1. We had briefly noted this anomaly in the original manuscript (line 239) with a possible explanation related to GPT-4o. On further reflection, we believe this counterintuitive result for human participants may also stem from the presence of the if-then-else operator in Medium complexity CVR-Cat tasks. Although Medium tasks involve more logical operations overall, the structured nature of if-then-else may restrict the number of objects and comparisons that need to be considered at a time, helping participants effectively segment instructions into manageable substeps. \n\nRelatedly, we also found that despite the apparent trend in accuracy across the three levels, the response times for human subjects consistently ramped up with higher complexity (averages: L=32.6s, M=39.1s, H=59.3s). These response times provide a more nuanced outlook on task difficulty. Despite having a higher response accuracy on medium CVR-Cat tasks than on the equivalent low-complexity tasks, subjects required more time to solve them. We will ensure that all response time data is included in the updated manuscript, along with a more detailed discussion of the specific result. We are also open to any explanations or approaches that the reviewer may have for further investigation.\n\n**Formatting Concerns:**\nThank you for pointing out these issues. They have been corrected in the updated manuscript!"}]

Author rebuttal:

Meta-review: **Summary**
This paper introduces a novel framework for evaluating VLMs with respect to isolated core cognitive abilities, by creating two procedurally generated benchmarks, PAM (Perception, Attention, Memory) and CVR (Composite Visual Reasoning). Benchmarking on seven SOTA models consistently reveal that while VLMs excel at object categorization, they exhibit significant weaknesses in spatial localization and selective attention. Vision-text decoupling analysis identifies the primary bottleneck not as flawed perception or language reasoning in isolation, but rather the VLM models' challenge in integrating visual information. To address this, the author team proposes a simple "Self-Captioning" prompting method that yields substantial gains, and demonstrate that targeted fine-tuning on synthetic cognitive tasks leads to improved performance that generalizes to established real-image benchmarks such as VQAv2.

**Strengths**
* Introduce a framework to reveal the cognitive profiles of SOTA VLMs that helps identify specific cognitive weaknesses (such as using spatial information and performing selective attention), drawing inspiration from cognitive science methods for assessing human core cognitive abilities.
* Analyses indicate that visual reasoning limitations primarily arise from challenges in integrating visual information, rather than from perceptual encoding or language reasoning alone. This suggests that current chain-of-thought training methods for visual reasoning remain largely ineffective, even in the strongest models.
* Based on this insight, propose a simple yet highly effective text-image decoupling approach that improves visual reasoning not only on our new cognitive datasets, but also on other prominent vision-language benchmarks. Also explored the effectiveness of fine-tuning on diverse cognitive task sets on downstream visual reasoning abilities, showing that improvements on the PAM and CVR datasets translate to broader gains in generalized visual reasoning performance.

**Potential Weaknesses, Rebuttal Discussion, and Reason for Accept/Reject**
The following key points were raised during the rebuttal period: Reviewer o9hF questioned the generalization of fine-tuning beyond the 7B model, and suggested testing on real-image datasets such as VQAv2 and exploring few-shot prompting; Reviewer Dy9Q requested details on human studies, statistical significance, an analysis of caption hallucinations, and runtime costs; Reviewer ptot expressed concerns about the synthetic data limiting ecological validity and the interpretation of fine-tuning gains; Reviewer n5pe requested clearer motivation, more dataset details, and a discussion of how the work differs from existing VLM reasoning studies.

The authors addressed these points comprehensively. They ran new experiments fine-tuning a 32B model, showing positive results. Crucially, they tested on VQAv2, revealing a 5-6% accuracy improvement from their methods, which strongly alleviates concerns about ecological validity and proves generalization to real-world images. They provided requested statistical analyses, runtime costs (~2x slower for self-captioning), and attention score analyses to support their capacity hypothesis. They also clarified the dataset descriptions, toned down certain claims, and added discussions of related work on reasoning.

All four reviewers were satisfied with the responses and increased their score ratings. The most critical concerns on generalization to diverse real-world settings and statistical rigor have been met with new experimental evidence. This paper should be a clear accept, as its core contributions of novel VLM evaluation framework & solution to a key perception-reasoning integration bottleneck stands even more strongly supported now.
Decision: Accept (spotlight)