Title: Caption This, Reason That: VLMs Caught in the Middle
Venue: NeurIPS 2025
Accepted: yes
Keywords: vlm

lmm

mllm

cognition

visual reasoning

reasoning

evaluation

finetuning
Abstract: Vision-Language Models (VLMs) have shown remarkable progress in visual understanding in recent years. Yet, they still lag behind human capabilities in specific visual tasks such as counting or relational reasoning. To understand the underlying limitations, we adopt methodologies from cognitive science, analyzing VLM performance along core cognitive axes: Perception, Attention, and Memory. Using a suite of tasks targeting these abilities, we evaluate state-of-the-art VLMs, including GPT-4o. Our analysis reveals distinct cognitive profiles: while advanced models approach ceiling performance on some tasks (e.g. category identification), a significant gap persists, particularly in tasks requiring spatial understanding or selective attention. Investigating the source of these failures and potential methods for improvement, we employ a vision-text decoupling analysis, finding that models struggling with direct visual reasoning show marked improvement when reasoning over their own generated text captions. These experiments reveal a strong need for improved VLM Chain-of-Thought (CoT) abilities, even in models that consistently exceed human performance. Furthermore, we demonstrate the potential of targeted fine-tuning on composite visual reasoning tasks and show that fine-tuning smaller VLMs moderately improves core cognitive abilities. While this improvement does not translate to large enhancements on challenging, out-of-distribution benchmarks, we show broadly that VLM performance on our datasets strongly correlates with performance on established benchmarks like MMMU-Pro and VQAv2. Our work provides a detailed analysis of VLM cognitive strengths and weaknesses and identifies key bottlenecks in simultaneous perception and reasoning while also providing an effective and simple solution.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the visual cognitive abilities of Vision-Language Models (VLMs), focusing on perception, attention, and memory. The authors follow a four-step approach. First, they build two datasets, PAM and CVT, using a toolkit called iWISDOM. Second, they test top-performing VLMs on these datasets to find their weaknesses. Third, they analyze the reasons behind these weaknesses. Finally, they fine-tune the models to improve their cognitive abilities. Overall, the paper builds custom datasets, uses them to test VLMs, and draws conclusions based on the results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper attempts to study the cognitive abilities of Vision-Language Models (VLMs), which is an exploratory and meaningful direction.\n\nWeaknesses:\n\n1. The motivation is not clearly presented. In the introduction, after the background section (Lines 25–41), the authors claim that \"VLMs struggle with multi-object reasoning tasks such as counting or identifying objects...\". However, the rest of the paper does not clearly explain how their method addresses the counting problem.\n\n2. The paper lacks a clear and detailed description of the PAM and CVR datasets in Section 3.2. There is no information available about the dataset statistics, the creation process of the datasets, or example cases. It’s also unclear why two separate datasets are needed. \n\n3. The description of the CVR dataset (Lines 139–145) is too brief.\n\n4. The paper claims it can \"identify specific weaknesses\" of VLMs (Line 60), but this is not clearly described.\n\n5. The claim that the paper reveals the reasoning bottleneck of VLMs (Line 62) is also not clearly described.\n\n6. Fine-tuning on a small dataset does not necessarily address cognitive reasoning issues if the training and testing follow the same distribution. To prove that fine-tuning improves general reasoning rather than just helping the model memorize the data, the train/test should be out-of-distribution data.\n\nQUESTIONS:\n1. The PAM and CVR datasets are built using iWISDOM, but the paper does not explain how they differ from the original iWISDOM dataset. It remains unclear why the existing dataset cannot be used directly. More explanation is needed to justify the creation of these new datasets.\n\n2. The perception ability of VLMs has already been extensively studied. The PAM and CVR datasets do not seem to provide significant new insights in this area. As for attention and memory, the tasks mostly involve referring to objects across single or multiple frames, which are still closely tied to perception rather than true reasoning. Therefore, the connection to reasoning ability is weak.\n\n3. There are already several works that explore the reasoning capabilities of VLMs, such as [1] and [2]. The paper should discuss these related works, clarify how its approach differs, and explain why it is necessary to create new datasets or tasks focused on reasoning.\n\n4. In Table 1, the performance of GPT-4o and human participants on CVR-Cat is confusing. Despite being labeled as a low-difficulty task, the results show performance worse than the medium-difficulty tasks. The paper lacks a detailed analysis and fails to provide convincing explanations for this inconsistency.\n\n[1] Visual Abductive Reasoning\n\n[2] The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning\n\nLIMITATIONS:\nThe paper aims to study the cognitive abilities of VLMs, but the problem definition remains more focused on perception rather than reasoning. It does not specifically address key reasoning challenges in existing VLMs, such as counting. In addition, the PAM and CVT datasets are not described in sufficient detail, and the paper lacks essential information such as dataset statistics, creation process, and sample examples. The differences between these datasets and the original iWISDOM dataset should also be clearly explained.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically investigates Vision-Language Models (VLMs) through the lens of cognitive science, breaking down their abilities into Perception, Attention, and Memory (PAM) and assessing how these core capabilities impact composite reasoning. It introduces:\n\n1. PAM and Composite Visual Reasoning (CVR) datasets, procedurally generated with iWISDM, to test fine-grained cognitive abilities.\n\n2. A vision-text decoupling analysis, using self-captioning and pre-captioning to understand if failures come from perception or integration.\n\n3. Experiments showing targeted fine-tuning significantly improves performance, especially on spatial reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. **Novel Analytical Framework:**\n   The paper reframes VLM evaluation around core cognitive primitives—**Perception, Attention, and Memory**—rather than simply task types. This perspective, inspired by cognitive science, provides clear, testable hypotheses about where and why models fail.\n\n2. **Interesting Experimental Observations:**\n   The work conducts several insightful experiments to probe specific deficiencies of current VLMs, such as the vision-text decoupling analyses that help disentangle perceptual encoding from reasoning limitations.\n\n---\n\n### **Weaknesses**\n\n1. **Limited Evaluation Data Diversity:**\n   All tasks are generated from a synthetic dataset with only 8 object categories and uniform backgrounds. This significantly limits ecological validity, as real-world images involve clutter, occlusion, and more diverse visual content.\n\n2. **Domain-Specific Testing Setup:**\n   The reliance on the iWISDM environment creates a highly specialized evaluation scenario. This makes it difficult to disentangle whether observed errors reflect true perception and reasoning deficits, or simply failures of out-of-distribution generalization.\n\n3. **Interpretation of Fine-tuning Gains:**\n   Regarding the claimed contribution (line 65), it is not surprising that fine-tuning on the target domain improves performance on tasks from that same domain. Therefore, it is hard to attribute these gains specifically to improved *diverse cognitive reasoning abilities*, rather than to domain memorization or overfitting.\n\nQUESTIONS:\nAs mentioned in the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PAM (Perception-Attention-Memory) and CVR (Composite Visual Reasoning)––two procedurally-generated suites that probe VLMs along cognitive-science axes. Seven state-of-the-art models (GPT-4o family, Qwen2.5-VL, MiniCPM-V, InternVL-2.5, LLaVA-OneVision) plus humans are evaluated. Results reveal strong object-category perception but large gaps in spatial reasoning and selective attention. A simple vision-text decoupling trick (self-captioning) substantially narrows the gap, and LoRA fine-tuning on 1k–100k CVR samples further boosts performance. PAM/CVR scores correlate tightly with MMMU-Pro, highlighting their diagnostic value.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths\n\n- Cognitive framing. Mapping VLM skills to classic P-A-M faculties is fresh and helps pinpoint bottlenecks. \n- Procedurally generated benchmarks. Tasks of unlimited size avoid test-set saturation and enable controlled ablations.\n- Comprehensive evaluation. Mix of open-source and proprietary models + human baseline yields a clear landscape.\n- Actionable insight. Self-captioning is a one-line prompt change yet delivers 18–55 pp gains on spatial tasks. \n- Practical fine-tuning study. Shows that even 1 k synthetic examples meaningfully lift location, attention, and memory scores. \n- Correlation to real-world benchmarks. Strong R²≈0.9 between PAM/CVR and MMMU-Pro argues external validity.\n\nQUESTIONS:\nQuestions for the Authors\n\n- Human study details. How many annotators? Were images shuffled to reduce practice effects?\n- Statistical significance. Are accuracy gaps tested (e.g., bootstrap CIs) or visually inspected only?\n- Caption quality. For self-captioning, do hallucinated captions ever hurt reasoning, and how is that measured?\n- Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?\n\nLIMITATIONS:\nLimitations (some acknowledged, some additional)\n\n- Synthetic bias. Only eight ShapeNet categories on plain backgrounds limit ecological validity. \n- Single-frame captioning. SC assumes each frame can be described independently; coherence across complex videos remains untested.\n- Overfitting risk. LoRA boosts CVR but slightly hurts MMMU-Pro at small data sizes; larger sets might magnify this trade-off. \n- Attention-capacity hypothesis unproven. The paper infers attention interference from SC-I drop, but no probing of cross-modal token limits is shown.\n- Missing energy/runtime analysis. Procedural tasks are cheap, yet large-scale SC + LoRA incurs extra inference cost.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses the VLM performances across three core cognitive axis - perception, attention and memory. They perform rigorous experiments across seven models spanning from open-source models to GPT-4o. The authors analyse several gaps pertains these models in tasks such as spatial understanding and selective attention. The authors state the need for improved chain-of-thought capabilities. This works presents a holistic understanding of the VLM cognitive strengths, weakness to explore the bottlenecks in simultaneous perception and reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. VLM evalaution on Perception, Attention and Memory based on human faculties presents a well cognitive grounded framework.\n2. The authors have well explored the VLM bottlenecks showing a consistent gaps in the spatial localisation and selective attention in almost all the models such as perception-loc, memory-loc and spatial-attention.\n3. The authors also replaced images with gournd truth captions  to improve Qwen2.5-VL by 55pp on perception-loc and +22pp on feature attention validating that lan can reason when the visual information is provided in a text form. \n\n\nWeakness:\n1. The generalization of the fine-tuning of LoRA is limited to Qwen2.5-VL-7B., why not other larger models?\n2. One of the major questions is relying on a single chain of thought template, the authors night explore few-shot prompting etc.\n3. The authors could have tested on unseen real-image dataset VQAv2, as there is marginal change on MMBench and MMMU-Pro for LoRA finetuned model.\n4. The authors do mention of LoRA overfitting and add dropout but might add a detailed ablation such as different dropout rates to quantify better.\n5. The paper could have presented attention heat maps for attention capacity issues.\n\nQUESTIONS:\nThe authors might refer to the weakness and answer the question presented.\n\nLIMITATIONS:\nYes the authors provided a separate section for the Limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}