What is the Visual Cognition Gap between Humans and Multimodal LLMs?
Authors: Xu Cao
Yifan Shen
Bolin Lai
Wenqian Ye
Yunsheng Ma
Joerg Heintz
Jintai Chen
Meihuan Huang
Jianguo Cao
Aidong Zhang
James Matthew Rehg
Venue: COLM 2025 (2025)
Keywords: Visual Cognition
Wechsler Intelligence Scale for Children
Multimodal LLMs
TL;DR: Exploring the visual cognition gap between Multimodal LLMs and humans.

Abstract: Recently, Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness in addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning -- the cognitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children. Inspired by the matrix reasoning tasks in Raven’s Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), we propose a new dataset MaRs-VQA to evaluate the visual cognition capability of MLLMs and compare their performance with existing human visual cognition studies. Based on the training data of MaRs-VQA, we also finetune a baseline model Qwen2-VCog with multi-stage cognition reasoning annotations. Our comparative experiments with different baselines reveal a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of MaRs-VQA and the Qwen2-VCog baseline model will drive progress toward the next generation of MLLMs with human-like visual cognition abilities. MaRs-VQA is available at huggingface.co/datasets/IrohXu/VCog-Bench. The training code of Qwen2-VCog is available at github.com/IrohXu/Cognition-MLLM.

Published as a conference paper at COLM 2025
What is the Visual Cognition Gap between Humans and Mul- timodal LLMs?
Xu Caol*, Yifan Shen'*, Bolin Lai?, Wenqian Ye², Yunsheng Ma", Joerg Heintz'
Jintai Chen Meihuan Huang", Jianguo Cao Aidong Zhang², James M. Rehg'
DDepartment of Computer Science, University of Illinois Urbana-Champaigh
2 Colleee of Computing, Georgia Institute of Technology
3 *Department of Computer Science, University of Virginia
# Digital Twin Lab, Purdue University
HKKUS (Guangzhou)
6 "Department of Rehabilitation Medicine, Shenzhen Children's Hospital (xucao2, yi fifa226, ,
jrehg Jeiehheeeiln weeeelnnnnii.... edu
Abstract
Recently, Multimodal Large Language Models (MLLMs) and Vision Lan- guage Models (VLMs) have shown great promise in language-guide perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning the cog- nitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children. Inspired by the matrix reasoning tasks in Raven's Progressive Matrices (RPM) and Wechsler Intel- ligence Scale for Children (WISC), we propose new cataset MaRs-VQA to evaluate the visual cognition capability of MLLMs and compare their performance with existing human visual cognition studies. Based the training data of MaRs-VQA, we also finetune a baseline model Qwen2- VCog with multi-stage cognition reasoning annotations. Our comparative experiments with different baselines reveal a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of MaRs-VQA and the Qwen2- VCog baseline model will drive progress toward the next generation of MLLMs with human-like visual cognition abilities. MaRs-VQA is available huggingface.co datasets Qwen2-VCog is available at atgithuub.com/ hatpi//oooonoooo///oootttonnnnii
Aittasets/Irohxu.. IrohXu Trohhu/VCoo-BBaal VVCogBeech Bench The training code of
1 Introduction
Matrix reasoning is crucial ability in human cognition. It is used in non-verbal, culture- reduced intelligence assessments because it minimizes the influence of acquired knowledge and skills (Jensen 1998 Jaeggi et al., 2010 Laurence & Macedo, 2023). Common matrix reasoning problems consist of images with simple symbols governed by underlying abstract rules MMakknnski & Mandziuk 2023) (see Figure (1.. Participants have identify and comprehend the rules based on few provided patterns, and then reason about the next pattern following the same rules. Matrix reasoning is an important reflection of many fundamental capabilities of human visual cognition, such as processing speed and working memory, that emerge in the early stages of children's neurodevelopment (Gentner 1977). It is also included in many assessment methods for fluid intelligence tests such as Wechsler Intelligence Scale for Children (WISC) (Wechsler & Kodama 1949) and Raven's Progressive Matrices (RPM) (Raven 2003
"qqual contribution.

---

Published as a conference paper at COLM 2025
GPT-o1
would choose Option C because matches the relative positions the three colored "arc" shapes in the larger 3*3 pattem. In cell of the big square, the orange "hite" shaoe sits roughly left Itited slightly toward downrright. When compere these prientations cell by cel, Opton C lines up with the posittons and spacing see repeated throughout the grld.
blue
up-right, the green
Claude 3.7 Sonnet
Looking this visual pattem recognition problem. believe answer D because completes sequence
transfomations seen across the grid where each contains a Y-sha ped structure with colored arcs and orange semi-circles in specific position;: the
A
B
option green are appears the side while orange semi-cirele of the Y-structure and the blue are remains on the right side, maintaining the consistent rotationll pattern observed in the first and third rows of the grid where green arcs systnmatically shift positions (inft battom, right) while orange elements follow complementary positioning system.
Figure 1: The example of the subpar performance of current state-of-the-ard MLLMs (GPT-ol, Claude 3.7 Sonnet) on a simple matrix reasoning task used in MaRs-VQA (similar to cases in RPM and WISC). Both models can recognize the basic shapes in the provided patterns but fail to reason the next pattern.
In psychometrics, matrix reasoning tasks for children are specifically designed to assess visual reasoning abilities without prior specialized training. Children typically approach these tests relying solely on their general cognitive skills developed from everyday interac- tions with natural environments. This raises intriguing question: Do Multimodal Large Language Models (MLLMs) exhibit visual cognitive capabilities similar to those of humans? MLLMs are trained on extensive general-domain data and have demonstrated the ability to generalize to unfamiliar tasks through in-context learning. However, current MLLMs still struggle with tasks that require advanced inductive reasoning, as evidenced by their poor performance on abstract reasoning tests such as the RAVEN TQ-test (Huang et al., 2024 Fu et al., 2024b Yiu et al. 2024). The RAVEN IQ-test itself has notable limitations, including a relatively small dataset of only 50 samples (Huang et al. 2024), potentially introducing randomness and failing to robustly evaluate MLLMs capabilities. Furthermore, it lacks comparative analyses with human performance, underscoring the need for more comprehensive and rigorous evaluation methods.
To address these gaps, we propose MaRs-VQA, new visual question answering (VQA) dataset for general-purpose MLLMs, based on psychologist-certiiee matrix reasoning items with extensive sample diversity and rigorous human reference Chierchia et al. (2019). Unlike recent works, our benchmark uniquely offers: (i) direct comparison between generic MLLMs and humans on scale order of magnitude larger than previous VQA benchmarks and with richer stimuli types; (ii) rigorous psychological validity and baseline from human subject studies; (iii) explicit dual-modality annotation-for every question, both image and matural-languar descriptions are provided for all options, enabling probing of language- vs-visual inference; (iv) full chain-of-thoughh (CoT) reasoning steps annotated, supporting deeper cognitive diagnosis and fine-tuning.
We also conduct thorough evaluation and comparison across 5 existing MILMs (including their variants) and human performance under zero-shot inference setting (no prior knowl- edge) on MaRs-VQL and another abstract reasoning datasets RAVEN containing human studies. In our experiments, we observe that MLLMs with more parameters generally perform better on our benchmark, adhering to established scaling laws in a limited scope. However, even the largest open-source MLLMs and GPT-4o fall short of surpassing human performance in matrix reasoning tasks. In conclusion, our contributions are sunmarized as follows:
2

---

Published as a conference paper at COLM 2025
We introduce new matrix reasoning VQA dataset MaRs-VQA, containing 1,440 image instances designed by psychologists, which the largest dataset for matrix reasoning zero-shot evaluation.
We conducted supervised fine-tuning (SFT) of Qwen2-VL using our annotated cognitive reasoning data. Our results indicate that fine-tuning enhances Qwen2- VL's accuracy on MaRs-VQA to match human performance levels; however, its generalization capabilities remain limited.
Our thorough experiments qualitatively reveal the visual cognition gap between MLLMs and humans in matrix reasoning problems. We also show additional insights of deficiencies in MLLMs, which can inspire more future investigations in model design.
2 Related Works
Cognitive Test of Large Language Models (LLMs) The rise of LLMs has aroused interest
in exploring human-like Al in psychology and cognition (Ullman 2023). Recent works tested
LLMs' cognitive abilities in causal reasoning (Binz & Schulz 2023), abstract reasoning (Xu
et al. 2023b Moskvichev al. 2023 jiang et al.) 2024 Ahrabian et al., 2024)), analogical
reasoning Webb et al. 2023), systematic reasoning (Hagendorfi et al. 2023),, and theory of
on the areas of MLLMs and visual cognition. Visual cognition involves the process by which cogming
mind (Strachan et al. 2024). Despite this success, only limited research has been conducted
the human visual system interprets and makes inferences about a visual scene using partial information. It is observed that while LLMs demonstrate basic understanding of physical laws and causal relationships, they lack deeper insights into intuitive human preferences and reasoning (Buchooff et al.) (2023). Almost all existing visual cognition benchmarks focus on testing MLLMs cognitive abilities in simple tasks (Zhou et al.) 2023 Jassim et al. 2023), and ignore testing complex abstract reasoning and logical reasoning ability. Theretore, new and challenging benchmarks based on the theory of visual cognition are needed to assess and improve Al systems' capabilities for human-like visual understanding.
Matrix Reasoning Matrix reasoning is often used to determine human intelligence related
to visual cogmition and working memory (Salthouse) 1993 Jaeggi et al. 2010 Fleuret et al., 2011) that widely used by RPM (Raven 2003 Soulières et al. 2009), WISC (Wechsler &
Kodama 1949
Kaufman et al. 2015) to evaluate human ability to detect the underlying
conceptual relationship among visual objects and use reasoning to find visual cues. Early research indicated that deep learning models can be trained to solve simple matrix rea- soning (Maikiñski & Mandziuk 2022 2023 Xu et al. 2023a Malkinski & Mandziuk 2024 and compositional visual relation tasks (Fleuret et al. 2011: Zerroug et al. 2022 Liu et al. 2021). Several datasets and benchmarks are also proposed, such as RAVEN (Zhang et al.
2019), RAVEN-I (Hu et al. 2021), RAVEN-FAIR (Benny et a 2021), ,, CVR (Zerroug et al.
2022). However, these works have a key limitation. They overlook the fact that humans can solve such puzzles in a zero-shot manner, without explicit training on large-scale data. Recently, there are also some useful zero-shot visual reasoning inference datasets such as RAVEN-IQ (Huang et al. 2024), Visual Reasoning Benchmark (Zhang al. 2024b), but all of them are limited by lacking rigorous human experiments as reference and conducting experiments on relatively small datasets without psychometrical validation.
Vision-Language Models Researchers have been actively investigating the utility of Vision- Language Models (VLMs) for addressing vision reasoning tasks (Zellers et al. 20019: Bordes et al. 2024). These latest VLMs are constructed using combination of the CLIP vision encoder, pretrained LLMs, and a connected adapter to align visual features with language space (Zhang et al. 2024a Shao et al. 2024 Gupta & Kembhavi 2023 Fu et al.) 2024b). Notably, methodologies such as MIn (Zhu et al. 2023), InstructBLIP (Dai et al. 2024), LLaVA (Liu et al. 2024, CogVLM (Wang et 2023) underscore the significance of employing high-quality visual instruction tuning data. Nevertheless, current VLMs encounter challenges in adapting to high-resolution and visually complex images. These problems stem from the absence of robust visual search mechanism (Wu & Xie 2023).
3

---

Published as a conference paper at COLM 2025
Dataset
Source
Sample
Instance
BCB
Itunon
Peychllngiol
Operacunc
VIA
+
losmoo-ii NNerrti (Neurt Huing 20z4
RAVEN-IQ
A
Visual Revsoning Benchmark Zhung 2024
Mensa Test, RAVEN,
A
TroeligenneTest
Mala [msr)]
Table 1: Comparison of recently released zero-shot matrix reasoning datasets to evaluate MLLMs.
few-shot reasoning (Guo et al. 2023), compositional understanding (Yuksekgonul et al. 2022) and the constrained visual grounding capabilities inherent in CLIP (Tong et al. 2024).
3 MaRs-VQA Dataset
The MaRs-VQA dataset designed evaluate the zero-shot abstract reasoning capabilities of MLLMs through various matrix reasoning VQA tasks. The images in MaRs-VQA are sourced from the questionnaires in Matrix Reasoning Item Bank, which is created by psy- chologists including 18 sets of abstract reasoning questionnaires (80 instances in each set) for non-verbal abstract reasoning assessment adolescents and adults (Chierchia et al. 2019). Each item presents incomplete 3 3 matrix of abstract shapes, requiring participants to identify relationships among the shapes. Then, we create VQA annotations the images from all questionnaires. The comparison among MaRs-VQ. and previous matrix reasoning benchmark datasets is shown in Table
To transform the matrix reasoning problem into VQA task, we define two option sets image-based set and language-based set. In the image-based set, we provide four candidates to the missing patch in the question. We further diversify the modalities of our dataset to support the evaluation of different kinds of models. Specifically, the author teams annotate language-based descriptions for each option, forming language-based set. Each option annotation is formatted by GPT-4o to ensure consistency. In this process, we first manually design 10 formatted language-based sample pairs. These examples are then used as few- shot samples to query GPT-4o through in-context learning. The context generation system prompt guides GPT-4o to re-caption all annotations. After generating all samples, human annotators in the author team review each option again and revise the incorrect description. In Table [ compared with other matrix reasoning datasets for MLLM's visual cognition evaluation, MaRs-VQA is the largest one with unique features psychological validity, human study reference, VQA annotations.
4 Problem Statement
In this section, we introduce the evaluation pipeline of MaRs-VQA under multi-image reasoning setting.
Assume that the test set contains n VQA samples, denoted as ((:;
qi represents
the question image showing the 3 matrix reasoning task. x; = [x] [x represents the images in the option set, where k is the number of options. y; is the answer of the matrix reasoning question. The inference pipeline can be formulated as:
ý: Faiqi Xsyss
(1)
4

---

Published as a conference paper at COLM 2025
relatirr ebjects,
Roo based Ligh--addern rules
What is the constroint options?
step myp-tyattp
Answar
attributes, objects of sach option.
Prtontel reletion the misiing colk
AI
A
Figure An overview of using CoT' solve matrix reasoning problem in MaRs-VQA. The left part is the model input, including question image, multiple option images. The right part shows the step-by-step CoT for multi-image reasoning for GPT series and Claude series inference.
Xsys the system prompt, including independent information about the matrix reasoning problem setting, step-by-step reasoning examples (optional), few-shot examples (optional), requirements for the output format. y: is the prediction result. is an autoregressive decoder in the MLLM for answer generation. It is defined as:
I.
X1 X,,y))
II (( Xays, Picfii)
where f is the visual encoder and adapter layer, L is the sequence length of answers and Pizej is all answer tokens before ýi,-
Methods
As we claim in previous sections, our initial goal is to complete the 3 x 3 matrix by finding the missing cell from multiple options by zero-shot prompt engineering under the same setting in human's 's matrix reasoning test. To this end, MLLMs have to deduce relationships across the other cells of the matrix and infer the missing cell accordingly. We use CoT prompt engineering to guide closed-source MLLMs solving this problem. To further promote related visual cognition foundation model, we also propose to use step-by-step structured reasoning annotations in MaRs-VOL to supervised finetune (SFT) Qwen2-VL with LoRA. Then we test the performance of Qwen2-VCog in both MaRs-VQA test set (in-domain) and a subset of RAVEN Out-of-Dommin (OOD) data. The observation in the experiment could reveal why this problem is hard for MLLM and highlight the gap between human intelligence and MILLM reasoning.
5.1 Multi-Image Reasoning via Chain-of-Thought (CoT)
Building recent insights into systematic, language-based reasoning (Wei et al. 2022 languuage
Kojima et al. 2022), we propose a straightforward yet effective division of the reasoning workflow into two distinct and clearly tagged stages (<think> and <answer>), illustrated in Figure 3 This design draws inspiration from methods such as OpenAl ol (Zhong et al., 2024), LLaVA-CoT (Xu et al.) 2024), and R1-V Chen et al. (2025) where each stage contributes different level of abstraction to the overall interential process:
Reasoning (<think>): It includes: (i) concise overview of the task (e.g., 'examining a 3x grid puzzle and determining the missing cell';; (ii) a structured description of relevant visual elements (color, shape, position, etc.) that guide the reasoning; and (iii) a methodical analysis of the discovered pattern(s). Crucially, this step covers both rule identification (the model pinpoints how objects or elements follow consistent patterns) and option verification (each candidate option is tested against the identified rule).

---

Published as a conference paper at COLM 2025
Conclusion (<answer>): A single, succinct statement that specifies the best or correct choice among the provided options. No extra explanation is given here. simply states which option is correct as the final answer.
This two-section format promotes a structured and transparent rea- soning style by first concisely out- lining the puzzle, then detailing all pertinent visual characteris- tics, followed by explicit discov- ery and testing of candidate rules, and finally isolating the single cor- rect response. To further clar- ify the CoT processes in LLM- based reasoning, our method an- notates each stage of the two- section format with dedicated tags, such as <think>... <//think> and <answer>... ./ans/err///swrr.. By explic- itly marking the beginning and end of each reasoning stage, the model is guided to retain clarity and precision throughout the en- tire solution path. Unlike the tradi- tional free-form CoT that allows the model to produce unconstrained self-talk, our approach enforces well-structured methodology. A detailed template demonstrating this two-section CoT format is provided in our code repository.
Question
Visual Encoder
Autoregressive Decoding
LLM
Option
<tinnk... </think>
<reasoning reasoning
Figure 3: Supervised fine-tuning VLM to generate two- section format for matrix reasoning problem.
5.2
SFT for Vision-Language Model (VLM)
To enhance the reasoning capabilities vision-language models (VLMs), we leverage the reasoning responses generated using the two-section CoT format training data (in Figure 3). LoRA Hu et al. (022) is employed to fine-tune Qwen2-VL, enabling efficient adaptation of the model to these structured reasoning tasks. Specifically, the step-by-step reasoning annotations in MaRs-VQA are used as supervision signals during fine-tuning. After training, the performance of the resulting model, Qwen2-VCog, is evaluated on MaRs- VQA test set for in-domain performance and a subset of RAVEN for nnt-of-Domain (OOD) performance.
To further probe the contribution of explicit cognitive supervision, we conduct an ablation study by fine-tuning variant of Qwen2-VL without the step-by-step reasoning annotations, in parallel to Qwen2-VCog (which was trained with full reasoning chains). This allows us to isolate the effect of reasoning supervision on both in-domain (MaRs-VQA) and out-of- domain (RAVEN) performance.
Experiments
6.1 Experimental Settings
Datasets For MaRs-VQA, We split 480 VQA samples as the test set and the rest of them is the training set for SFT. In addition to MaRs-VQA, we also select 560 VQA pairs serving as OOD samples from matrix reasoning dataset RAVEN (Zhang et al. 2019). Details of difference between MaRs-VQA and RAVEN are shown in Table 9
in appendix.
MLLM Baselines We selected the Claude 3 Sonnet, Claude 3 Opus, Claude 3.5 Sonnet (An-
thropic 2024), GPT-4V (OpenAl 2023), GPT-4o (OpenAl, 2024b), GPT-ol (OpenAl 2024c), LLa VA-NEXI (Liu et 2024), Qwen2-VL VL (Wang et al. 2024), Intern VL-
Chen et al. (2024
as the primary multi-image CoT baselines as they support multiple images input and can

---

Published as a conference paper at COLM 2025
Accuracy (%)
Method
Model Scale
MaRs- VQA (4-options)
RAVEN (8-options)
Random Select
25.00
12.50
LLaVA-NExT (Liu et al.
2024
7B
16.88
14.29
InternVL-2.5
2024
8B
20.00
13.21
Qwen2-VL Wang ef al.
2024
7B
23.75
29.27
Claude Sonnet [Anthropic
2024
23.22
13.39
Claude 3 Opus (Anthropic
2024
24.13
11.95
Claude 3.5 Sonnet [Anthropic 2124
24.28
15.36
GPT-VV (OpenAl 2025
33.13
15.63
GPT-4o (OpenAl 2024b
33.96
25.89
GPT-ool (OpenAT] 20Z43
52.29
25.36
Qwen2-VCog (Our SFT Baseline)
7B
72.71
31.96*
Human
81.00
84.41
Table 2: Benchmarking experiments on multi-image reasoning in MaRs-VQA (in-domain) and RAVEN (OOD). zero-shot means only provide the model system prompt about the matrix reasoning task definition. CoT denotes the implementation in section 5.1 The results averaged over three runs with three different random seeds. of Qwen2-VCog denotes CoT performance without finetuning.
generate reasoning process. As Qwen2-VL (Wang et al., 2024) is the best open-sourced VLM for zero-shot inference in T we choose it as the main backbone to finetune our baseline Qwen2-VCog with MaRs-VQA training data.
Human Baseline The human study results in Table 2 are reported from previous experi- ment results. The human subjects of RAVEN (Zhang et al. 2019) consists of college students from a subject pool maintained by the Department of Psyychoooyy. Only "easily perceptible" examples were used in the investigation. The human study results of MaRs-IB (Chierchia et al. 2019) are more rigorous. They are from 4 age groups (N = 659, aged 11-33 years). The accuracy for younger adolescents, mid-adolescents, older adolescents, and adults solving matrix reasoning in MaRs-IB are 61 %, 68,, 73%, 81% %. We use the adult result 81% in Table
Implementation For closed-source baseline models, we establish basic prompts to intro- duce the matrix reasoning problem setting, which serve as the system prompt for zero-shot inference. For open-source baseline models, we use the same system prompt settings across all models. Testing is conducted using two NVIDIA H100 GPUs for all models. All ex- periments are run with three different random seeds, and the results are averaged. We evaluate the results based on the accuracy of single-option matrix reasoning problems (Acc = Correct/ Total), consistent with other VQA benchmarks (Lu et al. 2022 Liu et al., 2023
6.2
Experimental Results
For all models in Table 2 we used multiple images as the input, including a question image and several option images, and guided the MLLMs decompose the problem into predefined structures before generating answers based on all available information. We tested the latest closed-source models like Claude 3, Claude 3.5, GPT-4V, GPT-4o, and GPT-ol for this task, as these models can generate step-by-step multi-image reasoning, In addition, we also compare them with small size open-sourced models like Qwen2-VL, Intern Our results show that even the state-of -aate-of-the-art closed-souree MLLMs GPT-o1 perform worse than humans in all matrix reasoning tasks.
After analyzing the reasoning outputs of current MLLMs, we identified three primary issues: (1) Limited Use of Visual Information: MLLMs struggle to directly utilize visual features during reasoning, rendering them insensitive to non-verbal spatial details, particularly evi- dent when interpreting positional relationships within images. For instance, distinguishing among the options in Figure 1 is challenging for MLLMs using language alone. (2) Restricted Visual Working Memory: MLLMs exhibit limited visual working memory, leading to rapid loss of crucial visual information during text-based reasoning processes. (3) Integration
7

---

Published as a conference paper at COLM 2025
Strategy
Accuracy (%)
GPT-4o+ CoT
33.96
Table 3: Ablation on few-shot sample CoT and multi-round CoT for GPT-4o in MaRs- VQA. GPT-ol still outperforms GPT-4o with different CoT strategy.
4P CoT 1-shot
35.22
GPT-40+ CoT 3-shot
36.10
GPT-40+ CoT 5-shot
36.03
GPT-1o+ multi-round CoT
41.96
GPT-4o+ mulli-round CoT 1-shot
42.08
GPT-o1
52.29
Despite excelling at specific visual tasks like recognition, segmentation, and ob- ject detection, MLLMs encounter significant difficulties integrating these skills for high-level visual reasoning tasks. Further examples illustrating GPT-4o's failure cases and discussion are provided in the Appendix.
Challenges:
To address these shortcomings, we leveraged the reasoning annotations described in Sec- tion 5.7 fine-tune Qwen2-VL, using Low-Rank Adaptation (LoRA) as our baseline model for MaRs-VoR VQA. After fine-tuning on the MaRs-VQA training set, the model achieved over 70% accuracy on the MaRs-VQA test set, nearing human adult performance. Additionally, its performance on the OOD RAVEN dataset improved from 29.27% to 31.96%. This ob- servation shows that VLM could exploit shortcut features from matrix reasoning problem. tends to perform well on the in-distribution test sets while slightly improving the out- of-distribution test performance. This indicates that there is still major gap for current MLLMs to learn intended features in the abstract reasoning tasks. The simple strategy of SFT cannot entirely solve this task. This also demonstrates the value of our proposed benchmark in examining the out-of-domain nature of abstract reasoning tasks.
6.3 Ablation Study
In this subsection, we conduct ablation experiments to analyze how to improve the zero-shot performance of MLLMs on the matrix reasoning problem.
Effect of CoT Strategy for Closed-source MLLMs Table compares the different CoT strategy: step-by-step CoT with hint, CoT reasoning with few-shot sample and multi- round CoT reasoning. Few-shot samples are a small number of question-answer examples alongside the CoT system prompt. Multi-round reasoning employs the advanced multi- round CoT strategy (step by step option elimination). After each round, the model will reflect on the correctness of the answer and run the reasoning steps again the answer is wrong. The results show that incorporating 1-shot and 3-shot samples gradually in- creases the accuracy of GPT-4o on LaRs-VQA from 34% to 36%. However, extending the number of examples to 5 does not yield further improvement. These findings suggest that while few-shot in-context learning helps the model better understand the matrix reasoning problem, it does not significantly enhance the MLLM's visual reasoning capabilities for these tasks. Additionally, using a multi-round elimination strategy improves accuracy from approximately 34% to 42%, but it is considerably slower than single-round CoT, and still cannot surpass GPT-ol (52.29%) and human adults (81.00%).
Effect of Reasoning Strings in SFT To more precisely quantify the impact of explicit step-by-step reasoning supervision in Supervised Fine-Tuning (SFT), we conducted an ablation study in which the reasoning strings were omitted from the MaRs-VQA training data. As reported Table 4 the absence of these structured cognitive annotations led to a dramatic decrease in accuracy for Qwen2-VCog on both MaRs-VQA and RAVEN. Specifically, performance on MaRs-VQA dropped by nearly 18 percentage points, while the accuracy on RAVEN also decreased, albeit to a lesser extent. This result highlights that detailed reasoning guidance not merely auxiliary, but instead serves as a crucial signal for the model to acquire transferable visual cognitive skills. Without such supervision, the model tends to rely on shallow pattern matching or spurious correlations, failing to generalize robustly even within the same task domain. These findings underscore the necessity of incorporating high-quality, stepwise annotations for training MLLMs to approach human-like reasoning on complex visual cognition benchmarks.
8

---

Published as a conference paper at COLM 2025
Model
MaRs-VQA (%)
RAVEN (%)
Owen2-VCop (with reasoning)
72.71
31.96
Qwen2-VCog (w/o reasoning)
54.82
29.30
Table 4: Effect of reasoning strings in SFT.
Vision vs. Language Bottleneck We further investigated whether the primary limitation of current MLLMs matrix reasoning tasks arises from their visual perception modules or from downstream language-based reasoning. To disentangle these factors, we designed three input conditions: (1) providing only the question image and selecting options via CLIP similarity, (2) providing both the question and all option images (the standard VQA setting), and (3) supplementing the images with perfect auman-annotated textual descriptions for both the question and options. As shown in Table both GPT-4o and Qwen2-VCog perform at near chance level when deprived of option images, indicating a failure to extract the underlying visual rules from the question image alone. When option images are provided, accuracy increases substantially, especially for Qwen2-VCog, which demonstrates strong visual pattern matching. Notably, the addition of ideal textual descriptions yields only marginal further improvement, suggesting that the language reasoning component is not the main bottleneck once high-quality visual features are available. These results collectively point to visual pattern extraction-rate rather than linguistic inference-as as the principal limiting factor for MILMs on abstract visual reasoning tasks, emphasizing the need for stronger visual encoders and better integration of visual working memory.
Input Setting
GPTT-0 (%)
Qwen2-VCog (%)
Question image only
24.58
26.32
Question image Option images
33.96
72.71
Question image Option images Option Description
36.46
71.43
Table 5: Vision vs. Language Bottleneck Analysis.
Generalizability To ensure that the improvements observed from SFT on MaRs-VQA do not come the expense of general visual-language understanding, we systematically evaluated both Qwen2-VCog and the original Qwen2- VL a suite of standard multimodal benchmarks, including MME (Fu al. 2024a), HallusionBench (Guan et al.) 2024), POPE [Li et al., 2202)), VQAv2 (validation set) (Goyal et al. 2017), SQA Image (Lu et al. 2022), and SeedBench (Li et al., 2024). As summarized identical performance across all tasks, with only negligible fluctuations that are well within the range of experimental noise. This demonstrates that reasoning-focte focusee finetuning on MaRs-VQA does not degrade the model's ability to perform generic vision-language tasks, nor does it induce catastrophic forgetting. Consequently, is feasible to endow MLLMs with enhanced visual reasoning skills via SFT on MaRs- without sacrificing their broader applicability or real-world utility.
Table 6 the two models exhibit nearly
Model
MME
Hallusion Bench 1
POPE 1
VQAv2_ Val
SQA Image f
SeedBench f
Owen2-VL-70
1666
53.83 55.73
88.84 88.71
79.88 79.45
83.73 83.29
69.16
Qwen2-VCog-7B
1680
68.87
Table 6: General VLM Benchmarks: SFT does not degrade generic ability.
6.4 Visualization
We also analyze the relationship between matrix reasoning accuracy and model scale in Figure 4 The figure illustrates the significant gap between MLLM's matrix reasoning performance and that humans. This gap is substantial and suggests that simply increasing model size according to scaling laws will not be sufficient to bridge it.

---

Published as a conference paper at COLM 2025
MaRs-Vor -VQA Acc
25%
50%
75%
100%
Homan (Addlt)
Human
Glosad-ae source
Models
6PT
Sweet
Open-tourca source
Medels
Quar2--Cag-7 78
Figure 4: There is still a substantial gap between MLLM's (zero-shot CoT or SFT training) matrix reasoning capability and human's (zero-shot). Bubble size corresponds to the model size. As we don't know the exact size of closed-source MLLMs, we set all of them to the largest value by default. The model size of human refers to the number of neurons (86B) in human's 's brain (Voytek 2013).
Discussion
In this work, we emphasize that zero-shot matrix reasoning is crucial testbed for human- level intelligence, even though the developmental origins of this ability in children remain unclear. Remarkably, children as young as four can solve matrix reasoning problems without explicit training, highlighting the unique strengths of human visual cognition. Our long- term goals are twofold: (1) to rigorously evaluate how close MLLMs are to human-like cognitive abilities, as posed by Chollet (2019); and (2) to develop MLLM-powered agents capable of human-level zero-shot matrix reasoning, which could in turn generate novel assessment tools to help psychologists and pediatricians understand neurodevelopment.
Our findings reveal a persistent visual cognition gap between current MLLMs and humans, even as model scale increases. Detailed ablation and analysis suggest that the primary bottleneck lies in visual pattern extraction and working memory, rather than language reasoning. This gap has concrete implications for real-world applications: limits the reliability of Al in dynamic or safety-critical environments (e.g., robotics, scientific discovery, education), where robust abstract visual reasoning and generalization are essential. Bridging this gap will require advances in both data and architecture, particularly in strengthening the visual encoders and multimodal integration of ML..Ms. Our benchmark and insights lay the groundwork for future research toward more human-like, generalizable visual cognition in Al systems.
8 Conclusion
We introduce MaRs-VQA, a publicly available matrix reasoning Visual Question Answering (VQA) dataset specifically designed to evaluate the visual cognitive capabilities of Multi- modal Large Language Models (MLLMs) and compare them with humans. Our findings indicate that state-of-the-art MLLMs, such GPT-4o, Qwen2-VL, and InternVL-2.5, demon- strate foundational competence in matrix reasoning but continue to struggle with more complex or abstract scenarios, performing substantially below human levels. Supervised fine-tuning (SFT) with cognitively designed step-by-step reasoning annotations from MaRs- VQA can significantly boost in-domain accuracy, yet these models still fall short of human performance and generalize poorly to out-of-domains (OOD). Notably, humans achieve strong performance without any task-specific training, underscoring the inherent gap in visual cognition between humans and MILMs. Our ablations further show that explicit reasoning supervision is crucial, and that vision-not
cacuuugee-emmnnn the dominant bottleneck. Bridging this gap will require continued research and innovation in both model architecture and multimodal learning paradigms, ultimately advancing the visual cognitive abilities of future MLLMs.
10

---

Published as a conference paper at COLM 2025
References
Kian Ahrabian, Zhivar Sourati, Kexuan Sun, Jiarui Zhang, Yifan Jiang, Fred Morstatter, and Jay Pujara. The curious case of nonverbal abstract reasoning with multi-modal large language models. arxiv preprint ar XrXi::2401.2111, 2024.
Anthropic. Introducing the next generation of claude. https://ww.antirooicc.coo/////// cllude-3--faml 2024.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12557-12565, 2021.
Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523112) 2023.
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, al. An introduction to vision-language modeling. ar preprint ar Xix::2455.17247 2024.
Luca M Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. Have we built machines that think like people? arXiv preprint arXie:2311.1609,, 2023.
Raymond Bernard Cattell and Alberta KS Cattell. Measuring intelligence with the culture fair fests. Institute for Personality and Ability Testing, 1960.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super gen- eralization ability in vision-language models with less than $3. https://githh: thub. com/ Deep-Agent/RI-V/ 2025. Accessed: 2025-02-02.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint axXio:2412.05271, 2024.
Gabriele Chierchia, Delia Fuhrmann, Lisa J Knoll, Blanca Piera Pi-Sunyer, Ashok L Sakhardande, and Sarah-Jayne Blakemore. The matrix reasoning item bank (mars-ib): novel, open-access abstract reasoning items for adolescents and adults. Royal Society open science, 6(10):190332, 2019.
François Chollet. On the measure of intelligence. arXiv preprint arXie:1911.01547, 2019.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-languaag models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
François Fleuret, Ting Li, Charles Dubout, Emma Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans visual categorization test. Proceedings of the National Academy of Sciences, 106(43):17621--16655 2011.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. /ttpsstss///axiv org/abs/2386. 13394
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. Xiv preprint ar XXXx:24444.12900 2024b.
Dedre Gentner. Children's performance on a spatial analogies task. Child development, pp. 1034-1039, 1977.
11

---

Published as a conference paper at COLM 2025
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 14375-14385, 2024.
Qing Guo, Prashan Wanigasekara, Jian Zheng, Jacob Zhiyuan Fang, Xinwei Deng, and Chenyang Tao. How do large multimodal models really fare in classical vision few-shot challenges? deep dive. In RO-FoMo: Robustness of Few-shot and Zero-shoo Learning in Large Foundation Models, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14953,1496,, 2023.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3(10):833-88) -33-888,, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1567-1574, 2021.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2024.
Susanne M Jaeggi, Barbara Studer-Luethi, Martin Buschkuehl, Yi-Fen Su, John Jonides, and Walter J Perrig. The relationship between n-back performance and matrix reason- ing-implications for training and transfer. Intelligence, 38(6)622--33,, 2010.
Serwan Jassim, Mario Holubar, Annika Richter, Cornelius Wolff, Xenia Ohmer, and Elia Bruni. Grasp: novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. arXiv preprint arXio:2311.09048, 2023.
Arthur R Jensen. The factor. Westport, CT: Prager, 1998.
Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, and Jay Pujara. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. arv preprint ar drXiv:2404.1359,, 2024.
Alan Kaufman, Susan Engi Raiford, and Diane L Coalson. Intelligent testing with the WISC-V. John Wiley & Sons, 2015.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, (1522199-222111 2022.
Paulo Guirro Laurence and Elizeu Coutinho Macedo. Cognitive strategies matrix- reasoning tasks: State of the art. Psychonomic Bulletin & Review, 30(1):147-159, 2023.
12

---

Published as a conference paper at COLM 2025
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13299-13308, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arviv preprint arXiv:2305.10355 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba, Learning to compose visual relations. Advances in Neural Information Processing Systems, 34:23166-233 23178, 2021.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi multi-modal modal model an all-around player? ar arv preprint arXi::2307 066281, 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Mikolaj Małkiński and Jacek Mandziuk. Deep learning methods for abstract visual reasoning:
A survey on raven's progressive matrices. arXiv preprint ar arXiv:2201.12382, 2022.
Mikotaj Malkiński and Jacek Mandziuk. A review of emerging research directions in abstract visual reasoning. Information Fusion, 91:713-736, 2023.
Mikolaj Malkiński and Jacek Mandziuk. One self-configurable model to solve many abstract visual reasoning problems. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 14297-14305, 2024.
Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The con- ceptare benchmark: Evaluating understanding and generalization in the arc domain. Transactions on Machine Learning Research, 2023.
OpenAI. Gpt-4v(ision) system card. 2023.
OpenAL. Learning to reason with 1lms.
https://openai.com/innea
Learning-tos to-reeson- whimshhls// 2024a.
OpenAL. Hello gpt-4o. https::/opemaii.com//dee/////ee gpt- 2024b.
OpenAl. Introducing openai ol. httpps://openai.com/// 2024c.
Jean Raven. Raven progressive matrices. In Handbook of nonverbal assessment, pp. 223-237. Springer, 2003.
Timothy A Salthouse. Influence of working memory on adult age differences in matrix reasoning. British Journal of Psychology, 84(2):171199, 1993.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. Xiv preprint ar xrXiv2400.11699, 2024.
Isabelle Soulières, Michelle Dawson, Fabienne Samson, Elise B Barbeau, Cherif P Sahyoun, Gary E Strangman, Thomas A Zeffiro, and Laurent Mottron. Enhanced visual processing contributes to matrix reasoning in autism. Human brain mapping, 30(12):4082-410 2009.
James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, pp. 1-11, 2024.
13

---

Published as a conference paper at COLM 2025
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint (rXiv:23022.8839 2023.
Bradley Voytek. Are there really as many neurons in the human brain as stars in the milky way. Scitable, Nature Educalion, 2013.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arv preprint arXiv:2409. 1922999. 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. Xiv preprint ar arXit:2311.00007 2023.
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 709)1126--1111 2023.
David Wechsler and Habuku Kodama. Wechsler intelligence scale for children, volume 1. Psychological corporation New York, 1949.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Xiv preprint arXiv:2206.07662, 2022.
Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal lims. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. ar Xiv preprint ar Xxx::2411.10440) 2024.
Jingyi Xu, Tushar Vaidya, Yufei Wu, Saket Chandra, Zhangsheng Lai, and Kai Fong Ernest Chong. Abstract visual reasoning: An algebraic approach for solving raven's 's progres- sive matrices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6715-6724, 2023a.
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. Xiv preprint arXiv:2305.18355 2023b.
Eunice Yiu, Maan Qraitem, Anisa Noor Majhi, Charlie Wong, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal models. Xrv preprint arXio:2407.17773, 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in neural information processing systems, 35.29776-29788, 2022.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317-5327, 2019.
14

---

Published as a conference paper at COLM 2025
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. aiv preprint arXio:2401.13601, 2024a.
Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. How far are we from intelligent visual deductive reasoning? ar arviv preprint 2i0::2403.04732, 2024b.
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai ol: Opportunities and challenges of agi. arXiv preprint ar Xiix:2409.18455 2024.
Liang Zhou, Kevin A Smith, Joshua B Tenenbaum, and Tobias Gerstenberg. Mental jenga: A counterfactual simulation model of causal judgments about physical support. Journal of Experimental Psychology: General, 152(8):2237, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Xiv preprint arX0o:2304. 10592, 2023.
15


Reviews:
[{"review_id": "9HapQdp7FF", "reviewer": "Reviewer_1UB9", "summary": "This work tries to reveal the cognition process of Vision Language Models (VLMs), especially focusing on the relationship between high-level multi-image reasoning and visual working memory. For this purpose, the authors introduce matrix reasoning, which requires VLMs to discern relationships among patterns in a set of images and predict subsequent patterns. Furthermore, this work provides a new dataset, MaRs-VQA, which is based on the matrix reasoning task. By fine-tuning Qwen2-VL with this dataset, the model outperformed humans, and detailed analysis shows the deficiencies of the current VLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The newly created matrix reasoning dataset, MaRs-VQA, includes novel characteristics not covered by previous studies.\n- MaRs-VQA includes more instances than the conventional datasets.\n- The authors propose a new Chain-of-Thought (CoT) prompt specific for this task.\n- Experiments cover both CoT and SFT settings.\n- Fine-tuned Qwen2-VL outperformed humans and shows the effectiveness of the proposed SFT procedure, which relies on the CoT prompt.", "reasons_to_reject": "- The matrix reasoning task itself has already been used in machine learning fields.\n- The ablation study only covers the proprietary models.", "questions_to_authors": "- Could you also report the ablation study results of Qwen2-VCog?", "extra_scores": {}}, {"review_id": "jO02YNSkPw", "reviewer": "Reviewer_dTKc", "summary": "This paper discusses a psychology task, matrix reasoning, and introduces a new dataset: MaRs-VQA, to evaluate the gap between MLLM capabilities and human-level cognition. A comprehensive benchmarking results are provided, including Claude 3, GPT-4V, GPT-o1, etc, and fine-tuned Qwen2-VL.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Matrix reasoning is a novel and interesting testing case for MLLM's ability.  This paper leverages psychology dataset to create the MLLM-related dataset and benchmark.\n\n- I really appreciate the discussion about difficulty levels in MaRs-VQA in Table 4 and 5, which provides very insightful analysis for matrix reasoning question.", "reasons_to_reject": "- A dataset consisting of 1,440 image is relatively small, especially only 480 VQA are used as the test set. The benchmarking results may contain high variance. \n\n- Although finetuning Qwen2-VL on MaRs-VQA obtains the best number(besides human level) in Table2, is it an overfit or does it have negative impact on Qwen2-VL's original MLLM abilities?\n\n- Have the authors studied the underexpressive performance of MLLM on matrix reasoning is mainly limited by its vision ability or LLM ability ? For example, limitation in vision ability means it can't recognize the patters accurately, like its visual grounding ability or its small object detection is not satisfying.", "questions_to_authors": "Although this paper still has several open questions to be answered, I tend to recommend to accept this paper for its introducing a new testing dimension for MLLM: Matrix reasoning/IQ-test, and it may inspire the community to re-think about the limitations of current status of MLLM.", "extra_scores": {}}, {"review_id": "6kA9o6QLx5", "reviewer": "Reviewer_9Mv5", "summary": "This paper presents a new benchmarking resource compiled out of an existing resource used by psychologists, which brings so called \"matrix tests\" (\"which of these images best fits the gap in the image?\") into a format so that they can be put to vision models. The paper finds that current models, and even a model fine-tuned on this kind of data, fall short of human performance on this type of task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting resource, usefully compiled and prepared from existing dataset", "reasons_to_reject": "- ultimately, relatives restricted contribution: a reformatting of an existing resource, and a couple of evaluation runs\n- a bit unclear what the experiments together show, how they were selected, what exactly we learn from the experiments", "questions_to_authors": "Line 89, the sentence seems to be corrupted.\n\nWhy is section 4 entitled \"problem statement\"? (And why does the first sentence directly talk about an \"evaluation pipeline\"? It seems that there is something missing here, something like \"the task for the model then is to reach a high accuracy in predicting the correct y_i\". \n\nLine 202: \"are showed in appendix\"; Table number missing", "extra_scores": {}}, {"review_id": "UCGpVnlwKf", "reviewer": "Reviewer_1KjF", "summary": "This paper presents MaRs-VQA, a new benchmark for evaluating the visual reasoning capabilities of Multimodal Large Language Models (MLLMs) using data derived from the Matrix Reasoning Item Bank. The benchmark is designed around visual question answering (VQA) tasks inspired by classic matrix reasoning problems, with a particular focus on challenging MLLMs to capture abstract visual patterns. The authors propose a Multi-Round Chain-of-Thought (CoT) strategy to enhance reasoning across multiple visual inputs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed Multi-Round CoT strategy is an interesting addition that may inspire future adaptations of step-by-step reasoning in MLLMs.", "reasons_to_reject": "- The dataset contribution is limited in originality: all the visual problems are repurposed from existing resources, leading to a marginally novel benchmark.\n- The analysis lacks depth; the paper fails to provide novel insights beyond what prior work has indicated regarding the performance gap between MLLMs and humans on visual reasoning tasks.\n- There is insufficient exploration of why the fine-tuned model shows major gains on the in-domain MaRs-VQA set but limited improvement on RAVEN—the discussion could have shed light on potential overfitting, domain shift, or symbolic reasoning limitations.\n- The paper does not meaningfully interpret what the gap between MLLMs and human performance implies for real-world tasks or downstream applications—this would have been valuable for the COLM audience.", "questions_to_authors": "Please refer to the weakness section.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors challenge vision+language models with reasoning tasks from RPM and WISC. They find that while their finetuned model performs best, all models significantly underperform humans, and argue that current scaling approaches may not bridge the gap.

The main reviewer concerns were about novelty/the magnitude of the contribution. Is this paper just reformatting an existing task+running prompts (e.g., 1UB9 points out that these types of tasks have already been considered in the ML community)? The authors respond theirs is the first to consider general purpose reasoning LLMs; is more rigorous/has a new question/answer format; and the authors propose a fine-tuning method.

Additional unresolved points:

- dTKc asks if the llm or the vision model is the bottleneck, and the author response doesn't fully address this question (the additional experiments take a first step towards this question by providing (ground truth?) visual aspects as text, but the details/how it fits in with the main manuscript is TBD)
- 1KjF wonders about why improvements happen on one dataset more than the other, and initial experiments in response similarly take just a first step towards answering this question.

Overall, the work provides a valuable re-formatting of some longstanding/existing cognitive science tasks, and some updated insights regarding evaluation/training of modern vision+language models.

Despite the claim in the paper, there's significant work in the space of cognitive-driven evaluation of MLLMs (KiVA, BLING, and KiloGram are just some recent examples that come to mind). It's strongly recommended that the authors situated the work more accurately and completely. 
Decision: Accept