The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models
Authors: Kian Ahrabian
Zhivar Sourati
Kexuan Sun
Jiarui Zhang
Yifan Jiang
Fred Morstatter
Jay Pujara
Venue: COLM (2024)
Keywords: multi-modal large language models, nonverbal abstract reasoning, in-context learning, raven's progressive matrices
TL;DR: A study of nonverbal reasoning abilities of multi-modal large language models using variations of Raven's Progressive Matrices.

Abstract: While large language models (LLMs) are still being adopted to new domains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs). These models integrate verbal and visual information, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities. However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices. Our experiments reveal the challenging nature of such problems for MLLMs while showcasing the immense gap between open-source and closed-source models. We also uncover critical shortcomings of visual and textual perceptions, subjecting the models to low-performance ceilings. Finally, to improve MLLMs' performance, we experiment with different methods, such as Chain-of-Thought prompting, leading to a significant (up to 100\%) boost in performance.

Published as a conference paper at COLM 2024
The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models
Kian Ahrabian; Zhivar Sourati", Kexuan Sun", Jiarui Zhang, Yifan Jiang, Fred Morstatter & Jay Pujara Information Sciences Institute University of Southern California Marina Del Rey, CA 90292, USA
(ahrabian, souratih, kexuansu, jzkuang,,, yyjaangg7.yyiann4444ouus.. edu [fredmors, (epujara]@isi.. .du
Abstract
While large language models (LLMs) are still being adopted to new do- mains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs). These models integrate verbal and visual infor- mation, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities. However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reason- ing abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices. Our experiments reveal the challenging nature of such problems for MLLMs while showcasing the immense gap between open-source and closed models. We also uncover critical shortcomings of visual and textual perceptions, subjecting the models low-performance ceilings. Finally, to improve MLLMs' performance, we experiment with different methods, such as Chain-of-Thonn Thhought prompting, leading to significant (up to 100%) boost in performance. Our code and datasets are available at https:/ / http//gttthhhott/ www.abccccuussssss1111 wtt.:///mniiiiiiiiiiiimmmm...... isi-mmmm-rpm
1 Introduction
Foundation models - mostly large language models (LLMs) and large vision models (LVMs) have revolutionized the field of artificial intelligence, demonstrating zero-shot (Radford et al. 2019 Wang et al. (202a)) and few-shot (i.e., in-context) learning abilities (Brown et al. 2020 Zhang al. 2023b) that perform on-par even surpass humans in some tasks (Webb et al. 2023). These tasks cover both dimensions of general intelligence: crystallized in- telligence focusing on retrieving knowledge from memory (Hartmann al.
2023 and
fluid intelligence involving novel and abstract reasoning (Cattell) 1987). Following these advancements, there has been a recent surge in the development of new generation of foundation models, namely, multi-modal large language models (MLLMs). These models can process visual and textual cues (OpenAï 2023 Zhao et al., 2023 Huang et al. 2023), paving the way for solving far more complex tasks concerning both modalities.
Nonverbal abstract reasoning been studied extensively for measuring fluid intelligence (Shakeel et al.) (017), in which reasoners need to demonstrate strong visual perception and high-level explicit reasoning abilities to solve these tasks. Prior studies have explored the performance of LVMs (Zhuo & Kankanhalli 2020) and LLMs (Hu et al., 2023 Webb et al., (023) on transformed versions of these tasks in uni-modal setting. However, theoretical and empirical evidence exists for the benefits of the interplay between verbal and visual perceptions (Vyshedskiy 2019
family of tasks that involve both modalities. It has
Winawer et al. 2007
Vygotsky 1962), suggesting that visual perception helps better
Authors contributed equally.

---

Published as a conference paper at COLM 2024
understand our surroundings, while language helps symbolize and facilitate reasoning through notions such as self-talk (Berk) 1994).
Inspired by prior works (Vygotsky 1962 Lupyan 2012 Nam et al. 2017 Colas et al. 202 that have explored the fusion of visual and verbal cues and their eftect on cognuiion and reasoning abilities, and considering the opportunity of experimenting with both modalities in MLLMs, in this study, we strive to answer the following research question: "Do MLLMs demonstrate faithful nonverbal abstract reasoning abilities?"
Our contributions are as follows:
Prompt
1. We evaluate the nonverbal abstract reasoning abilities of 24 open-source and closed-source MLLMs under three Raven's Progressive Matrices (RPM) (Raven] 2003) benchmarks (Seereiguee for an example);
We evaluate MLLMs' textual and visual abilities in semi-isolated set- tings that mitigate cross-modality contamination, providing insights into their performance ceiling.
You are given a puzzle. The puzzle features a set visual patterns arranged piece missing. and six options at the bottom (marked by 1, 3. 4, 5, or 6). Which option (either 1, 2. 3, 4, 5, or 6) fills the missing piece best? Briefly state your reasoning.
top, with
right
3.
We evaluate MLLMs' zero-shot and few-shot abilities, drawing more accurate picture of the alignment be- tween their verbal and visual per- ceptions;
Response
The puzzle's solution requires identifying consistent pattern across rows and columns. Here's the reasoning:
All in all, we observe that while open- source MLLMs perform poorly non- verbal abstract reasoning tasks, closed- source models such as GPT-4V (Ope- nAI. 2023) showcase non-trivial abil- ities (See Section 4). Moreover, discover critical shortcomings in both visual and verbal capabilities across open-source and closed-soure models, partially explaining the observed poor performances (See Section 5). Finally, we find closed-source models' textual and visual perceptions to be relatively aligned, allowing us improve their performance significantly by providing guided prompts and in-context demon- strations (Sctionn 6).
Top row: Solid blue shapes. Bottom row: Patterned yellow line shapes.
Left column: Triangles.
Right column: Diamonds.
The missing piece
the bottom right comerr which should follow the bottom row's pattern of having patterned yellow lines and the right column's pattemn of being a diamond. Among the options provided, only option
this description: diamond shape with patterned yellow lines. Therefore, option is the correct answer.
Figure An example of model's prediction on sample from the IQ50 dataset. Given prompt with a visual puzzle (top), the model generates response that includes its reasoning and the chosen option.
Related Work
Foundation Models' Reasoning Abilities.
With the advancements of large pre-trained
(i.e., foundation) models (Vaswani et al., 2017 Dosovitskiy et al. 2020), researchers have
extensively evaluated their reasoning abilities (Bubeck al. al.] 2023). These evaluations go
beyond simple knowledge retrieval (Zhu et al. 2023b Hart et al. 2023), focusing on tasks that require novelty and abstractions built on models' knowledge (Rytiing WWngate
2021),
covering visual (Zhuo& & Kankanhalli 2020
Jahrens & Martinetz 2020eBarrett Barrett et al.
2018
and textual (Webb et al. 2023 dimensions.
al. 2023
Luetal. 2022
al. 2019 Wei al.
2022
MLLMs.
Wide-spread utilization of foundation models and their role as general-purpose interfaces for different modalities (Hao et al.) 2022) have led to the development of
2

---

Published as a conference paper at COLM 2024
MLLMs (Li et al. 2022 Chen et al. 2022: Alayrac al. 2022 Li et al. 2023a Wang el. 2022
Zhu et al. 2023a that can generate text conditioned on the combination of different modal-
ities, demonstrating zero-shot (Li et al. 2022), few-shot (Alayrac et al., 2022 Zhao et al.
2023), and chain-of-thoughe abilities (Huang et al. 2023). To better understand their range of
abilities, prior studies have evaluated MLLMs for geometric understanding and reasoning
(Kazemi et 20.3) text recognition (Liu et al., 2023b), mathematical reasoning (Lu et al.
2024), college-level deliberate reasoning (Yue et al. 2023), , and open-ended reasoning (Han
et al. 2023) abilities. Most similar to our study are the works Qi et al. (2023) and Mitchell Mithhet
et al. 2023), evaluating the abstract reasoning abilities of MLLMs; however, their evaluation
is either limited to a few examples or lack the rigor to provide an in-depth understanding. In this study, we bridge the gap the literature by conducting extensive experiments that
produce comprehensive insights into MLLMs' abstract nonverbal reasoning abilities.
3 Experimental Setup
3.1 Datasets
IQ50. Introduced by Huang et al. 2023), 1Q50 is a nonverbal reasoning benchmark contain- ing 50 visual puzzles crawled from the internet. Given set of images arranged in matrix or a sequence, the goal is to predict the missing piece of the puzzle from six given options. To conduct our extensive experiments, we augment each puzzle with textual description and hint annotations to explore the interplay between textual cues and visual perceptions.
Introduced by Zhang et al. 2019),, RAVEN is visual reasoning dataset containing 70,000 synthetic samples in seven categories (each 10,000). Each instance is created following sampled rule and contains 3x3 matrix of images (with the bottom right piece missing) and eight options, with a goal similar to IQ50. To reduce the computational costs, we randomly sample 500 examples from each category (3,500 in total) create RAVEN-S.
RAVEN.
CCSE. We collected 175 visual abstractions and reasoning problems from the China Civil Service Examination (CCSE), each containing matrix or a sequence of images with four options. This newly curated dataset serves as challenging benchmark across various reasoning patterns. See Appendix for more details.
3.2 Models
Pre-Trained. The first set of models that we utilize in our experiments are state-of-the-art pre-trained MLLMs. Specifically, we chose the following models due to their popularity and accessibility: 1) BLIP-2 (Li et al. 2023a), 2) Fuyu (Bavishi al. 2023), 3) IDEFICS (Laurençon) et al. 2023), and 4) VL [Bai et al. 2023). Note that all these models, except for the BLIP-2 family, have undergone multi-tase pre-training procedure, presumably allowing them to have zero-shot abilities a wide range of tasks (See Appendix B).
Instruction-Tuned The second set of models that we use in our experiments are state-of- the-art instruction-tuned MLLMs. Specifically, we chose the following models with similar criteria as pre-trained MLLMs: 1) InstructBLIP (Dai et al. 2023), 2) MMICL (Zhao et al.
2023), 3) LLaVA (Liu et al. 2023a 4) IDEFICS [Laurençon et al. 2023), 5) Qwen-VL (Bai
et al. 2023 6) GPT-4V (OpenAl 2023), and 7) Gemini (Google 2023
Heuristics. Since most of our samples follow similar spatial patterns and the task is to fill the missing image with the best candidate, we can write the expected representation of the target image function of the provided query images. To this end, we first compute the target image's expected representation as
Σ Rg R(9), QEQ
(1)
where a function mapping images to vector representations, Q is the set of all query images, and ag IR is the weight of image 9. For simplicity, only consider linear
3

---

Published as a conference paper at COLM 2024
combinations. Then, select the candidate p that has the highest similarity to our prediction:
argmax (r,, argmaxS(r, R()),
(2)
cEC
where the set of candidate images, and S is the Euclidean distance. For example, in Figure 1 heuristic could be formulated as:
Q12 911 = 922 921 > 922 = 921 + 912 911,
(3)
which leads to aqu = -1, 4yz2 = 1 and 4421 = 1. See Appendix for details on selecting R
and calculating kg.
Control Baselines. We also report the random and majority-based performance for all the datasets as control baselines, enabling us to put the posted performances in perspective.
3.3 Implementation Details
Model
IQ50
RAVEN-S
CCSE
We use greedy decoding (i.e., no sam- pling) with temperature = 0.0 and top-p- = 1.0 for all the tested mod- els. Moreover, set 512 across all experiments. Furthermore, for gpt-4v, we set the model's resolution to auto. All our ex- periments are carried out server with x Quadro RTX 8000 GPUs with 48GB VRAM, 251GB RAM, and 32 CPU cores. Finally, we implemented our code using Hugging Face Trans- formers (Wolf et al. 2020) and Py- Torch (Paszke et al.. 2019) libraries.
Pre-Trained
blip2-opt-2.78
0.160
0.122
0.194
blip2-opt-6.7b
0.140
0.117
0.229
blip2-flan-t5-xl
0.160
0.117
0.229
bl fa2---aan--5-x
0.180
0.131
0.211
idefics-9b
0.120
0.120
0.194
idefics-80b*
0.240
0.240
fuyu-8b
0.160
0.1274
0.2971
Qwen-VL
0.180F
0.117
0.206
Instruction-Tuned
WMTCL-vicuna77b
0.200
0.115
0.257F
WWICL-vicum--13-
0.180f
0.126t
0.223
MilLL-Instrucctbllp-T---
0.160
0.1261
0.229
MitCI-Innttructbllpp---wxx
0.200
0.1264
0.229
instruuttblip-viicna-16
0.140
0.126t
0.240
instruutbblip-vvicuaa113
0.160
0.117
0.217
How good are MLLMs nonverbal abstract reasoning?
instrucebbllp=flan-tt-x
0.120
0.121
0.240
instrucctblip--fla--t--vv
0.240
0.126
0.211
idefice-9b-instrucct
0.120
0.121
0.217
ideficcs-80-1nnttucct
0.140
0.251
4.1 Automatic Scoring
1lava-1,5-7b
0.160
0.123
0.2691
11ava-1.5-13bhhf"
0.240
0.121
0.229
Our early observations of the MLLMs' responses indicated the possibility of option markers (i.e., 1, 2, etc.) appear- ing at different token positions. For example, if the model generates "Num- ber 4.", the marker will be at position 3; however, if the model generates "The answer is 4." the marker will be po- sition 7. As such, we pivoted away from the common next-token scoring approach and used pattern-matching scheme. Specifically, we find all the number tokens in the response and then take the one with the largest logit as the chosen option to be compared to the ground truth. We further discuss this approach in An(ppendix A) making a comparison another common au- tomatic scoring method.
bakl lava-v1-hf
0.080
0.122
0.314
Qwen-VL-Chat
0.220
0.117
0.286
Heuristics
Pixel
0.200
0.051
0.257
CLIP-ViT
0.480
0.099
0.234
Control Baselines
Random
0.167
0.125
0.250
Majority
0.220
0.130
0.314
Table 1: Zero-shot accuracy on IQ50, RAVEN-S, and CCSE datasets using the generalized next- token scoring method. For each dataset, the best performance by MILMs is boldfaced while the second best performance is underlined. Legends: T + Timeout after one week of running, * - Ran with half-precision (e.g., bfloat16) to fit in GPU memory, $ -> Performance better than the random baseline.
Table I presents our experimental results on IQ50, RAVEN-S, and CCSE datasets using our automatic scoring method. Our main observation from this table is that no model
4

---

Published as a conference paper at COLM 2024
umali (D
morell (1:00)
WeX 14
renubio (222)
NENES (1420)
608
MAN (88
%%
Namber of Paramstars
Number Paranctora
Number Paramziera
(a) IQ50
(b) RAVEN-S
(c) CCSE
Figure 2: Zero-shot accuracy concerning the number of parameters using the automatic scoring method. Models are sorted from smallest (left) to largest (right), and those within the same family are colored the same. The red dashed lines indicate the random baselines.
consistently beats the random baselines over the three datasets. Moreover, compared to the random baselines, the models perform within the -877% +7.3%), [-1.0%, +0.6%], and [-5.6%, +6.4%] ranges for IQ50, RAVEN-S, and CCSE, respectively. Although some models achieve non-trivial gains over the random baselines and even outperform some of the heuristic baselines, these results expose the difficulty of solving variations of the RPM-style questions across a wide range of MLLMs.
Comparing the best-achieved performances with the majority baselines, we observe stark similarity (+2%). This begs the question of whether the posted numbers are true representatives of the reasoning powers of MLLMs or simply side-effect of their generation biases (e.g., model that always generates 1s will do very well on dataset with many 1s as gold labels). To this end, in in(Section 4.2, we manually examined the generated answers by the instruction-tuned models. Moreover, looking at the results posted by the pre-trained models, apart from fuyu-8b, we cannot observe a significant upside to the multi-task pre-trained models.
is also worth noting that, by examining the results posted within and beyond the same family of models, observe that the scaling law in terms of model size (Kaplan et al. 2020) (i.e., the larger the model, the higher the performance) does not hold here. Figure illustrates the models' zero-shot accuracy concerning the number of parameters.
4.2 Manual Scoring
Model
VA /R
XA /R
VA XR
One critical aspect of assessing such models' abilities is ensuring the correct- ness and faithfulness of their reasoning. As such, we manually inspect the gen- erated responses to provide insights into the results posted by the models in the automatic scoring schemes. All the inspections were done by group of three graduate students (See AAp pendix D for the rubric). Moreover, to elicit reasoning, we appended the phrase "Let's think step by step." to the prompt (Kojima et al. 2022). Out of the 14 open-source instruction-tuned mod- els 3able we were only able to get meaningful and coherent responses from the follow- ing models: Ilava-1, 1.wwa-1.57b---. 5-7b-hf, 1lava-1. 1-1.5-11--bh 3b-hff, Qwen-VL-Chat, and idefics-9b-insrruce Besides these models, we also manually measured the abstract reasoning abilities of closed- source MILMMs (i.e., gpt-4v and cemini-pro-vision) as we don't need access to the gen-
gpt-4v
0.26
0.16
0.10
gemini-pro-vision
0.10
0.14
0.16
11a
0.00
0.00
0.02
11ava-1.5-13b-hf
0.04
0.02
0.10
idefics-9b-instruct
0.00
0.00
0.08
Qwen-VL-Chat
0.00
0.00
0.04
Table 2: Performances of instruction-tuned models on IQ50, assessed by manual inspection, in terms of Answer and Reasoning correctness, indicated by A and R, respectively. The best performance is boldfaced while the second best performance is underlined.
5

---

Published as a conference paper at COLM 2024
erated logits anymore. For the remainder of experiments, relied IQ50, small, challenging test set that is easy for humans to solve (see Appendixx E based on our prelimi- nary tests, but difficult for MLLMs (See[Table 1). This choice also allows us to expand our study more efficiently with various experiments.
Table 2 presents the performance of the aforementioned models IQ50, demonstrating their poor explicit reasoning capabilities. More specifically, only one of the open-source instruction-tuned models achieves non-zero performance the joint answer and rea- soning correctness, with an abysmal performance of 4%, (i.e., an average of 1% across open-source models). Meanwhile, the trend the closed-source MLLMs more promising with gpt-4v outperforming random and majority baseline, providing correct reasoning and answers in 26% of the samples. Nevertheless, their performance still lags behind the simple heuristics by a large margin (See TTalle 1p. Regarding the faithfulness of answers to rea- sonings, we only observe a meaningtul level of faithfulness (alignment between reasoning and answer when either is correct) in closed-source models, 50% for gpt-4v and 25% for gemini-pro-visi Observing such a high percentage of unfaithfulness to reasoning fur- ther asserts the importance of conducting manual evaluations rather than merely reporting performances using noisy automatic measurements.
During our inspections, observed that although models generally un- derstand the shapes in the query im- ages, they tend to hallucinate about specific features such as rotation, shadows, and orientations. On the reasoning side, the most prominent issue was the models being overly descriptive rather than being focused on providing grounded logical rea- soning. In other words, most of the time, the generated responses were the descriptions of the puzzle and the candidate options. We believe this to be an artifact of their train- ing datasets; as such, we expected much better showing from the mod- els that receive multi-task training (See Table 3).
8pt-4v
gemini gemini-pro-veee pro-vision
Qwen-VL-Chat
Model
Ilava-1 1----1
Ilava-1.5-13b-
idefics-9b-instruct
bakLlava-vI-hf
Accuracy
Figure 3: Zero-shot CoT accuracy on IQ50 using text- only prompts.
gvt-4v
Mava-115-7b.6.
gemini-pro-vision
Ilava-1.5-1 5-13-hhf
bakLlava-v1-hff
instructblip-vicune
idefics-9b-instrucc
instructblip-fian-t5
MMICL-Instructbiip-S55 T5-xo
instructblip-vicun--76
What are the root causes of the poor performance?
Model
MMICL-vicuna-76
idefics-80b-instrucc
instructblip-fla--
blip2: f5an-t5-xll
MMIIL-Instructblip
Textual Reasoning. Since most MLLMs fuse the information from visual and textual modalities, they are prone to error propagation from each. Hence, if we bypass one module, we can effectively evaluate the remaining one. For the textual module, we can do this by providing the text-only version of each sample (i.e,, the textual description of the puzzles) written by a human expert (See Apppnndix G). Through these experiments, we can gain insights into the ceiling reasoning capabilities of each model. In experiments, due to their incompatibility with text-only prompts, we had to drop MMICL* and InstructBLIPs models. Moreover, we dropped idef feefic--8bb- -0b--instrurt we could not elicit reasonings from it.
blip2 flan x5
Qwen-VL
idefics-800 80b
Qwen VL-Cht
idefics-9b
fuyu-8b
11p2-opt-6.76
blip2-opt-2.. 2.76
0.0
0.1
0.2
0.3
0.5 Accuracy
0.4
0.6
0.7
0.8
0.9
Figure 4: Visual awareness questions performances on a subset IQ50.

---

Published as a conference paper at COLM 2024
3 presents comparison between open-source and closed-source MLLMs on IQ50 using text-only zero-shot CoT prompts. As evident, gpt-4v is the only model that achieves coming in as distant second. These
high level of performance, with gemini-pro-viss results are consistent with our previous observations, where open-source models struggled to achieve good results, showcasing lag in textual reasoning abilities.
Visual Awareness. Correctly perceiving visual details is critical to nonverbal abstract reasoning. However, Zhang et al. (2023a) have shown that MLLMs face difficulties when isolating granular details large images, As such, we ran a set experiments to determine the extent to which our tested MLLMs understand the presented puzzles. More concretely, we developed 20 questions designed to test the models on understanding shape, relative position, orientation, color, filling pattern, and fine-grained details. In our experiments, dropped MMICL-vicuna- cuna-13b 13b as we could not elicit proper responses.
Figure presents the performances posted the visual awareness questions across open- source and closed-source models. As evident, gpt-4v dominates the benchmark with comfortable lead; however, we find it very promising that some open-source models such as 1lava* can keep up with the other closed-source model (i.e., gemini-provvision) Overall, observing the low performances of open-source MILMs in visual awareness and textual reasoning paints a clearer picture of models' shortcomings, explaining their results in Section
Can MLLMs' performance be improved?
6.1 Guided Prompting
Prompt engineering has been one of the prominent methods to guide LLMs towards specific desired outputs through better conditioning (Wei et al. 2022 Kojima et al. 2202 Wang et al. 2023b et al. 2023b). We experiment with three guided prompting setups, each providing the models with different cues to understand how supplementary textual information is utilized while generating responses. The three setups are as follows (See PeeeApppiixx G for examples):
General. In this setup, we provide the models with broad cues approaching such visual puzzles, hinting at the common strategies without being sample-specific.
Sample-specific. In this setup, we provide the models with one sample-specific hint about the desired reasoning for solving each puzzle.
Corrective, In this setup, an interactive process, looking at the model's reasoning when prompted in a zero-shot setting, we add one hint to correct the most prominent error.
Figure 5) illustrates the performance of closed-source models on IQ50 given different hints. Looking at these re- sults, we find sample-specific and gen- eral hints detrimental to reasoning and accuracy while observing a sig- nificant boost when interactively uti- lizing corrective hints, especially for gpt-4v. Considering the distinct em- bedded cues of sample-specific and cor- rective hints, we believe that the mod- els' inherent chain of reasoning on the provided puzzles misaligned with humans, which makes sample-specific hints confusing rather than helpful. However, can correct models' al- ready doid-outsllutionn with corrective hints, improving their performance by much as 100%.
gpt-4-vision-prees
gemini-pro-vision
0.7
Answers
0.6
Reasonings
0.5
0.4
0.3
0.2
0.1
0.0
Z-S
Sam
Z-S
Gen
Sam
Cor
Setting
Setting
Figure 5: Guided prompting performance of gpt-4v and gemini-pro-vision IQ50 using dif- ferent types of hints. Legend: Z-S + Zero-shot, Gen General, Sam Sample-specific, and Cor + Corrective.
7

---

Published as a conference paper at COLM 2024
0.25
0.25
O-shot 1-4hot
3-shot 5-shot
O-shot 1-4hot
3-shot
5-shot
0.20
0.23
Accuraay
Accuracy
0.15
0.15
0.05
0.05
idefics 9b
idefics 9b instruct
Qwen-VL
Qwen-VL-Chat
idefics 9b
idefics 9b. instruct
Qwen-VL
Qwen-VL-Chat
Model
Model
(a) In-Distribution
(b) Out-of-Distribution
Figure 6: Zero-shot and symmetrical few-shot accuracy IQ50. In (a) In-Distribution, the demonstrations are taken from IQ50, while in (b) are taken from RAVEN-S. Each variation was executed five times with different seeds to mitigate the effect of random sampling. The red dashed lines indicate the random baselines.
the demonstrations
6.2 In-Context Learning
In-context learning (ICL) refers to emergent behavior in LLMs where they perform a task conditioned on the provided demonstrations without further parameter optimization, Min et al. (2022) have suggested that ICL is mere mechanism for locating the already learned ability of the model to respond the query prompt. However, more recent studies have shown that LLMs can learn various function classes through ICL (Garg et al. 2022
Mirchan
dani et al. 2023
Lee et al. 2023
6.2.1 Symmetrical Few-Shot
The most common form of ICL few-shot, in which the model is provided with demonstra- tions similar to the test sample. We call this variation "Symmetrical"" as there is no imbalance in the demonstrations' textual and visual information pieces. Since symmetrical ICL. requires processing multiple image and text pairs, we only utilize models capable of processing such inputs (See Table 3 for a list). Moreover, due to burdensome computation costs, we exclude idefics-800 80bx models. In our experiments, we explored the effect of 1) changing the sampling distribution of demonstrations and 2) including step-by-step reasoning with demonstrations (See Appendix for examples).
Effect sampling distribution. Our experiments cover open-source models and utilize our automatic scoring scheme over two variations: 1) In In-Distribution (ID): demonstrations are uniformly taken from the same dataset, and 2) Out-of- Duitof-DDstributtion (OOD): demonstrations are uniformly taken from another dataset. We used IQ50 as the evaluation source with RAVEN-S as the OOD source. Moreover, we ran each variation five times to reduce the impact of random sampling and up to 5-shot (ie., 10% of the dataset) due to GPU limits.
Figure 6 presents the results of our experiments. As evident, there no consistent patterns across models and variations. For example, while idefics-9be models generally benefit from the few-shot scheme, we don't see monotonically increasing perfor- mances with more demonstrations, contrary to expected patterns in LLMss Simultaneously, we see consistent de- cline in performance in some variations of Qwen-VL* models. We believe these irregularities caused by inability to understand the utility of the demonstrations, suggesting that the tested models do not exhibit strong and consistent symmetrical ICL capabilities. As result, the models cannot
gpt-4v
Model
gemini-pro-viioon
0-shot
3-shot
5-shot
0.0
0.1
0.2
0.3
0.4
0.5
Accuracy
Figure 7: Symmetrical few-shot CoT accuracy on IQ50.

---

Published as a conference paper at COLM 2024
take advantage of the provided demonstrations properly, leading to poor responses and subpar performances.
Effect of step-by-step reasoning. Inspired by success, we experiment with the few- shot Chain-of-Thought (CoT) prompting (Wei et al., 2022) to improve the performance of our models. Our early experiments found open-source models, such as Qwen-VL-Chal and adefics-9b-instrre
unable to comprehend CoT prompting. As such, we focused on examining the closed-soure models: gpt-4v and gemini -pro-vision.
Figure presents the performance of closed-source models with CoT prompting. We can observe that both models benefit significantly from CoT demonstrations, with gpt-4v's performance being boosted as much 100%. These results emphasize the immense gap be- tween the open-source and closed-source models while showcasing meaningful symmetrical ICL abilities in these models. During our evaluation, with gemini ceminni-ro-vve iapppoosiionooooo in which the one-shot variation prevented the model from generat- ing reasonings in almost all the examples, leading to an initial performance drop. Moreover, in a similar setting for gpt-4v, we observed 1) massive jump in the number of safeguard triggers (4% + 18%), which precluded the model from generating response, and an increased confusion regarding the boundaries of the provided demonstration (0% + 10%), diminishing the model's performance.
noticed unusual phenomenon
6.2.2 Asymmetrical Few-Shot
Adding new modalities has brought forth the possibility of providing lopsided (i.e., Asym- metrical) information in the input prompt. As such, conduct set of few-shot experi- ments that provide the models with text-only CoT demonstrations while keeping the query unchanged (i.e,, image text). To encourage the models use the demonstrations, we append "Let's solve the puzzle in the image, step by step, similar to the demonstrations. to our prompt. We hypothesize that similar the findings of Min al. (2022), the models will better understand the input and output spaces and achieve improved performance.
Since we could not make open-source models properly utilize the textual demonstrations in our preliminary ex- periments, we continued our experi- ments only on closed-source models. Figure 8 illustrates the results of our experiments with asymmetrical few- shot CoT prompting. As evident, this approach does not yield meaningful improvements and even causes degra- dation in some cases, contrasting our hypothesis. Moreover, noticed in- creased hallucinations, mostly confus- ing the demonstrations with queries and detecting non-existent details in the shapes, leading to unfaithfulness and instability in the responses. We leave further investigations and exper- iments on this ICL scheme future works.
gpt-4v
Model
gemini-pro-viiion
3-shot
1-shat 5 shat
0.4
Accuracy
Figure 8: Asymmetrical few-shot CoT accuracy on IQ50.
7 Conclusion
In this study, we utilized different RPM-style tasks as proxy for measuring the nonverbal abstract reasoning abilities of MILMs,, covering 24 different open-source and closed-sourg models. Although closed-source MLLMs showcased promising capabilities in our exper- iments, we found the abilities of open-source models to insufficient for solving these tasks. Moreover, using pseudo-isolated experimental environments, we found that MLLMs often fail at 1) gathering precise visual details from puzzles and 2) reasoning correctly and faithfully, even when provided with expert-written and complete descriptions of the puzzles. Furthermore, our experiments highlighted 1) the inability open-source models to consistently utilize demonstrations and 2) the ICL prowess of the clored-source models,

---

Published as a conference paper at COLM 2024
which helped them benefit from interactive guidance provided demonstrations with step- by-step (i.e, CoT) reasoning. Although MLLMs have previously demonstrated proficiency at various tasks, study using relatively simple reasoning task for humans has exposed some critical shortcomings in MLLMs while highlighting the importance of more grounded evaluations, even at small scales.
Acknowledgements
This work has been sponsored and funded by the Defense Advanced Research Projects Agency via awards HR00011220005 and N660011924033 and contract HR00112390066 We thank Dong-Ho Lee, Pei Zhou, and Pegah Jandaghi for their invaluable feedback.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 15-23716-2375355 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. Xrv preprint arXiv:230 129666 2023.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp- 511-520. PMLR, 2018.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşrlar. Introducing our multimodal models, 2023. URL https adept. wwwp.aadpt..a./llg/ffuuee ai/blog/fuyu-BBl
Laura E Berk. Why children talk to themselves. Scientific American, 271(5):78-83, 1994.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint ar arXiv:2303.12711 2023.
Raymond Bernard Cattell. Intelligence: Its structure, growth and action. Elsevier, 1987.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXio:2209.06794 2022.
Cédric Colas, Tristan Karch, Clément Moulin-Frier, and Pierre-Yves Oudeyer. Lan-
guage cognitive tool: Dall-e, humans and vygotskian rl agents, 2021. URL
https:://deellopeenntaallsss hwtpessoogggglaagaggg htt///laagggeaaee as coggiiiiiii.. tive tool vyoooskkgnk
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpost vision-language models with instruction tuning, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ar Xiv preprint ar Xiv: 11929, 2020.
10

---

Published as a conference paper at COLM 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
URL /nttps://openreviee.. wttp://wwwww.eoorriiiciiiioooooa
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Gemini Team Google. Gemini: A family of highly capable multimodal models, 2023.
Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, al. Core-mm: Complex open- ended reasoning evaluation for multi-modal large language models. arXiv preprint arXio:2311.11567 2023.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpor interfaces. ar preprin! 2XXt:2206.06333 2022.
Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. Sok: Memorization in general-purposs large language models. ar preprint Xiv:231001188 2023.
Felix Hill, Adam Santoro, David GT Barrett, Ari S Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. arv preprint arXiv:1902.001100 2019.
Xiaoyang Hu, Shane Storks, Richard Lewis, and Joyce Chai. In-context analogical reason- ing with pre-trained language models. arXiv preprint arXiv:2305.17626 2023.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint ar axXiv:2002.14045, 2023.
Marius Jahrens and Thomas Martinetz. Solving raven's progressive matrices with multi- layer relation networks. In 2020 International joint Conference on Neural Networks (IJCNN), pp. 1-6. IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint ar Xf1::23122.22211 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing 2022.
systemss, 35:22199-22222
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, An- ton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. Temporal knowledge graph forecasting without knowledge using in-context learning, In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 544-557, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10.18653 / vv1 /tt////...22///// https://acllntthooogg.org org/2203 www.::ewww..m......mmm.mmmmmm...
2023.emnlp-mainn
URL
11

---

Published as a conference paper at COLM 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2
Bootstrapping language- image pre-training with frozen image encoders and large language models. aro preprint #rXio:2301.12597 2023a.
Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting In Chirty-sevenlly Conference on Neural Information Processing Systems, 2023b. URL https: http::::///peneiiew....o www.reeiiwweeefooumiiiiieee
UVIN8oQ4uI
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint ar (xXi::2304.08485, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the TEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arviv preprint xxXi=23355008855 2023b.
Hongjing Lu, Nicholas Ichien, and Keith J Holyoak. Probabilistic analogical mapping with semantic relation networks. Psychological review, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math- ematical reasoning of foundation models in visual contexts, In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviieee wwtpreeewwwweweeeeuueeeeee net/formm?ff KUNZEQMMU7
Gary Lupyan. What do words do? toward a theory of language-augmented thought. In Psychology of learning and motivation, volume 57, pp. 255-297. Elsevier, 2012.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. e022eemllpmmi.. 759
URL
org/
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arvio preprint ar Xii::2307004211, 2023.
Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. ar preprint ar arXiv:2311.09247, 2023.
Hyeonseob Nam,Jung Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 299-307,,00
OpenAI. Gpt-4 technical report, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high--eeformmnce deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at COLM 2024
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /v15/3/vvv// vv 2023.findings-emmlp Findings-emml..3 378
URL https
//aclanthology.orgg//
Zhangyang Qi, Ye Fang, Mengchen Zhang, Zeyi Sun, Tong Wu, Ziwei Liu, Dahua Lin, Jiaqi Wang, and Hengshuang Zhao. Gemini vs gpt-4y: A preliminary comparison and combina- tion of vision-language models through qualitative cases. arXiv preprint ar a'Xiv:2312.1501 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Jean Raven. Raven progressive matrices. In Handbook of noneerbal assessment, pp. 223-237. Springer, 2003.
Christopher Rytting and David Wingate. Leveraging the inductive bias of large language models for abstract textual reasoning. Advances in Neural Information Processing Systems, 34:17111-17122, 2021.
Mohammed K Shakeel, Vina M Goghari, et al. Measuring fluid intelligence in healthy older adults. Journal of Aging Research, 2017, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Lev Vygotsky. Thought and language. MIt Press Cambridge, MA, 1962.
Andrey Vyshedskiy. Language evolution to revolution: the leap from rich-vocabulary non- recursive communication system to recursive language 70,000 years ago was associated with acquisition of a novel component of imagination, called prefrontal synthesis, enabled by a mutation that slowed down the prefrontal cortex maturation simultaneously in two or more children the romulus and remus hypothesis. Research Ideas and Outcomes, 5: e38546, 2019.
Jiaqi Wang, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai, Qiushi Yang, Yiheng Liu, Songyao Zhang, et al. Review of large vision models and visual prompt engineering. Meta-Radiology, pp. 100047, 2023a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Peng Lim. Pían-and-solve prompting: Improving zhro mhain-of-thoughle reason- ing by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2609-2634, Toronto, Canada, July 2023b. As- sociation for Computational Linguistics. doi: 10.18653 11.8653/vl.. http//11222222// acl.o.....1o URL https://aclanthol//
org/ 2023 2tt.2222......nn...... ac1-
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Ag- garwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXio:2208.10442 2022.
Taylor Webb, Keith J Holyoak, and Hongjng Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526-154,, 2023.
13

---

Published as a conference paper at COLM 2024
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevien net/foorumiid--jjiMMeSS.
Jonathan Winawer, Nathan Witthoft, Michael Frank, Lisa Wu, Alex Wade, and Lera Boroditsky. Russian blues reveal effects of language on color discrimination. Proceedings of the national academy of sciences, 104(19):7780-7785 2007.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653 /v 20220eemllp-demo....66 URL https:///claatthollgg org/ 2020. emnlp-demos.. 6
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502 2023.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5317-5327, 2019.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Visual cropping improves zero-shot question answering of multimodal large language models. arXiv preprint ar 1tf:::330016033: 2023a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? arxiv preprint xrXiv:2301.13670, 2023b.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. ar Xiv preprint ar Xix::2239900791, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. arXio preprint ar arXio:2308.07107, 2023b.
Tao Zhuo and Mohan Kankanhalli. Solving raven's progressive matrices with neural networks. Xiv preprint arXiv:2002.01646 2020.
14

---

Published as a conference paper at COLM 2024
A One-by-One Scoring
Introduced by Huang et al. (2023) in this scoring method, we first flatten the query image matrix and feed it into the model along with exactly one op- tion. We also surround these images with textual instructions to help the
MMICL-Instructllip T5-ood
idefics-800 80b
instructblip-flanee xoa
Model
idefics 9b-instruct
model better understand the desired task (See Appendix Gj for examples). Then, we calculate the probability of the model generating "Yes", represent- ing the probability of that option being the true missing piece. Finally, we de- termine the model's choice by taking the option with the highest probabil- ity. To improve the original method, we introduce an equivalency condition in which the max probability of dif- ferent variations of the target token (egg,, "YES" and "yes") is taken as the probability, The main shortcomings of this method include 1) being more computation-heary as needs to pro- cess each option separately and 2) being compatible with only the models that accept multiple images as input. Notably, among the open-source models utilized in this study, only Qwen-VL* and idefics* models support multi-image inputs (See Table 3).
Qwen-VLLCCat
idefics-9b
Qwen-VL
Mode
One-by One Ours
idefics 80b instruct
0.00
0.05
0.10
0.15
0.20
0.25 0.30 0.35
Accuracy
Figure 9: Zero-shot accuracy comparison on the IQ50 dataset using the one-by-one and our auto- matic scoring methods. Results with t marker are taken from ZZao et al. (2023). Due to runtime errors, could not replicate them (neither Hug- gingface nor GitHub versions). The red dashed line indicates the random baseline.
Figure 9 compares the experimental results with the one-by-one and our automatic scoring methods. As evident, some models such as MMICL-Ins tessstuuulllpsstt idefics-bb-intree benefit significantly from this scoring mode, while others such as and Qwen-VL- -Chhat suffer extensively. This observation demon- strates the validity of both methods, making the choice subject the lask/datal evaluated. However, given these mixed results and the downsides of using this scoring method, we conclude that our automatic scoring method more practical for future studies.
idefics-9b, and
being
B
Models
lable
presents an attribute comparison over open-source and closed-source models.
BLIP-2 Li et al. 2023a). BLIP-2 is a generic and efficient pre-training strategy that boot- straps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer (Q-former), which is pre-trained in two stages. Despite having signif- icantly fewer trainable parameters than existing methods, BLIP-2 achieves state-of-the-art performance on various vision-language tasks.
Fuyu (Bavishi et al. 2023). Fuyu is multi-modal text and image transformer trained by Adept AI. Archifecturally Fuyu is a vanilla decoder-only transformer with no image encoder. Image patches are instead linearly projected into the first layer of the transformer, bypassing the embedding lookup. This simplification allows the model to support arbitrary image resolutions. Fuyu-8B improves over Qwen-VL, on 2 out of the 3 most commonly used image-understanding datasets despite having 2B fewer parameters.
IDEFICS (Laurençon et al. 2023). IDEFICS (Image-aware Decoder Enhanced la Flamingo with Interleaved Cross-attentionS) is open-access reproduction of Flamingo, closed- source visual language model developed by Deepmind. It is built on top of two unimodal open-access pre-trained models with newly initialized parameters in the form of Transformer
15

---

Published as a conference paper at COLM 2024
Open Source
Multi-Task Pre- Tre-Training
Multi-Image Input
Model
Size
Pre-Trained
blip2-opt-2.75
3.7b
X XXP XXX
blip2-opt-6.76
7.8b
blip2-flan-t5-x1
3.9b
blip2-flan-t5-xi
12.2b
X
X
idefics-9b
8.9b
x
V
idefics-80b
80.0b
x
V
fuyu-8b
9.4b
/
x
Qwen-VL
9.7b
Instruction-Tuned
gpt-4-vision-previer
x
UC
Bard (Gemini Update)
X
WIILL-vicun--7b
7.9b
WMICL-Vicuna-132
14.2b
MICL--Instrucctllip-155-l
4.0b
MICL-IInstructtliip115xxx
12.3b
instructbllp-vvicuna-bb
7.9b
instructtblip-vvccun-1133
14.2b
instructbblip-flan-t---
4.0b
instructbllpp-flant6---
12.3b
idefics-9b-instruct
8.9b
idefics-80b-instruce
80.0b
11ava-1.5-7b-hr
7.1b
11ava-1.5-13b-hf
13.4b
/
bakLlava-v1-hf
7.6b
U
Qwen-VL-Chat
9.7b
Table 3: Comparison of open-source and closed-source models' attributes. Legend: U
Undisclosed, - We could not utilize this feature using the official code.
blocks to bridge the gap between the vision encoder and the language model. The model trained on image-text pairs and unstructured multi-modal web documents. IDEFICS instruct is the model obtained by further training IDEFICS on Supervised Fine-Tuning and Instruction Fine-Tuniee Tunnng datasets. IDEFICS on par with the original closed-source model on various image-text benchmarks, including visual question answering (open-ended and multiple choice), image captioning, and image classification when evaluated with in-context few-shot learning.
VW (Bai et al. 2023). Qwen-VL models large-scale vasion-language models (LVLMs) designed to perceive and understand texts and images. Starting from the Qwen-LM as foundation, the model endowed with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multi-modal cleaned corpus. The resulting models, including Qwen-VL and Qwen-VL--Caa,, set new records for generalist models under similar model scales.
InstructBLIP (Dai et al., 2023). InstructBLIP models are instruction-tuned MLLMs based on the pre-trained BLIP-2 models. They have been trained using 13 publicly available datasets transformed into instruction tuning format. Additionally, the authors introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains stateoof-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models.
16

---

Published as a conference paper at COLM 2024
LLaVA-1.5 [Liu et al., 2024). LLaVA-1.5 achieves state-of-the-art performance various multimodal benchmarks through increased input resolution, additional layers of multimodal projection, and comprehensively curated visual instruction tuning dataset. The model's efficient and lightweight architecture facilitates high reproducibility, positioning as versatile foundation for further research and development in the multimodal community.
MMICL (Zhao et al. 2023). Unlike previous work, MMICL utilizes novel context scheme, treating image and text representations equally and establishing the reference between image and text via image declaration. enables users have the flexibility input multiple images and text in any desired order, with no restrictions on the quantity or placement of images in contexts. MMICL achieves new state-of-the-art zero-shot performance on wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Moreover, MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges with impressive ICL ability.
GPT-4V (OpenAl 2023). GPT-4 with vision (GIT-4V) enables users to instruct GPTT44 to analyze image inputs provided by the user and is the latest capability OpenAl is making broadly available.
Gemini (Google 2023). The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained broad range of benchmarks shows that the Gemini Ultra model
use cases. Evaluation advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU and improving the state-of-the in every one of the 20 multi-modal benchmarks the authors examined, At the time of this publication, only the Pro version was publicly available.
C Heuristics Details.
C.1
Selecting R
Pixel. Since raw pixel values are inputs to all MLLMs in this study, we utilize the flattened pixel values the first variation of R in our heuristics.
CiIP-ViT (Dosovitskiy et al., 2021 Radford et al. 2021). Contrasting Language-Image
Pre-Traine Pre-Training (CLIP) is adopted by most of the open-sourced MLLMs as their visual encoder. As such, we select CLIP encoding as the second variation of R our heuristics.
C.2
Calculating Qq
We first identify all possible axis, considering that each 2 pattern spans across two columns, the number of unique horizontal positions is π - 1. Similarly, m 1 unique vertical positions are on the vertical axis. Therefore, the total number of distinct
submatrices in the m x n query matrix. For the horizontal
patterns that can be extracted from m x n matrix is (m - 1) x (n 1). To determine the expected target representation, we first 3
each submatrix; then, we average over all the calculated values. Finally, we choose the option with the smallest Euclidean distance (i.e., S) to the expected target representation.
apply the formula
D Manual Inspection Rubric
Our main challenge was to overcome the nuances that appear in reasonings generated by LLMs. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples in the dataset and 2) review series of sample responses generated by the models (e.g,, GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/ extra/wrong /wong details
17

---

Published as a conference paper at COLM 2024
?
2
4
Figure 10: Example of the location pattern. Across the pieces, the triangle moves clockwise, one block at a time in the inner circle, while the five-pointed star moves clockwise, two blocks at time in the outer circle.
in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths. For example, in some instances, the models perceived shadows (or incorrect colors) in the shapes that were not present the puzzle; however, as long as they correctly detected the row-wise and column-wise change of patterns (e.g,, square turning to circle) and grounded their reasoning on them, the evaluators marked the response correct. Moreover, each sample was annotated and assessed by one person, and any uncertain case was flagged and shared among all three evaluators for discussion. After discussions, the final label was determined by majority vote (i.e., at least 2 out of 3). All evaluators were mid-level to senior Computer Science PhD students specializing NLP with extensive experience working with and evaluating LLMs.
While we acknowledge the difficulty of scaling such manual experiments, one of the main challenges of correctly assessing generated responses that automatic metrics like ROUGE, BERTScore, etc., fall short of adequately evaluating the semantic nuances. Hence, we decided to go down the path of highly labor-intensive human expert evaluation to provide precise, concrete, and grounded insights.
E Human Performance on IQ50
To establish a baseline, we ran study with 25 college-level participants from diverse educa- tional and demographic backgrounds. We provided each participant with ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed them to solve the puzzle and give a short reason for their answer. The average performance this group was 95.9%, with standard deviation of 6.92%.
F CCSE Dataset Examples
CCSE tests models' reasoning abilities over five general patterns (i.e., location, logic, pro- gression, self-geometry, and relative-geometry) in three types of figure configurations (i.e., one-row, two-rows, matrix). Figure 10 and Figure 17 for examples.
18

---

Published as a conference paper at COLM 2024
3
4
Figure 11: Example of the logic pattern. Each row follows cell color XOR operation (0 for white and for black) between the first and the second columns make the piece in the third column.
G Prompt Examples
Table and Table present examples of each prompt used in our experiments.
H
Limitations
In this work, we focused on only a specific type of reasoning task (i.e., nonverbal abstract rea- soning) while using only IQ50, small but challenging dataset, in most of our experiments. However, despite these limitations, we hypothesize that similar shortcomings could be replicated in other reasoning tasks due to the fundamental and not task-specific nature of the observed problems. Moreover, although our experiments yielded insightful results, further analysis of the observations is still possible. For example, given the remarkable effectiveness of corrective hints, we can utilize methods such as self-talk Press et al. (2023), automating the whole process. Another example is the visual awareness tests, where it is possible to examine the internal values of the models (e.g,, attention weights, token probabilities, etc.), as opposed to evaluating the generated responses. Finally, since the -oourdsouuce models' training datasets are unknown, test set contamination is possible, leading to their superior performance. However, based on the relatively low performance of these models and considering the low-level difficulty of the tests for humans, we believe contamination to be unlikely.
19

---

Published as a conference paper at COLM 2024
Experiment Automatic scoring
Prompt Example
[IMG] You are given a puzzle. The puzzle features a set visual patterns arranged matrix the top, with the bottom right piece missing and options the bottom (marked by I, 2, 3,4, 5, or 6). Which option (either 1, 2, 3,4,, 5, or 6) fills the missing
piece best?
One-by-one scoring
Here are three images: (IMGI) (IMG2) [IMG] The following image is: /IMG4) correct?
Zero-shot CoT
[[MG/ You given puzzle. The puzzle features
visual patterns arranged
matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 4,
Which option (either 1, 2, 5, 6) fills missing piece best? Let's think by step.
Textual reasoning
Puzzle:
[[yellow percentage sign, yellow percentage sign].
[yellow percentage sign, ?]]
Options:
yellow percentage sign
2: yellow plus sign
two yellow circles
yellow circle
5: yellow division sign
yellow cross
You are given puzzle. The puzzle features set of patterns arranged in matrix on
top, with bottom right piece missing and six options bottom (marked by
2, 3, 5, 6). Which option (either 1, 2, 3, 4, 5, or fills the missing piece best?
Let's s think step by step.
Visual awareness
In candidate 4 at the bottom, are the arrows arranged clockwise or counterclockwise?
Guided prompting (General)
JIMGI You are given puzzle. The puzzle features a set of visual patterns arranged matrix on the tup, with the bottom right piece missing and six options at the buttom Which option (either 1, 2, 3,4,, piece best? Hint: Focus on the row-wise and column-wise changes regarding color, orientation, and shape the puzzle pieces. Let's think step by step.
(marked by 1, 2,3, 5,
6) fills the missing
Guided prompting (Sample-specific
[IMG] are given a puzzle. The puzzle features a visual patterns arranged matrix on the top, with the bottom right piece missing and six options the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2,3, 4, 5, or 6) fills the missing piece best? Hint: the focus should be the column- wise changes. Let's think step by step.
Guided prompting (Corrective)
Tur 1. [IMG] You are given puzzle. The puzzle features a set of visual patterns arranged in matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 2, 4,5, or Which option (either 2, 3, 4,5, fills the missing piece best? Let's think step by step.
[Mode's Response]
Turn 2. Hint: Option 1 does not have a small circle inside it, and option 5 is a very circle itself.
Symmetrical few-shot
You are given a puzzle. The puzzle features set visual patterns arranged a matrix the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, or 6). Which option (either 1,2, 3, 5, or 6) fills the missing piece best? Let's think step by step.
[IMGI]
answer 4.
[MG2]
The answer is 1.
[[MG3]
Table 4: Prompt examples for our experiments. Note that all "[IMG*]" are replaced with actual images during inference.
20

---

Published as a conference paper at COLM 2024
Experiment
Prompt Example
Symmetrical few-shot CoT
visual patterns arranged matrix the top, with the bottom right piece missing and six options at the bottom (marked by 2, 5, 6). Which option (either 1, 2, 3, 4, 3, or 6) fills the missing piece best? [MMG]]
given a puzzle. The puzzle features
To solve this puzzle, we need to identify pattern or rule that applies to the rows or columns we discemn any patterns.
the matrix. Let's examine the rows and columns Looking at the first row, we see yellow percentage sign followed by a yellow percentage sign. Hence, nothing changes the row, moving from left right In the second row, yellow percentage sign. Following the above pattem, we deduce that the missing piece in the second row is a yellow percentage sign. Now, let's look at the columns. The first column has a yellow percentage sign, fullowed by yellow percetagee sign. Hence, nothing changes the column, moving from top to bottom. The second column has yellow percentage sign. Following the above pattern, the missing, piece the second column must be yellow percentage sign. Combining the observations from rows and columns, we conclude that the missing piece yellow percentage sign.
The provided options the bottom are as follows:
1. Yellow percentage sign
2. Yellow plus sign
3. Two yellow circles
One yellow circle
5. Yellow division sign
Yellow
Given these options and our conclusion, option 1 fits our criteria and best fills the missing piece. [IMG2]
Asymmetrical few-shot CoT
given a puzzle. The puzzle features a visual patterns arranged top, with the buttom right piece missing and six options at the bottom (marked by 1, 2, 3,4,, 4,5, 6). Which option (either 1, 2,3, 4,5, or fills the missing piece best?
the
Demonstration 1:
Puzzle:
[lyellow percentage sign, yellow percentage
[yellow percentage sign, 2]]
Options:
I: yellow percentage sign
2: yellow plus sign
3: two yellow circles
4: one yellow circle
5: yellow division sign
6: yellow cress
solve this puzzle, we need to identify pattern rule applies of the matrix. Let's examine the rows and columns to see if we can discern any patterns. Looking at the first row, we see a yellow percentage sign followed by a yellow percentage moving from left right. In the second row, there's
rows columns
sign. Hence, nothing changes in the a yellow percentage sign. Following the above pattem, we deduce that the missing piece in the second row is yellow percentage sign. Now, let's look the columns. The first column has a yellow percentage sign, fullowed by yellow pencettgee sign. Hence, nuthing changes the column, moving
top bottom. The second column y sign. Following the above pattern, the missing piece in the second column must be yellow percentage sign. Combining the observations from rows and columns, we conclude that the missing piece yellow percentage sign.
yellow percentage
The provided options at the bottom are as follows:
1. Yellow percentage sign
2. Yellow plus sign
Two yellow cineles
4. One yellow circle
5. Yellow division sign Yellow
Given these options and our conclusion, option 1 fits our criteria and hest fills the missing piece.
solve
solve the puzzle in the image, step by step, similar to the demonstrations.
[IMG]
Table 5: Prompt examples for our experiments (Continued). Note that all "[IMG*]" are replaced with actual images during inference.
21


Reviews:
[{"review_id": "ndMTZR3fr5", "reviewer": "Reviewer_XKpq", "summary": "This study explores the nonverbal abstract reasoning capabilities of multi-modal large language models (MLLMs) using variations of Raven's Progressive Matrices, a benchmark traditionally used to assess human cognitive abilities. The research aims to bridge the gap in understanding the complex reasoning abilities of MLLMs, which integrate both verbal and visual information. Through rigorous testing, the study reveals significant performance disparities between open-source and closed-source MLLMs and identifies limitations in their visual and textual processing capabilities. The authors also investigate various enhancement strategies, notably Chain-of-Thought prompting, which dramatically improves the models' reasoning performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:\n\n**Q1:**\nRegarding resolution, all samples were high-resolution (> 1000x600 pixels) to ensure all details were adequately presented. We will add this information to the paper for clarification. Regarding the objects' position, while it is true that each model has a slightly different preference, we tested them on the widespread use case of one picture with details centered from both directions. It is important to note that many models performed well on our visual awareness tests (Section 5, Figure 4), showcasing their understanding of the puzzles' details from a purely visual standpoint. At the same time, many performed exceptionally poorly when provided with all the details in a textual format (Section 5, Figure 3), showcasing a gap in reasoning abilities rather than perception abilities.\n\nRegarding the effect of textual description, our experimental goal was to test the current state of the models in a natural scenario where the user does not need to guide the model. In parallel, we have a set of experiments in Section 6.1 (Figure 5) to show how much we can boost their performance through targeted textual guidance.\n\nIn general, we assumed that any well-trained model must work well in typical scenarios, as it would be challenging and costly for a user to find the optimal setting.\n\n**Q2:**\nRegarding the demographic and expertise of the evaluators and annotators, all were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience working with and evaluating LLMs.\n\nRegarding the rubric and the inter-annotator agreement, our main challenge was to overcome the nuances that appear in generated reasonings. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples and 2) review examples of generated responses (e.g., GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths.\n\n*Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.*\n\n**Q3:**\nNo, the non-annotated version of the datasets has already been used in previous publications and is publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Timely topic.\n2. Rich analysis and results.\n3. Help to accelerate future research.", "reasons_to_reject": "1. The reviewers has some concerns on the current experiment settings.\n\na. The impact of resolution and position representation. Current set up doesn't ablate much on whether the limited performance especially on zero-shot comes from the fact that the model is more suitable for other representation of locations and resolution. For example, is it possible that certain issue comes from the resolution/scale of the main object in the image? Is it possible that it is better to use language description of the locations such as top right, top left, bottom right and bottom right to let the model better understand the relative positions?\n\nb. The manual evaluation lacks more rigorous analysis. There are only 3 annotators and the rubric is vague on the hallucination part. It is also unclear about inter-annotator agreement on the annotation results and the current setup makes it almost impossible to reproduce the results on one of the most important parts of the work: reasoning quality. It is also worthy to set up an automatic score for evaluation of reasoning process.", "questions_to_authors": "Will there be copy right issues of using these data?", "extra_scores": {}}, {"review_id": "0YoP1pwswj", "reviewer": "Reviewer_QZHA", "summary": "This paper presents a comprehensive evaluation of the nonverbal abstract reasoning abilities of multi-modal large language models (MLLMs) using variations of Raven's Progressive Matrices (RPM) as benchmarks. The authors test a wide range of open-source and closed-source MLLMs, providing insights into their current capabilities and limitations. The experiments reveal that while closed-source models like GPT-4V demonstrate non-trivial reasoning abilities, open-source models generally struggle with these tasks. The paper also uncovers critical shortcomings in both the visual and textual reasoning capabilities of MLLMs, setting performance ceilings. Interestingly, the authors find that techniques like Chain-of-Thought prompting and corrective hints can significantly boost performance, especially for closed-source models.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:\n\n**Q1:**\nWhile we acknowledge that RPM and CoT are not new methods, our paper is more geared toward providing novel analysis and insight into MLLMs and their potential shortcomings rather than introducing novel approaches.\n\n**Q2 & Q3:**\nWe agree that there is a need for further investigations on the effect of model architectures, training data, training regimens, fine-tuning, etc.; however, given the space of possibilities and cost of training these large models, such a study is prohibitively costly and broad for one article. Moreover, many of these details are often not disclosed publicly for closed-source models. As a result, we focused on testing a wide range of models (implicitly diversifying the approaches) in common zero/few-shot scenarios, providing novel insights into their strengths and limitations. We hope this work emphasizes the importance of these open challenges in future works.\n\n**Q4:**\nWe ran a study with 25 college-level participants with diverse backgrounds. Each participant was given ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed to provide a short reason for their answer. The average performance of this group was 95.9%, with a 6.92% standard deviation. We will add these results to the paper to emphasize the performance disparity.\n\n**Q5:**\nRegarding the rubric and the inter-annotator agreement, our main challenge was to overcome the nuances that appear in generated reasonings. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples and 2) review examples of generated responses (e.g., GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths.\n\nFor example, in some instances, the models perceived shadows (or incorrect colors) in the shapes that were not present in the puzzle; however, as long as they correctly detected the row-wandd/or column-wise change of patterns (e.g., square turning to circle) and grounded their reasoning on them, the evaluators marked the response as correct.\n\n*Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a thorough evaluation of a wide range of open-source and closed-source MLLMs on nonverbal abstract reasoning tasks. The authors use variations of Raven's Progressive Matrices as benchmarks, which are well-suited for assessing abstract reasoning abilities. The extensive experiments offer good insights into the current state-of-the-art in MLLM reasoning.\n- The paper uncovers several interesting findings, such as the  performance gap between open-source and closed-source models, the critical shortcomings in both visual and textual reasoning capabilities, and the effectiveness of techniques like Chain-of-Thought prompting and corrective hints in boosting performance. These findings contribute to a better understanding of MLLMs and their limitations.", "reasons_to_reject": "- The use of Raven's Progressive Matrices and techniques like Chain-of-Thought prompting have been explored in previous studies, albeit not as extensively as in this work.\n- The paper compares a wide range of MLLMs but does not provide a detailed analysis of how the architectural differences between these models might influence their reasoning capabilities. A more thorough investigation of the relationship between model architecture and performance on abstract reasoning tasks could have provided additional insights and made the contributions more substantial.\n- The authors acknowledge that the superior performance of closed-source models could be due to the nature of their training data. However, the paper does not delve into this aspect in detail. How fine-tuning on these tasks improve the open-source models' performance? A more comprehensive analysis of the potential impact of training data on the models' reasoning abilities would have strengthened the study's findings and implications.\n- The paper compares the performance of MLLMs against random and majority baselines but does not include a human performance baseline. Including human performance on the same tasks would have provided a more informative context for interpreting the models' reasoning capabilities and limitations.\n- The authors employ a manual inspection process to assess the reasoning abilities of the models, there may be concerns about the scalability and reproducibility of this approach. Providing more details on the rubric and the inter-annotator agreement could have addressed these concerns and increased the reliability of the manual evaluation results.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RM55IHSde1", "reviewer": "Reviewer_kAJ8", "summary": "This paper aims to study abstract reasoning of multimodal LLMs, more specifically the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven’s Progressive Matrices. The results are mostly negative, showing low performances, with some exceptions is some tasks solved by closed-source models. Chain-of-thought prompting is shown to be useful, as found in the literature.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:\n\n**Q1 & Q2:**\nWe used the term \"nonverbal abstract reasoning,\" as defined by Huang et al. [1] (Section 4.2), referring to the task of providing IQ-style puzzles as images paired with short textual instructions of desired output. Moreover, Huang et al. [1] and Zhao et al. [2] provide evidence of non-trivial performance for such models. Due to our skepticism about the correctness of the underlying reasonings, we investigated this task using a much broader set of models and experimental questions in this work. Moreover, we used labor-intensive manual assessment to provide more grounded insights compared to automatic scores that don't capture the nuances of LLM-generated responses.\n\nTo solve these puzzles, a model must first correctly detect the relations between the presented objects (e.g., a red circle turning into a blue circle in the first row) and then apply them to determine the missing piece (e.g., what would a red square turn into in the second row?), which requires a certain level of analogical reasoning. To the best of our knowledge, this is the first analysis of MLLMs on this task. We will publicly release our codes, datasets, annotations, and assessments to mitigate any concern over reproducibility or annotation quality.\n\n[1] Huang, Shaohan, et al. \"Language is not all you need: Aligning perception with language models,\" NeurIPS 2023\n\n[2] Zhao, Haozhe, et al. \"MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning,\" ICLR 2024\n\n**Q3:**\nAll evaluators and annotators were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience evaluating LLMs. Moreover, each sample was annotated by one person, and any uncertain case was flagged and shared among all three evaluators for discussion. After discussions, the final label was determined by a majority vote.\n\n**Q4:**\nWe ran a study with 25 college-level participants with diverse backgrounds. Each participant was given ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed to provide a short reason for their answer. The average performance of this group was 95.9%, with a 6.92% standard deviation. We will add these results to the paper to emphasize the performance disparity.\n\n**Q5 & Q6:**\n*Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this is an important area of investigation and the paper makes a good attempt to explore some of this space.", "reasons_to_reject": "I don’t think this paper is convincing. Since it is mostly about negative results, we need to be reassured that the process was correct and this is a true negative result, and we are not.\n\n- First, the datasets are visual and the textual part is simply a description of the visual input. So, it is really not clear why the authors expect MLLM to work on this kind of datasets, since the text part does not reproduce the abstract reasoning structure of the visual part. \n                   Q1 What is the reasoning you have applied to think that this kind of data would reveal abstract analogical reasoning?\n\n- Second, the paper is imprecise in its definition of the problem (what exactly constitutes nonverbal reasoning?) and methods used to assess the MLLM. Specifically, the non-computational aspects are  ill-defined and are missing crucial information for reproducibility and assessment of work.\n\n- Q2 What are the qualifications of the result validators and how was their assessment validated?\n- Q3 Why do you state that these tasks are  easy for humans?\n- Q4 How do you define and quantify correct reasoning? \n- Q5 How do you perform Inter-annotator Agreement?", "questions_to_authors": "Please see my question above. They all request more information on how the assessments were done, on the motivations of the procedures for assessment.  This information and level of detail seems to be crucial given the goals of the paper.", "extra_scores": {}}, {"review_id": "xllg35ewJX", "reviewer": "Reviewer_G9SG", "summary": "This work focuses on evaluating MLLMs on `nonverbal abstract reasoning` tasks. The work reveals that the open-sourced models perform poorly on this set of tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:\n\n**Q1:**\nAt the time of submission, we used most of the available state-of-the-art models, including Instruct-BLIP, for our evaluations; however, as you mentioned, keeping up with the rapid pace of changes is costly.\n\n**Q2:**\nSimilar to the previous questions, most of these models were not released during our evaluations. Our only examples of high-res models are Gemini and GPT-4v, which are closed-source. However, it’s important to note that the puzzles we used in our evaluations involve extremely simple shapes, such as squares or circles, with simple features like blue or yellow colors that do not require heavy reliance on detecting small details in a high-resolution image. Moreover, many models performed well on our visual awareness tests (Section 5, Figure 4), showcasing their understanding of the puzzles' details from a purely visual standpoint. At the same time, many of them performed extremely poorly when provided with all the details in a textual format (Section 5, Figure 3), which showcases a gap in reasoning abilities rather than perception abilities.\n\n**Q3:**\nWe have done experiments on both vision-only (Section 5, Figure 4) and text-only variations (Section 5, Figure 3) as detailed in Section 5. However, given the focus on MLLM, we kept the model pool to existing MLLMs so as not to diverge from the paper's goal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work reveals a major limitation of the current open-sourced MLLMs, leaving room for exploration and improvement in this area for future work.", "reasons_to_reject": "- Many MLLMs are considered obsolete, given the rapid pace of advancement in this field. For example, the BLIP-2 series is not instruction-tuned. A more appropriate comparison could be made with Instruct-BLIP.\n- The performance of `nonverbal abstract reasoning` tasks could heavily rely on image resolution. Evaluating MLLMs with higher resolutions (such as LLaVA-Next, VILA-1.5) may be necessary for further investigation, although these models have only recently been introduced.\n- I would suggest adding experiments to ablate (1) the VIT visual encoder (to investigate the impact of higher resolution) and (2) the LLM decoders (to determine which part of MLLMs is more crucial for this task). This would help identify whether the bottleneck lies in visual perception or if performance is bounded by the reasoning ability of LLMs (such as in models 7b, 13b, 34b, or 70b).", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work tests the ability of vision-language models to solve raven progressive matrices puzzles. The authors reveal the difficulties that models encounter in this setup, that closed-book models do better, and that chain-of-thought prompting helps.

In general, reviewers were in agreement this is a thorough study that extends our understanding of the abilities and shortcomings of current models. There were questions on the experimental setup and the authors provided good detail on how annotation was done, who the annotators were, what is human performance, the focus on human-based metrics. etc.
Decision: Accept