The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

Authors: kian  ahrabian
zhivar  sourati
kexuan  sun
jiarui  zhang
yifan  jiang
fred  morstatter
jay  pujara
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 19:15:24

Abstract: While large language models (LLMs) are still being adopted to new domains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs). These models integrate verbal and visual information, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities. However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices. Our experiments reveal the challenging nature of such problems for MLLMs while showcasing the immense gap between open-source and closed-source models. We also uncover critical shortcomings of visual and textual perceptions, subjecting the models to low-performance ceilings. Finally, to improve MLLMs' performance, we experiment with different methods, such as Chain-of-Thought prompting, leading to a significant (up to 100\%) boost in performance.

Main paper: 
 introduction: 
 Foundation models -mostly large language models (LLMs) and large vision models (LVMs) -have revolutionized the field of artificial intelligence, demonstrating zero-shot (Radford et al., 2019; Wang et al., 2023a ) and few-shot (i.e., in-context) learning abilities (Brown et al., 2020; Zhang et al., 2023b ) that perform on-par or even surpass humans in some tasks (Webb et al., 2023) . These tasks cover both dimensions of general intelligence: crystallized intelligence focusing on retrieving knowledge from memory (Hartmann et al., 2023) and fluid intelligence involving novel and abstract reasoning (Cattell, 1987) . Following these advancements, there has been a recent surge in the development of a new generation of foundation models, namely, multi-modal large language models (MLLMs). These models can process visual and textual cues (OpenAI, 2023; Zhao et al., 2023; Huang et al., 2023) , paving the way for solving far more complex tasks concerning both modalities.Nonverbal abstract reasoning is a family of tasks that involve both modalities. It has been studied extensively for measuring fluid intelligence (Shakeel et al., 2017) , in which reasoners need to demonstrate strong visual perception and high-level explicit reasoning abilities to solve these tasks. Prior studies have explored the performance of LVMs (Zhuo & Kankanhalli, 2020) and LLMs (Hu et al., 2023; Webb et al., 2023) on transformed versions of these tasks in a uni-modal setting. However, theoretical and empirical evidence exists for the benefits of the interplay between verbal and visual perceptions (Vyshedskiy, 2019; Winawer et al., 2007; Vygotsky, 1962) , suggesting that visual perception helps us better understand our surroundings, while language helps us symbolize and facilitate reasoning through notions such as self-talk (Berk, 1994) .Inspired by prior works (Vygotsky, 1962; Lupyan, 2012; Nam et al., 2017; Colas et al., 2021) that have explored the fusion of visual and verbal cues and their effect on cognition and reasoning abilities, and considering the opportunity of experimenting with both modalities in MLLMs, in this study, we strive to answer the following research question: "Do MLLMs demonstrate faithful nonverbal abstract reasoning abilities?" Figure 1 : An example of model's prediction on a sample from the IQ50 dataset. Given a prompt with a visual puzzle (top), the model generates a response that includes its reasoning and the chosen option.Our contributions are as follows:1. We evaluate the nonverbal abstract reasoning abilities of 24 open-source and closed-source MLLMs under three Raven's Progressive Matrices (RPM) (Raven, 2003) benchmarks (See Figure 1 for an example); 2. We evaluate MLLMs' textual and visual abilities in semi-isolated settings that mitigate cross-modality contamination, providing insights into their performance ceiling. 3. We evaluate MLLMs' zero-shot and few-shot abilities, drawing a more accurate picture of the alignment between their verbal and visual perceptions;All in all, we observe that while opensource MLLMs perform poorly on nonverbal abstract reasoning tasks, closedsource models such as GPT-4V (Ope-nAI, 2023) showcase non-trivial abilities (See Section 4). Moreover, we discover critical shortcomings in both visual and verbal capabilities across open-source and closed-source models, partially explaining the observed poor performances (See Section 5). Finally, we find closed-source models' textual and visual perceptions to be relatively aligned, allowing us to improve their performance significantly by providing guided prompts and in-context demonstrations (See Section 6).MLLMs (Li et al., 2022; Chen et al., 2022; Alayrac et al., 2022; Li et al., 2023a; Wang et al., 2022; Zhu et al., 2023a) that can generate text conditioned on the combination of different modalities, demonstrating zero-shot (Li et al., 2022) , few-shot (Alayrac et al., 2022; Zhao et al., 2023) , and chain-of-thought abilities (Huang et al., 2023) . To better understand their range of abilities, prior studies have evaluated MLLMs for geometric understanding and reasoning (Kazemi et al., 2023 ), text recognition (Liu et al., 2023b) , mathematical reasoning (Lu et al., 2024) , college-level deliberate reasoning (Yue et al., 2023) , and open-ended reasoning (Han et al., 2023) abilities. Most similar to our study are the works of Qi et al. (2023) and Mitchell et al. (2023) , evaluating the abstract reasoning abilities of MLLMs; however, their evaluation is either limited to a few examples or lack the rigor to provide an in-depth understanding.In this study, we bridge the gap in the literature by conducting extensive experiments that produce comprehensive insights into MLLMs' abstract nonverbal reasoning abilities. 
experimental setup: 
 3.1 Datasets IQ50. Introduced by Huang et al. (2023) , IQ50 is a nonverbal reasoning benchmark containing 50 visual puzzles crawled from the internet. Given a set of images arranged in a matrix or a sequence, the goal is to predict the missing piece of the puzzle from six given options.To conduct our extensive experiments, we augment each puzzle with textual description and hint annotations to explore the interplay between textual cues and visual perceptions.Introduced by Zhang et al. (2019) , RAVEN is a visual reasoning dataset containing 70,000 synthetic samples in seven categories (each 10,000). Each instance is created following a sampled rule and contains a 3x3 matrix of images (with the bottom right piece missing) and eight options, with a goal similar to IQ50. To reduce the computational costs, we randomly sample 500 examples from each category (3,500 in total) to create RAVEN-S.We collected 175 visual abstractions and reasoning problems from the China Civil Service Examination (CCSE), each containing a matrix or a sequence of images with four options. This newly curated dataset serves as a challenging benchmark across various reasoning patterns. See Appendix F for more details.Pre-Trained. The first set of models that we utilize in our experiments are state-of-the-art pre-trained MLLMs. Specifically, we chose the following models due to their popularity and accessibility: 1) BLIP-2 (Li et al., 2023a) , 2) Fuyu (Bavishi et al., 2023) , 3) IDEFICS (Laurenc ¸on et al., 2023) , and 4) Qwen-VL (Bai et al., 2023) . Note that all these models, except for the BLIP-2 family, have undergone a multi-task pre-training procedure, presumably allowing them to have zero-shot abilities on a wide range of tasks (See Appendix B).Instruction-Tuned. The second set of models that we use in our experiments are state-ofthe-art instruction-tuned MLLMs. Specifically, we chose the following models with similar criteria as pre-trained MLLMs: 1) InstructBLIP (Dai et al., 2023) , 2) MMICL (Zhao et al., 2023) , 3) LLaVA (Liu et al., 2023a ) 4) IDEFICS (Laurenc ¸on et al., 2023) , 5) Qwen-VL (Bai et al., 2023) , 6) GPT-4V (OpenAI, 2023), and 7) Gemini (Google, 2023).Heuristics. Since most of our samples follow similar spatial patterns and the task is to fill the missing image with the best candidate, we can write the expected representation of the target image as a function of the provided query images. To this end, we first compute the target image's expected representation asEQUATIONwhere R is a function mapping images to vector representations, Q is the set of all query images, and α q ∈ R is the weight of image q. For simplicity, we only consider linear combinations. Then, we select the candidate p that has the highest similarity to r as our prediction:EQUATIONwhere C is the set of all candidate images, and S is the Euclidean distance. For example, in Figure 1 , a heuristic could be formulated as:q 12q 11 = q 22q 21 ⇒ q 22 = q 21 + q 12q 11 , (3) which leads to α q 11 = -1, α q 12 = 1 and α q 21 = 1. See Appendix C for details on selecting R and calculating α q .Control Baselines. We also report the random and majority-based performance for all the datasets as control baselines, enabling us to put the posted performances in perspective. 4 How good are MLLMs at nonverbal abstract reasoning? 
automatic scoring: 
 Our early observations of the MLLMs' responses indicated the possibility of option markers (i.e., 1, 2, etc.) appearing at different token positions. For example, if the model generates "Number 4.", the marker will be at position 3; however, if the model generates "The answer is 4.", the marker will be at position 7. As such, we pivoted away from the common next-token scoring approach and used a pattern-matching scheme. Specifically, we find all the number tokens in the response and then take the one with the largest logit as the chosen option to be compared to the ground truth. We further discuss this approach in Appendix A, making a comparison to another common automatic scoring method. consistently beats the random baselines over the three datasets. Moreover, compared to the random baselines, the models perform within the [-8.7%, +7.3%], [-1.0%, +0.6%], and [-5.6%, +6.4%] ranges for IQ50, RAVEN-S, and CCSE, respectively. Although some models achieve non-trivial gains over the random baselines and even outperform some of the heuristic baselines, these results expose the difficulty of solving variations of the RPM-style questions across a wide range of MLLMs.Comparing the best-achieved performances with the majority baselines, we observe a stark similarity (±2%). This begs the question of whether the posted numbers are true representatives of the reasoning powers of MLLMs or simply a side-effect of their generation biases (e.g., a model that always generates 1s will do very well on a dataset with many 1s as gold labels). To this end, in Section 4.2, we manually examined the generated answers by the instruction-tuned models. Moreover, looking at the results posted by the pre-trained models, apart from fuyu-8b, we cannot observe a significant upside to the multi-task pre-trained models.It is also worth noting that, by examining the results posted within and beyond the same family of models, we observe that the scaling law in terms of model size (Kaplan et al., 2020) (i.e., the larger the model, the higher the performance) does not hold here. Figure 2 illustrates the models' zero-shot accuracy concerning the number of parameters. One critical aspect of assessing such models' abilities is ensuring the correctness and faithfulness of their reasoning.As such, we manually inspect the generated responses to provide insights into the results posted by the models in the automatic scoring schemes. All the inspections were done by a group of three graduate students (See Appendix D for the rubric). Moreover, to elicit reasoning, we appended the phrase "Let's think step by step." to the prompt (Kojima et al., 2022) . Out of the 14 open-source instruction-tuned models (See Table 3 ), we were only able to get meaningful and coherent responses from the following models: llava-1.5-7b-hf, llava-1.5-13b-hf, Qwen-VL-Chat, and idefics-9b-instruct.Besides these models, we also manually measured the abstract reasoning abilities of closedsource MLLMs (i.e., gpt-4v and gemini-pro-vision), as we don't need access to the gen-erated logits anymore. For the remainder of our experiments, we relied on IQ50, a small, challenging test set that is easy for humans to solve (see Appendix E), based on our preliminary tests, but difficult for MLLMs (See Table 1 ). This choice also allows us to expand our study more efficiently with various experiments.Table 2 presents the performance of the aforementioned models on IQ50, demonstrating their poor explicit reasoning capabilities. More specifically, only one of the open-source instruction-tuned models achieves a non-zero performance on the joint answer and reasoning correctness, with an abysmal performance of 4%, (i.e., an average of 1% across open-source models). Meanwhile, the trend in the closed-source MLLMs is more promising, with gpt-4v outperforming random and majority baseline, providing correct reasoning and answers in 26% of the samples. Nevertheless, their performance still lags behind the simple heuristics by a large margin (See Table 1 ). Regarding the faithfulness of answers to reasonings, we only observe a meaningful level of faithfulness (alignment between reasoning and answer when either is correct) in closed-source models, 50% for gpt-4v and 25% for gemini-pro-vision. Observing such a high percentage of unfaithfulness to reasoning further asserts the importance of conducting manual evaluations rather than merely reporting performances using noisy automatic measurements. During our inspections, we observed that although models generally understand the shapes in the query images, they tend to hallucinate about specific features such as rotation, shadows, and orientations. On the reasoning side, the most prominent issue was the models being overly descriptive rather than being focused on providing grounded logical reasoning. In other words, most of the time, the generated responses were the descriptions of the puzzle and the candidate options. We believe this to be an artifact of their training datasets; as such, we expected a much better showing from the models that receive multi-task training (See Table 3 ).5 What are the root causes of the poor performance? 6 Can MLLMs' performance be improved? 
guided prompting: 
 Prompt engineering has been one of the prominent methods to guide LLMs towards specific desired outputs through better conditioning (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2023b; Li et al., 2023b ). We experiment with three guided prompting setups, each providing the models with different cues to understand how supplementary textual information is utilized while generating responses. The three setups are as follows (See Appendix G for examples):• General. In this setup, we provide the models with broad cues on approaching such visual puzzles, hinting at the common strategies without being sample-specific.• Sample-specific. In this setup, we provide the models with one sample-specific hint about the desired reasoning for solving each puzzle.• Corrective. In this setup, in an interactive process, looking at the model's reasoning when prompted in a zero-shot setting, we add one hint to correct the most prominent error. Figure 5 illustrates the performance of closed-source models on IQ50 given different hints. Looking at these results, we find sample-specific and general hints detrimental to reasoning and accuracy while observing a significant boost when interactively utilizing corrective hints, especially for gpt-4v. Considering the distinct embedded cues of sample-specific and corrective hints, we believe that the models' inherent chain of reasoning on the provided puzzles is misaligned with humans, which makes sample-specific hints confusing rather than helpful. However, we can correct models' already laid-out solutions with corrective hints, improving their performance by as much as 100%. In-context learning (ICL) refers to emergent behavior in LLMs where they perform a task conditioned on the provided demonstrations without further parameter optimization. Min et al. (2022) have suggested that ICL is a mere mechanism for locating the already learned ability of the model to respond to the query prompt. However, more recent studies have shown that LLMs can learn various function classes through ICL (Garg et al., 2022; Mirchandani et al., 2023; Lee et al., 2023) .The most common form of ICL is few-shot, in which the model is provided with demonstrations similar to the test sample. We call this variation "Symmetrical" as there is no imbalance in the demonstrations' textual and visual information pieces. Since symmetrical ICL requires processing multiple image and text pairs, we only utilize models capable of processing such inputs (See Table 3 for a list). Moreover, due to burdensome computation costs, we exclude idefics-80b* models. In our experiments, we explored the effect of 1) changing the sampling distribution of demonstrations and 2) including step-by-step reasoning with demonstrations (See Appendix G for examples).Our experiments cover open-source models and utilize our automatic scoring scheme over two variations: 1) In-Distribution (ID): demonstrations are uniformly taken from the same dataset, and 2) Out-of-Distribution (OOD): demonstrations are uniformly taken from another dataset. We used IQ50 as the evaluation source with RAVEN-S as the OOD source. Moreover, we ran each variation five times to reduce the impact of random sampling and up to 5-shot (i.e., 10% of the dataset) due to GPU limits. Figure 6 presents the results of our experiments. As evident, there are no consistent patterns across models and variations. For example, while idefics-9b* models generally benefit from the few-shot scheme, we don't see monotonically increasing performances with more demonstrations, contrary to expected patterns in LLMs.Simultaneously, we see a consistent decline in performance in some variations of Qwen-VL* models. We believe these irregularities are caused by an inability to understand the utility of the demonstrations, suggesting that the tested models do not exhibit strong and consistent symmetrical ICL capabilities. As a result, the models cannot take advantage of the provided demonstrations properly, leading to poor responses and subpar performances.Effect of step-by-step reasoning. Inspired by its success, we experiment with the fewshot Chain-of-Thought (CoT) prompting (Wei et al., 2022) to improve the performance of our models. Our early experiments found open-source models, such as Qwen-VL-Chat and idefics-9b-instruct, unable to comprehend CoT prompting. As such, we focused on examining the closed-source models: gpt-4v and gemini-pro-vision.Figure 7 presents the performance of closed-source models with CoT prompting. We can observe that both models benefit significantly from CoT demonstrations, with gpt-4v's performance being boosted as much as 100%. These results emphasize the immense gap between the open-source and closed-source models while showcasing meaningful symmetrical ICL abilities in these models. During our evaluation, we noticed an unusual phenomenon with gemini-pro-vision in which the one-shot variation prevented the model from generating reasonings in almost all the examples, leading to an initial performance drop. Moreover, in a similar setting for gpt-4v, we observed 1) a massive jump in the number of safeguard triggers (4% → 18%), which precluded the model from generating a response, and 2) an increased confusion regarding the boundaries of the provided demonstration (0% → 10%), diminishing the model's performance.Adding new modalities has brought forth the possibility of providing lopsided (i.e., Asymmetrical) information in the input prompt. As such, we conduct a set of few-shot experiments that provide the models with text-only CoT demonstrations while keeping the query unchanged (i.e., image + text). To encourage the models to use the demonstrations, we append "Let's solve the puzzle in the image, step by step, similar to the demonstrations." to our prompt. We hypothesize that similar to the findings of Min et al. (2022) , the models will better understand the input and output spaces and achieve improved performance. Since we could not make open-source models properly utilize the textual demonstrations in our preliminary experiments, we continued our experiments only on closed-source models.Figure 8 illustrates the results of our experiments with asymmetrical fewshot CoT prompting. As evident, this approach does not yield meaningful improvements and even causes degradation in some cases, contrasting our hypothesis. Moreover, we noticed increased hallucinations, mostly confusing the demonstrations with queries and detecting non-existent details in the shapes, leading to unfaithfulness and instability in the responses. We leave further investigations and experiments on this ICL scheme to future works. 
conclusion: 
 In this study, we utilized different RPM-style tasks as a proxy for measuring the nonverbal abstract reasoning abilities of MLLMs, covering 24 different open-source and closed-source models. Although closed-source MLLMs showcased promising capabilities in our experiments, we found the abilities of open-source models to be insufficient for solving these tasks. Moreover, using pseudo-isolated experimental environments, we found that MLLMs often fail at 1) gathering precise visual details from puzzles and 2) reasoning correctly and faithfully, even when provided with expert-written and complete descriptions of the puzzles. Furthermore, our experiments highlighted 1) the inability of open-source models to consistently utilize demonstrations and 2) the ICL prowess of the closed-source models, which helped them benefit from interactive guidance or provided demonstrations with stepby-step (i.e., CoT) reasoning. Although MLLMs have previously demonstrated proficiency at various tasks, our study using a relatively simple reasoning task for humans has exposed some critical shortcomings in MLLMs while highlighting the importance of more grounded evaluations, even at small scales. Introduced by Huang et al. (2023) , in this scoring method, we first flatten the query image matrix and feed it into the model along with exactly one option. We also surround these images with textual instructions to help the model better understand the desired task (See Appendix G for examples). Then, we calculate the probability of the model generating "Yes", representing the probability of that option being the true missing piece. Finally, we determine the model's choice by taking the option with the highest probability. To improve the original method, we introduce an equivalency condition in which the max probability of different variations of the target token (e.g., "YES" and "yes") is taken as the probability. The main shortcomings of this method include 1) being more computation-heavy as it needs to process each option separately and 2) being compatible with only the models that accept multiple images as input. Notably, among the open-source models utilized in this study, only Qwen-VL* and idefics* models support multi-image inputs (See Table 3 ). 
 
 Appendix: Figure 9 compares the experimental results with the one-by-one and our automatic scoring methods. As evident, some models such as MMICL-Instructblip-T5-xxl, idefics-9b, and idefics-9b-instruct benefit significantly from this scoring mode, while others such as instructblip-flan-t5-xxl and Qwen-VL-Chat suffer extensively. This observation demonstrates the validity of both methods, making the choice subject to the task/dataset being evaluated. However, given these mixed results and the downsides of using this scoring method, we conclude that our automatic scoring method is more practical for future studies.Table 3 presents an attribute comparison over open-source and closed-source models.). BLIP-2 is a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer (Q-former), which is pre-trained in two stages. Despite having significantly fewer trainable parameters than existing methods, BLIP-2 achieves state-of-the-art performance on various vision-language tasks. ✓ ✗ ✗ blip2-opt-6.7b 7.8b ✓ ✗ ✗ blip2-flan-t5-xl 3.9b ✓ ✗ ✗ blip2-flan-t5-xxl 12.2b ✓ ✗ ✗ idefics-9b 8.9b ✓ ✗ ✓ idefics-80b 80.0b ✓ ✗ ✓ fuyu-8b 9.4b ✓ ✓ ✗ Qwen-VL 9.7b ✓ ✓ ✓ Instruction-Tuned gpt-4-vision-preview U ✗ U ✓ Bard (Gemini Update) U ✗ U ✓ MMICL-vicuna-7b 7.9b ✓ ✓ ✓ * MMICL-vicuna-13b 14.2b ✓ ✓ ✓ * MMICL-Instructblip-T5-xl 4.0b ✓ ✓ ✓ * MMICL-Instructblip-T5-xxl 12.3b ✓ ✓ ✓ * instructblip-vicuna-7b 7.9b ✓ ✓ ✗ instructblip-vicuna-13b 14.2b ✓ ✓ ✗ instructblip-flan-t5-xl 4.0b ✓ ✓ ✗ instructblip-flan-t5-xxl 12.3b ✓ ✓ ✗ idefics-9b-instruct 8.9b ✓ ✗ ✓ idefics-80b-instruct 80.0b ✓ ✗ ✓ llava-1.5-7b-hf 7.1b ✓ ✓ ✗ llava-1.5-13b-hf 13.4b ✓ ✓ ✗ bakLlava-v1-hf 7.6b ✓ U ✗ Qwen-VL-Chat 9.7b ✓ ✓ ✓Table 3 : Comparison of open-source and closed-source models' attributes. Legend: U → Undisclosed, * → We could not utilize this feature using the official code.blocks to bridge the gap between the vision encoder and the language model. The model is trained on image-text pairs and unstructured multi-modal web documents. IDEFICSinstruct is the model obtained by further training IDEFICS on Supervised Fine-Tuning and Instruction Fine-Tuning datasets. IDEFICS is on par with the original closed-source model on various image-text benchmarks, including visual question answering (open-ended and multiple choice), image captioning, and image classification when evaluated with in-context few-shot learning.Qwen-VL models are large-scale vision-language models (LVLMs) designed to perceive and understand texts and images. Starting from the Qwen-LM as a foundation, the model is endowed with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multi-modal cleaned corpus. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales.InstructBLIP (Dai et al., 2023) . InstructBLIP models are instruction-tuned MLLMs based on the pre-trained BLIP-2 models. They have been trained using 13 publicly available datasets transformed into instruction tuning format. Additionally, the authors introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models.LLaVA-1.5 (Liu et al., 2024) . LLaVA-1.5 achieves state-of-the-art performance on various multimodal benchmarks through increased input resolution, additional layers of multimodal projection, and a comprehensively curated visual instruction tuning dataset. The model's efficient and lightweight architecture facilitates high reproducibility, positioning it as a versatile foundation for further research and development in the multimodal community.MMICL (Zhao et al., 2023) . Unlike previous work, MMICL utilizes a novel context scheme, treating image and text representations equally and establishing the reference between image and text via image declaration. It enables users to have the flexibility to input multiple images and text in any desired order, with no restrictions on the quantity or placement of images in contexts. MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Moreover, MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges with impressive ICL ability.GPT-4V (OpenAI, 2023). GPT-4 with vision (GPT-4V) enables users to instruct GPT-4 to analyze image inputs provided by the user and is the latest capability OpenAI is making broadly available.). The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use cases. Evaluation on a broad range of benchmarks shows that the Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks -notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU and improving the state-of-the-art in every one of the 20 multi-modal benchmarks the authors examined. At the time of this publication, only the Pro version was publicly available. As such, we select CLIP encoding as the second variation of R in our heuristics.We first identify all possible 2 × 2 submatrices in the m × n query matrix. For the horizontal axis, considering that each 2 × 2 pattern spans across two columns, the number of unique horizontal positions is n -1. Similarly, m -1 unique vertical positions are on the vertical axis. Therefore, the total number of distinct 2 × 2 patterns that can be extracted from an m × n matrix is (m -1) × (n -1). To determine the expected target representation, we first apply the formula 4 = 3 + 2 -1 to each submatrix; then, we average over all the calculated values. Finally, we choose the option with the smallest Euclidean distance (i.e., S) to the expected target representation.Our main challenge was to overcome the nuances that appear in reasonings generated by LLMs. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples in the dataset and 2) review a series of sample responses generated by the models (e.g., Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths. For example, in some instances, the models perceived shadows (or incorrect colors) in the shapes that were not present in the puzzle; however, as long as they correctly detected the row-wise and column-wise change of patterns (e.g., square turning to circle) and grounded their reasoning on them, the evaluators marked the response as correct. Moreover, each sample was annotated and assessed by one person, and any uncertain case was flagged and shared among all three evaluators for discussion. After discussions, the final label was determined by a majority vote (i.e., at least 2 out of 3). All evaluators were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience working with and evaluating LLMs.While we acknowledge the difficulty of scaling such manual experiments, one of the main challenges of correctly assessing generated responses is that automatic metrics like ROUGE, BERTScore, etc., fall short of adequately evaluating the semantic nuances. Hence, we decided to go down the path of highly labor-intensive human expert evaluation to provide precise, concrete, and grounded insights.To establish a baseline, we ran a study with 25 college-level participants from diverse educational and demographic backgrounds. We provided each participant with ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed them to solve the puzzle and give a short reason for their answer. The average performance of this group was 95.9%, with a standard deviation of 6.92%.CCSE tests models' reasoning abilities over five general patterns (i.e., location, logic, progression, self-geometry, and relative-geometry) in three types of figure configurations (i.e., one-row, two-rows, matrix). See Figure 10 and Figure 11 for examples. Table 4 and Table 5 present examples of each prompt used in our experiments.In this work, we focused on only a specific type of reasoning task (i.e., nonverbal abstract reasoning) while using only IQ50, a small but challenging dataset, in most of our experiments. However, despite these limitations, we hypothesize that similar shortcomings could be replicated in other reasoning tasks due to the fundamental and not task-specific nature of the observed problems. Moreover, although our experiments yielded insightful results, further analysis of the observations is still possible. For example, given the remarkable effectiveness of corrective hints, we can utilize methods such as self-talk Press et al. (2023) , automating the whole process. Another example is the visual awareness tests, where it is possible to examine the internal values of the models (e.g., attention weights, token probabilities, etc.), as opposed to evaluating the generated responses. Finally, since the closed-source models' training datasets are unknown, test set contamination is possible, leading to their superior performance. However, based on the relatively low performance of these models and considering the low-level difficulty of the tests for humans, we believe contamination to be unlikely.Automatic scoring[IMG] You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best?One-by-one scoring Here are three images: [IMG1] [IMG2] [IMG3]In candidate 4 at the bottom, are the arrows arranged clockwise or counterclockwise?Guided prompting (General)[IMG] You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best? Hint: Focus on the row-wise and column-wise changes regarding color, orientation, and shape of the puzzle pieces. Let's think step by step.Guided prompting (Sample-specific)[IMG] You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best? Hint: the focus should be on the column-wise changes. Let's think step by step.Guided prompting (Corrective) Turn 1.[IMG] You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best? Let's think step by step.[Model's Response] Turn 2. Hint: Option 1 does not have a small circle inside it, and option 5 is a very small circle itself.Symmetrical few-shot You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best? Let's think step by step.[IMG1]The answer is 4.[IMG2]The answer is 1.[IMG3]Table 4 : Prompt examples for our experiments. Note that all "[IMG*]" are replaced with actual images during inference.Symmetrical few-shot CoT You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best?[IMG1]To solve this puzzle, we need to identify a pattern or rule that applies to the rows or columns of the matrix. Let's examine the rows and columns to see if we can discern any patterns.Looking at the first row, we see a yellow percentage sign followed by a yellow percentage sign. Hence, nothing changes in the row, moving from left to right. In the second row, there's a yellow percentage sign. Following the above pattern, we deduce that the missing piece in the second row is a yellow percentage sign. Now, let's look at the columns. The first column has a yellow percentage sign, followed by a yellow percentage sign. Hence, nothing changes in the column, moving from top to bottom. The second column has a yellow percentage sign. Following the above pattern, the missing piece in the second column must be a yellow percentage sign. Combining the observations from rows and columns, we conclude that the missing piece is a yellow percentage sign.The provided options at the bottom are as follows:1. Yellow percentage sign 2. Yellow plus sign 3. Two yellow circles 4. One yellow circle 5. Yellow division sign 6. Yellow cross Given these options and our conclusion, option 1 fits our criteria and best fills the missing piece.[IMG2]Asymmetrical few-shot CoT You are given a puzzle. The puzzle features a set of visual patterns arranged in a matrix on the top, with the bottom right piece missing and six options at the bottom (marked by 1, 2, 3, 4, 5, or 6). Which option (either 1, 2, 3, 4, 5, or 6) fills the missing piece best? Demonstration 1: Puzzle: [[yellow percentage sign, yellow percentage sign],[yellow percentage sign, ?]] Options: 1: yellow percentage sign 2: yellow plus sign 3: two yellow circles 4: one yellow circle 5: yellow division sign 6: yellow cross To solve this puzzle, we need to identify a pattern or rule that applies to the rows or columns of the matrix. Let's examine the rows and columns to see if we can discern any patterns.Looking at the first row, we see a yellow percentage sign followed by a yellow percentage sign. Hence, nothing changes in the row, moving from left to right. In the second row, there's a yellow percentage sign. Following the above pattern, we deduce that the missing piece in the second row is a yellow percentage sign. Now, let's look at the columns. The first column has a yellow percentage sign, followed by a yellow percentage sign. Hence, nothing changes in the column, moving from top to bottom. The second column has a yellow percentage sign. Following the above pattern, the missing piece in the second column must be a yellow percentage sign. Combining the observations from rows and columns, we conclude that the missing piece is a yellow percentage sign.The provided options at the bottom are as follows:1. Yellow percentage sign 2. Yellow plus sign 3. Two yellow circles 4. One yellow circle 5. Yellow division sign 6. Yellow cross Given these options and our conclusion, option 1 fits our criteria and best fills the missing piece.Let's solve the puzzle in the image, step by step, similar to the demonstrations.[IMG]Table 5 : Prompt examples for our experiments (Continued). Note that all "[IMG*]" are replaced with actual images during inference. 


Decision: This work tests the ability of vision-language models to solve raven progressive matrices puzzles. The authors reveal the difficulties that models encounter in this setup, that closed-book models do better, and that chain-of-thought prompting helps.

In general, reviewers were in agreement this is a thorough study that extends our understanding of the abilities and shortcomings of current models. There were questions on the experimental setup and the authors provided good detail on how annotation was done, who the annotators were, what is human performance, the focus on human-based metrics. etc.

Reviews: review: paper_summary: paper_summary: This study explores the nonverbal abstract reasoning capabilities of multi-modal large language models (MLLMs) using variations of Raven's Progressive Matrices, a benchmark traditionally used to assess human cognitive abilities. The research aims to bridge the gap in understanding the complex reasoning abilities of MLLMs, which integrate both verbal and visual information. Through rigorous testing, the study reveals significant performance disparities between open-source and closed-source MLLMs and identifies limitations in their visual and textual processing capabilities. The authors also investigate various enhancement strategies, notably Chain-of-Thought prompting, which dramatically improves the models' reasoning performance.
strength_weakness: strength_weakness: 1. The reviewers has some concerns on the current experiment settings.

a. The impact of resolution and position representation. Current set up doesn't ablate much on whether the limited performance especially on zero-shot comes from the fact that the model is more suitable for other representation of locations and resolution. For example, is it possible that certain issue comes from the resolution/scale of the main object in the image? Is it possible that it is better to use language description of the locations such as top right, top left, bottom right and bottom right to let the model better understand the relative positions?

b. The manual evaluation lacks more rigorous analysis. There are only 3 annotators and the rubric is vague on the hallucination part. It is also unclear about inter-annotator agreement on the annotation results and the current setup makes it almost impossible to reproduce the results on one of the most important parts of the work: reasoning quality. It is also worthy to set up an automatic score for evaluation of reasoning process.
questions: questions: Will there be copy right issues of using these data?
paper_summary: paper_summary: This paper presents a comprehensive evaluation of the nonverbal abstract reasoning abilities of multi-modal large language models (MLLMs) using variations of Raven's Progressive Matrices (RPM) as benchmarks. The authors test a wide range of open-source and closed-source MLLMs, providing insights into their current capabilities and limitations. The experiments reveal that while closed-source models like GPT-4V demonstrate non-trivial reasoning abilities, open-source models generally struggle with these tasks. The paper also uncovers critical shortcomings in both the visual and textual reasoning capabilities of MLLMs, setting performance ceilings. Interestingly, the authors find that techniques like Chain-of-Thought prompting and corrective hints can significantly boost performance, especially for closed-source models.
strength_weakness: strength_weakness: - The use of Raven's Progressive Matrices and techniques like Chain-of-Thought prompting have been explored in previous studies, albeit not as extensively as in this work.
- The paper compares a wide range of MLLMs but does not provide a detailed analysis of how the architectural differences between these models might influence their reasoning capabilities. A more thorough investigation of the relationship between model architecture and performance on abstract reasoning tasks could have provided additional insights and made the contributions more substantial.
- The authors acknowledge that the superior performance of closed-source models could be due to the nature of their training data. However, the paper does not delve into this aspect in detail. How fine-tuning on these tasks improve the open-source models' performance? A more comprehensive analysis of the potential impact of training data on the models' reasoning abilities would have strengthened the study's findings and implications.
- The paper compares the performance of MLLMs against random and majority baselines but does not include a human performance baseline. Including human performance on the same tasks would have provided a more informative context for interpreting the models' reasoning capabilities and limitations.
- The authors employ a manual inspection process to assess the reasoning abilities of the models, there may be concerns about the scalability and reproducibility of this approach. Providing more details on the rubric and the inter-annotator agreement could have addressed these concerns and increased the reliability of the manual evaluation results.
paper_summary: paper_summary: This paper aims to study abstract reasoning of multimodal LLMs, more specifically the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven’s Progressive Matrices. The results are mostly negative, showing low performances, with some exceptions is some tasks solved by closed-source models. Chain-of-thought prompting is shown to be useful, as found in the literature.
strength_weakness: strength_weakness: I don’t think this paper is convincing. Since it is mostly about negative results, we need to be reassured that the process was correct and this is a true negative result, and we are not.

- First, the datasets are visual and the textual part is simply a description of the visual input. So, it is really not clear why the authors expect MLLM to work on this kind of datasets, since the text part does not reproduce the abstract reasoning structure of the visual part. 
                   Q1 What is the reasoning you have applied to think that this kind of data would reveal abstract analogical reasoning?

- Second, the paper is imprecise in its definition of the problem (what exactly constitutes nonverbal reasoning?) and methods used to assess the MLLM. Specifically, the non-computational aspects are  ill-defined and are missing crucial information for reproducibility and assessment of work.

- Q2 What are the qualifications of the result validators and how was their assessment validated?
- Q3 Why do you state that these tasks are  easy for humans?
- Q4 How do you define and quantify correct reasoning? 
- Q5 How do you perform Inter-annotator Agreement?
questions: questions: Please see my question above. They all request more information on how the assessments were done, on the motivations of the procedures for assessment.  This information and level of detail seems to be crucial given the goals of the paper.
paper_summary: paper_summary: This work focuses on evaluating MLLMs on `nonverbal abstract reasoning` tasks. The work reveals that the open-sourced models perform poorly on this set of tasks.
strength_weakness: strength_weakness: - Many MLLMs are considered obsolete, given the rapid pace of advancement in this field. For example, the BLIP-2 series is not instruction-tuned. A more appropriate comparison could be made with Instruct-BLIP.
- The performance of `nonverbal abstract reasoning` tasks could heavily rely on image resolution. Evaluating MLLMs with higher resolutions (such as LLaVA-Next, VILA-1.5) may be necessary for further investigation, although these models have only recently been introduced.
- I would suggest adding experiments to ablate (1) the VIT visual encoder (to investigate the impact of higher resolution) and (2) the LLM decoders (to determine which part of MLLMs is more crucial for this task). This would help identify whether the bottleneck lies in visual perception or if performance is bounded by the reasoning ability of LLMs (such as in models 7b, 13b, 34b, or 70b).

Comments: title: reply to author's response
Details of Manual Evaluation and Human Performance
comment: Thanks for your convincing clarifications, I think it is important you find a way of including both the explanation of the motivation and a much more detail description of the assessment procedure in the final paper. I assume you will, and I have changed my score accordingly.
Dear Reviewer G9SG,

Thank you again for your time. As we approach the end of the discussion period, we believe our response has addressed your questions and concerns. Please let us know if you have any further concerns or questions; we will be more than happy to address them.

Best
Dear Reviewer kAJ8,

Thank you again for your time. As we approach the end of the discussion period, we believe our response has addressed your questions and concerns. Please let us know if you have any further concerns or questions; we will be more than happy to address them.

Best
Dear Reviewer QZHA,

Thank you again for your time. As we approach the end of the discussion period, we believe our response has addressed your questions and concerns. Please let us know if you have any further concerns or questions; we will be more than happy to address them.

Best
Dear Reviewer XKpq,

Thank you again for your time. As we approach the end of the discussion period, we believe our response has addressed your questions and concerns. Please let us know if you have any further concerns or questions; we will be more than happy to address them.

Best
Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.
Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.
Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.
We thank the reviewers for their insightful feedback. We are encouraged that they agreed with us on the timely manner and importance of this topic (R1, R3) and have found our analysis to be rich (R1) and insightful (R2, R3), revealing significant shortcomings in the current open-source MLLMs (R2, R4) and enabling future research (R1, R4). In the following, we address their questions and concerns regarding human performance and manual evaluation:

**Human Performance:**
To address the questions regarding the comparison to human performance, we ran a study with 25 college-level participants from diverse educational and demographic backgrounds. We provided each participant with ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed them to solve the puzzle and give a short reason for their answer. The average performance of this group was **95.9%**, with a standard deviation of 6.92%. We will add these results to the paper to further showcase the performance disparity.

**Manual Evaluation:**
Here, we provide more details to address the questions regarding the manual evaluation procedures:

a) Regarding the demographic and expertise of the evaluators and annotators, all were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience working with and evaluating LLMs.

b) Regarding the rubric and the inter-annotator agreement, our main challenge was to overcome the nuances that appear in reasonings generated by LLMs. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples in the dataset and 2) review a series of sample responses generated by the models (e.g., GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths.

For example, in some instances, the models perceived shadows (or incorrect colors) in the shapes that were not present in the puzzle; however, as long as they correctly detected the row-wise and column-wise change of patterns (e.g., square turning to circle) and grounded their reasoning on them, the evaluators marked the response as correct.

Moreover, each sample was annotated and assessed by one person, and any uncertain case was flagged and shared among all three evaluators for discussion. After discussions, the final label was determined by a majority vote (i.e., at least 2 out of 3).

c) Regarding the scalability of evaluations, while we acknowledge the difficulty of scaling such manual experiments, one of the main challenges of correctly assessing generated responses is that automatic metrics like ROUGE, BERTScore, etc., fall short of adequately evaluating the semantic nuances. Hence, we decided to go down the path of highly labor-intensive human expert evaluation to provide precise, concrete, and grounded insights.

While we acknowledge that manual evaluation for LLM-generated responses is nuanced and challenging, we tried to avoid any pitfall that could invalidate our results (e.g., using only expert PhD students for assessments and annotations). Furthermore, we will release the models' outputs and the evaluators' annotations to mitigate further concerns about the quality of the evaluations.