Reasoning Limitations of Multimodal Large Language Models. A case study of Bongard Problems
Authors: Mikołaj Małkiński
Szymon Pawlonka
Jacek Mańdziuk
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models
Abstract Visual Reasoning
Bongard Problems
Primary area: deep_learning->foundation_models

Abstract: Abstract visual reasoning (AVR) involves discovering shared concepts across images through analogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal description. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited solution strategies and testing $4$ proprietary and $4$ open-access models on $3$ BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard-OpenWorld) images. Despite some successes on real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-RWR, a dataset representing synthetic BP concepts using real-world images. Our findings suggest that weak MLLM performance on classical BPs is not due to the domain specificity, but rather comes from their general AVR limitations. Code and dataset are available at: https://github.com/pavonism/bongard-rwr

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Mikolaj Malkiński! Szymon Pawlonka
Jacek Mańdziuk 2
Abstract
Abstract visual reasoning (AVR) involves discov- ering shared concepts across images through anal- ogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal de- scription. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited so- lution strategies and testing 4 proprietary and 4 open-access models on BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard- Saeerd-OppnWWnoo World) images. Despite successes real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-Ryy RRR,, a dataset represent- ing synthetic BP concepts using real-world im- ages. Our findings suggest that weak MLLM performance on classical BPs is not due to the do- main specificity, but rather comes from their gen- eral AVR limitations. Code and dataset are avail- able at: https: / // witthub.com com/ pavonism/ bongard-rwr
X
8
(a) Bongard Problem #31
(b) Bongard -Winddd-RRRRa- #31
A a
O
A
V
o
4
V
V
(c) Bongard Problem #36
(d) Bongard-RWP #36
Figure I: Bongard Problems. (a), (c): Manually-designed
synthetic BPs (Bongard, 1970). (b), (d): Their real-world
equivalents from the Bongard-RWe dataset proposed in this work. BP #31: Left: One line. Right: Two lines. BP #36;
Left: Triangle above circle. Right: Circle above triangle.
resentations are already given. Chalmers et al. (1992) high- light that forming useful representations is an intricate chal- lenge. In particular, perception is not merely a passive reception of sensory data, but rather an active interpreta- tion influenced by prior knowledge. This process involves the detection of patterns, recognition of analogies, and ab- straction of concepts. The resultant representations may vary significantly depending on the context, which under- scores the importance of modeling perception and cognition jointly (Hofstadter, 1995).
1. Introduction
mtalogy-making is a critical aspect of human cognition, tightly linked with fluid intelligence, the capacity to apply learned skills in novel settings (Lake et al., 2017). Several approaches have been proposed to build systems capable of making analogies. Notably, the structure-mapping theory ex- plores methods for discovering structural correspondences between pre-existing object representations (Winston, 1982: Gentner, 1983; Carbonell, 1983; Falkenhainer et al., 1989; Holyoak & Thagard, 1989). However, these approaches often overlook the perceptual aspect, assuming object rep-
Multiple problems that necessitate combined perception and reasoning have been identified (Hofstadter, 1999). Among these tasks are Bongard Problems (BPs), introduced by Bon- gard (1968; 1970). Initial BPs were designed manually, leading to the formulation of a few hundred task instances by individual contributors (Foundalis, 2006b). A typical BP consists of two sides, left and right, each comprising six image panels arranged in a grid (Figs. 1a, 1c). All im- ages on one side illustrate a shared concept absent in the
Warsaw University of Technology, Warsaw, Poland 2 AGH Uni- versity of Krakow, Krakow, Poland. Correspondence to: Mikolaj Malkiński <mikolaj walliiskiidokkeee ptw..........................
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
images on the opposite side. The task is to identify the underlying rule that differentiates the sides and articulate it in natural language. Initial BPs (Bongard, 1968), akin to human IQ tests, featured abstract 2D geometric shapes, putting the focus on abstract reasoning. However, recent works have expanded the set of BPs to include real-world images, which broadens the scope of presented objects, at- tributes and relations. Specifically, the matrices in Bongard HOI (Jiang et al., 2022) depict human-object interactions, (Wu et al., 2024) employs open- world free-form concepts, increasing the diversity of fea- tured scenes.
pointed certain LLM limitations in solving AVR problems represented as text despite using information lossless trans- lation through direct-grid encoding. Recent works have combined the vision and language modalities into multi- modal large language models (MILMM)) (Achiam et al., 2023; Reid et al., 2024; Anthropic, 2024), inviting their ap- plication to diverse tasks (Yin et al., 2024; Wu et al., 2023). Motivated by these recent developments we examine the reasoning capabilities of MILMS in solving BPs.
while
Contribution.
The contribution of this paper is fourfold.
(I) We consider BPs in the context of MLLMs and propose a diverse set of strategies to solve BP instances in two setups: open-ended language generation and binary classification.
A central theme in BPs is recognition of concepts in a context-dependent manner, as object representations need to be formed specifically for the presented matrix, rather than deseribed priori (Linhares, 2000). For example, consider the matrix in Fig. 1a an analysis restricted to its left side may yield multiple concepts, such as the presence of curves or an object centered in the image. Only through a compre- hensive understanding of both matrix sides one can recog- nize that the left side depicts a single line, while the right side presents two lines. Such concept-basee tasks are argued to offer a more accurate evaluation of the model's generaliza- tion ability and its capacity for abstraction (Mitchell, 2021; Odouard & Mitchell, 2022). In contrast to other abstract reasoning problems, such as Raven's Progressive Matrices (RPMs) (Raven, 1936; Raven & Court, 1998; Malkiński & Mandziuk, 2025a) that have recently witnessed the de- velopment of large- scale benchmarks (Barrett et al., 2018: Zhang et al., 2019; Malkiński & Maádziuk, 2025c), BPs allow to assess model's ability to derive concepts from limited set of examples (typically six images per matrix side), which positions the task within few-shot learning setting (Fei-Fei et al., 2006; Wang et al., 2020). The above aspects make BPs a valuable testbed for assessing abstract reasoning abilities of AI models.
proprietary MLLMs and open MLLMs on both synthetic and real-world BPs, and identify their severe abstract reasoning limitations.
(2) We evaluate 4
(3) To further examine the observed MLLM limitations in solving both synthetic and real-world BPs, we introduce a focused dataset of BPs (Bongard- RWRR Figs. 1b, Id) that represent concepts from synthetic BPs using real world im- ages. Thanks to relying on the same abstract concepts as synthetic BPs, Bongard-RWF facilitates direct comparisons of the MLLM performance in both domains.
(4) We perform detailed comparative analysis of 8 MLLMs on Bongard-RWE vs. synthetic BPs, shedding light on the reasons of their generally poor performance.
2. Related Work
AVR Tasks. The AVR field encompasses a broad set of problems aimed at studying various aspects of visual cog- nition (Gardner & Richards, 2006; Malkińísk & Mańdziuk, 2023). Recent DL research in this domain gravitated to- wards utilizing certain well-established datasets, e.g. with visual analogics (Hill et al., 2019; Webb et al., 2020) or RPMs (Zhang et al., 2019; Barrett et al., 2018), to mea- sure the progress of DL models (Matkiński & Mańdziuk, 2024a;b; 2025b). However, such benchmarks evaluate sys- tem performance in learning a particular task, rather than assessing its general ability to acquire new AVR skills. To address this limitation, certain tasks have adopted few-shot learning setups, requiring models to learn from a few demon- strations, as exemplified by SVRT (Fleuret et al., 201 1) or Bongard-L LOGO (Nie et al., 2020). Nonetheless, these benchmarks follow discriminative setting where a set of possible answers is provided. Conversely, other datasets such as ARC (Chollet, 2019) or PQA (Qi et al., 2021) pose generative challenge, which may be considered more diffi- cult due to its open-ended nature. In addition to synthetic tasks fcaturing 2D geometric shapes, certain datasets present analogous reasoning tasks using real-world images (Teney
Motivation. The quest to build systems capable of form- ing abstract concepts dates back the 1950s (McCarthy et al., 2006). The advent of Deep Learning (DL.) opened new possibilities to tackle BPs (Kharagorgiev, 2018; Nie et al., 2020). However, despite significant advancements, methods for consistently solving BPs (and other problems that involve abstract reasoning) are still lacking (Mitchell, 2021; van der Maas et al., 2021: Stabinger et al., 2021), Typically, DL approaches omnt the generation of natural lan- guage answers by casting BP into binary classification task, in which test image had to be assigned to the matching side of the matrix. Conversely, a parallel stream of research on large language models (LLMs) demonstrated promising results in open-ended language generation (Brown et al., 2020). In particular, LLMS were applied to selected AVR tasks (Webb et al., 2023), though, lately Xu et al. (2024)
2

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
al., 2020; Ichien et al., 2021; Bitton al., 2023). This ap- proach extends the range of concepts that can be expressed and, above all, allows employing models pre-trained on large image datasets. In this work, we concentrate on sev- eral BP datasets that present a few-shot learning challenge, cover both synthetic and real-world images, and consider settings involving both binary classification and answer gen- eration in natural language.
resentation taken in these works simplifies the perception task by presenting the model with pre-existing higher level representations. Only recently, thanks to the appearance of MLLMs, vision and text started to be treated jointly in a unified manner. Perception Test (Patraucean et al., 2023) as- sessed multimodal video understanding across various skills and types of reasoning. Cao et al. (2024) proposed a suite of AVR tasks to compare MLLM and human performance. Jiang et al. (2024) assessed AVR skills of MILMs on an introduced multidimensional benchmark combining AVR and perceptual questions. Ki (Yiu et al., 2025) evaluated how models generalize visual transformations to analogical reasoning tasks inspired by the developmental psychology. Most relevant to our paper is the concurrent work of Wüst et al. (2024), which investigated the capabilities of MLLMs in solving synthetic BPs, using both open-ended text genera- tion and multi-clas, concept classification setups. Our work complements this stream of research by exploring BPs and providing comparative insights into MLLM analogy-making performance in both synthetic and real-world domains.
Approaches to solve BPs. Initial approaches to tackle BPs involved cognitive architectures (Foundalis, 2006a), program synthesis coupled with inductive logic program- ming (Saito & Nakano, 1996; Sonwane et al., 2021), and the application of Bayesian inference within visual language framework (Depeweg et al., 2018; 2024). (Kharagorgiev, 2018) trained convolutional network on a generated syn- thetic dataset with geometric shapes and applied a one-level decision tree to solve BPs framed as a binary classification task. Nie et al. (2020) introduced Bongard-LOGG with syn- thetically generated BPs and used it to evaluate -Nsed models focused on meta-learning (Snell et al., 2017; Mishra et al., 2018; Lee et al., 2019; Raghu al., 2020; Chen et al., 2021) and relational reasoning (Barrett et al., 20.8). Jiang et al. (2022) applied the Relation Network (Santoro et al., 2017) to objects detected with Faster R-CNN (Ren et al., 2015) and employed the model to solve matrices from the real world Bongard HOI dataset. Despite high diversity of approaches, none of them has fully addressed the abstract and open-ended nature of BPs. Most related to our work, Wu et al. (2024) considered hybrid approaches that caption each image panel with an image-to- text model and applied LLMs for processing these text descriptions. Differently, in this work we focus on MLLMs that are inherently capable of jointly processing images and text.
3. Solving BPs with MLLMS
In this paper, we propose set of novel strategies for solving BPs using MLLMs. Definition of each strategy includes the input on which the model operates and the sequence of reasoning steps performed by the model. A high-level overview of these methods is provided in Fig. 2. In the main tested setting, we follow the initial BP formulation that requires providing an answer in natural language, and propose model approach to automatically evaluate such model predictions. In addition, we consider simpler formulations of the problem, casting it into a binary classifi- cation framework that enables detailed evaluation of AVR abilities of the tested MLLMss An illustration of these eval- uation settings is presented in Fig. 3. In what follows, let Bpx = {C>, R "y² denote a BP instance (X e A is an index), composed of L { ..... L } left and R* =(RA ..... R, } right panels, resp., and its concept yx expressed in natural language.
Abstract reasoning of MLLMs. While the abstract rea- soning capabilities of MILMs have only recently begun to be explored, they have been studied more extensively in several related tasks. Initial works focused on LLMs and evaluating their abstract reasoning performance in simplified analogy tasks. Webb et al. (2023) showed that GPT-3 and GPT-4 (text-only variants) performed on the human level, or even outcompeted humans, in certain RPM-like tasks in a zero-shot manner without additional fine-tuning. However, they represented the image objects as text using a fixed small vocabulary, thus omitting the need for identifying concepts from
3.1. Answer Generation in Natural Language
We start by defining the strategies for generating answers in natural language. In each strategy, the model receives a general description of Bongard Problem with two BP examples with correct answers. Additionally, besides this generic introductory information, a given task XPX to be solved is presented in a strategy-speciffi way. Appendix 1.4 presents the exact prompt formulations.
shapes, a key challenge of BPs. Recent re- search concerning the evaluation of abstract reasoning skills of LLMs concentrates around the Abstraction and Reason- ing Corpus (ARC) task (Chollet, 2019). It was demonstrated that LLMs can solve certain ARC problems transformed to the text domain (Moskvichev et al., 2023; Mirchandani et al., 2023; Camposampiero et al., 2023; Xu et al., 2024). Despite these important stepping stones, the text-based rep-
Direct (Fig. 2a). The model receives image presenting BPX and is asked to directly formulate an answer (i.e., de- scribe the difference between C and R panels in natural language).
3

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Shared context per side
MLLM
MLLM
MLLM
MILMM
j
MILM
MILMM
y
UR
(a) Direct
(b) Descriptive(-direed
MLLM
MLLM
Shared context
R
MILM
(c) Descriptive-iterative
MILM
Ø
D
MLLM
MLLM
MLLM
b
MILM
1
(d) Contrastive(-direc)
(c) Contrastive-iterativ
Figure 2: Generation strategies. Direct (a) feeds the image of the whole matrix to the model. Descriptive (b), Contrastive (d), and their iterative variants, (c) and (e), present individual image panels to the model in a fixed order. Their direct variants, (b) and (d), additionally include the image of the whole matrix. Grey background marks a sequence of requests run in single context window.
Descriptive (Fig. 2b). Defines a more granular approach in which the model is first requested to generate textual description of each image panel of the matrix. Each de- scription is generated in a separate context, such that the model doesn't 't have access to the prior panels nor to their descriptions. Next, the model is requested to provide an answer to the problem based only the generated textual descriptions of all image panels. In this strategy and in all subsequent ones, we evaluate only the final answer provided by the model, without considering the intermediate outputs.
account the images from the other side. Conversely, in the Contrastive strategy, the model is tasked with describing the difference between a pair of corresponding images from both sides of the problem (L2*, ) , ).... (L R,Y) Af- ter describing the differences between all six image pairs
in separate contexts, the model generates its final answer based on these textual descriptions.
Contrastive-iteratiee (Fig. 2e). Extends Contrastive by performing all reasoning steps in a single context window, enabling the model to gradually improve its understanding of the rule separating both sides.
Descriptive-iteratiee (Fig. 2c). Evaluates the role of the reasoning context and utilizes context window comprising the dialog history concerning all images in the given side of the problem. After generating the description of the first image, the model iteratively refines its output based on subsequent images from the same side. Based on the textual descriptions of both sides of the problem, the model is requested to provide the final answer.
-ontrastive-direet (Fig. 2d with dashed element). Ex- tends Contrastive by adding the image of the whole matrix to textual descriptions of differences within each panel pair.
3.2. Evaluation of Natural Language Solutions
The correct answer to BP may be formulated in natural language in many different ways. To account for this inher- ent variability, we utilize a model-based approach to assess whether the generated answer ÿ matches the ground-truth y. In the proposed setting, an MLLM ensemble receives both ÿ and y and is requested to output a binary yes/no answer whether both descriptions refer to the same concept (see Fig. 3a). Specifically, we assess the correctness of ÿ using all four considered proprictary MLLMs and count it as cor- rect if at least two models agree with this class (see details in Appendix E). In contrast to solving the BP task, which requires abstract reasoning abilities, the task of determining whether two answers refer to the same concept boils down to assessing the semantic similarity between two texts, and
Descriptive-direct (Fig. 2b with a dashed element). In both above Descriptive strategies, the model is never presented with the image of the whole matrix Bpx Descriptive- direct strategy extends Descriptive by providing the image of BP along with the textual panel descriptions.
Contrastive (Fig. 2d). A critical aspect of BPs is the fo- cus on forming concepts within the specific context of the matrix x It's often the case that correct identification of the concept governing one side requires analysis of the other side to identify their key differences. In Descriptive strategies, the model provides image descriptions concern- ing a single problem side _ or R without taking into
4

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
y
K
E
0
MLLM
MLLM
X>X
O
MLLM
8
O
x
0/1
L/R
L/R
(a) Evaluate generated description
(b) Assess solution correctness
(c) Assign images sides
Figure 3: Evaluation settings. We consider the three settings to solve BPs: (a) the ground-truth answer y is paired with a description y generated by the MILM and the model needs to verify if they describe the same concepts; (b) given a possible solution y the model needs to assess whether it's correct; (c) two test images corresponding to the left (L) and right (R) sides, resp., are randomly shuffled, and the model needs assign each image from the pair the proper side of the problem.
MILMMs are known excel in such setup (Lu al., 2023).
Longard-OpenWoree datasets is not meaningful, these datasets depict different concepts. To enable meaningful comparison and additionally determine whether the MLLMs performance score is domain-related, we introduce Bon- gard Real-World Representations, a focused dataset that expresses concepts present in synthetic BPs using real- world images, thus creating their real-life equivalents, as illustrated in Fig. 1. Appendix F contains additional examples.
3.3. Binary Classification Formulations
To dive deeper into the AVR capabilities of the studied models, cast the BP task into three binary classification settings, reducing the task's difficulty.
Firstly, we provide the model with the image of the whole matrix BP along with a possible solution, and the model is prompted to generate a binary score assessing the cor- rectness of the provided answer (Fig. 3b). Two settings are considered, in which the solution is formed by either the actual ground-truth answer (the expected answer is yes), by an incorrect answer taken from a different matrix BP where X' = (X + 20) mod 100 (the expected answer is no). We manually reviewed the resulting pairings to confirm that the selected negative concept was distinct from the origi- concept. We refer to these two classification setups as Ground-Truth and Incorrect Label, resp.
Dataset generation. For given instance BPx we first use GPT-4o to describe its underlying concept yx in N = 10 different ways using the prompt listed in Prompt 4 (Ap- pendix 1). We obtain N real-world textual descriptions D.r = ( D = .,..., - 1, of each side S R). Then, we use image search engine Pexels API (Pexels, 2024) to download M 15 images per each described side D.K.. We employ GPT-4o (see Prompt 5 in Appendix I) to select only those images that prop- erly illustrate the concept of the respective side and are indeed distinguishable from the alternative concept. We stop the selection procedure after having a set of T' = 3 descriptions {Dx D DX each with 2 appropriate im- ages: (1), 1, (2), & = in iz 2g per each side S (6 left and 6 right ones).
Secondly, we follow setting exemplified in the Bongard- LOGO dataset (Nie et al., 2020) in which two test im- ages have to be classified to different sides of the problem (Fig, 3c). To this end, we take two test images correspond- ing to the respective sides of the matrix, randomly shuffle the images, and request the model to determine the side to which each image belongs. In synthetic BPs we create the test set by removing the 6th image from each side of the ma- World
To decrease the possibility of generating a problem with trivial answer, which is more probable if the images come from common textual description D.X the corresponding real-world problem instance RWR x = {(x RR*} is con- structed as follows (see Algorithm in Appendix F): S (I (1), X (1), XXS (1), I (2), X (2), I (2)). SE (L, R).
trix, while in BPs from Bongard HOI, Bongard- WpenWorld and Bongard-Re RWR we use the additional test images. We refer to this setup as Images to Sides.
Prompts for all three binary classification formulations are listed in Appendix I.3.
We run Algorithm for the first 100 synthetic BPs. After applying the exclusion criteria, this led to the generation of 50 instances R.RRX However, as we noticed through manual inspection, some of them were not well depicting the respective problem concept. Hence, we modified the dataset in the following way: 14 problems were entirely removed and, out of the remaining 36, 24 were adjusted through
4. Bongard-RWR
One of the interesting research avenues is to compare the MLLMs performance on synthetic BPs vs. real-world ones. Note, however, that a direct performance comparison on synthetic Bongard dataset vs. real-world Bongard HOI and
5

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Ground-Truth
Tncorrect Tabel
Groand-Truth
Incorrect Tabel
Ground-Truth
Incorrect Label
Ground-Trith
Incorrect Label
4e
79
GF 44
200
SPE4e
100
2913
24
GFT4e
59
06
Duolo
74
Taree
200
GP14 Sudbo
100
143%
0254 Tulo
60
78
Cozaia 1.5 hts
70
80
Gesazi 2.5 Pro
36
29
Cenaau 3.5 Pro
58
(6..0)
26
Genttn Pm
52
Clardo 3.5 Sonniee
Clvvito 3 Srreet
100
Claods Sonret
10
Claods
bnarrtt22 on
Internit:2 88
InterV122 UB
IntereVi2
L 24
L 78
72
LL 18
22
21
100
Images to Sides
Images to Sides
Images to Sides
Images to Sides
GPT44
CPT-40
99
4F
94
GPT:40
15 (25.0%)
GPT-4 Turbo
56
GPT-4 Turbo
GPT-4 Turbo
GPT4 Turbo
17 (28.3%)
Gemini 1.5 Pro
69
Gemini 1.5 Pro
Gemini 1.5 Pro
96
Gemini 1.5 Pro
19 (31.7%) (%)
Cloude 3.5 Sonnet
75
Claude 3.5 Sonnet
99
Claude 3.5 Sonnet
96
Claude 3.5 Sonnel
24 (40.0%)
InterVLL2
32
13
19
IntermVL2 16 (26.7%)
LL4VA-1.6
33
0
2
0 (0.0%
9
1 (1.7%)
Pixtral
Poxtral
Pixtral
87
Pistral
26 (43.3%)
25"
50
75
100
25
30
75
100
25
50
75
100
15
30 (50.0%)
(a) Synthetic BPs
(b) Bongard HOI
(c) Bongard- Kongard-OOenWWort
(d) Bongard-RWR
Figure 4: Binary classification. The top row presents the results for the Ground- Srouhd and Incorrect Label strategies, while the bottom row refers to the Images to Sides strategy. Results of a random baseline are marked with dashed line. For Bongard-RWR. we also report the percentage of correctly solved problems in parentheses.
a manual selection of the images that well represent the considered concept. Furthermore, we extended the dataset by adding 17 problems with manually translated concepts (i.e., with no use of GPT-4o), for which images were also selected manually, and constructed by hand, i.e., by means of making photos of manually-buile scenes reflecting the respective concepts.
sification setups and 7 proposed generation methods. Our evaluation spans a range of MILMs, including 4 proprietary models accessible via API: GPT-4o, GPT-4 Turbo (Achiam et al., 2023), Gemini 1.5 Pro (Reid et al., 2024), and Claude 3.5 Sonnet (Anthropic, 2024), alongside 4 open-access mod- els run locally on an NVIDIA DGX A100 node: Intern 8B (Chen et al., 2024c:b). LLaVA-1.6 Mistral-7B (Liu et al., 2023; 2024: Jiang et al., 2023), Phi-3.5- 5ision (Abdin et al., 2024), and Pixtral 12B (MistralAI, 2024). We consider four BP datasets covering both synthetic and real-world images. To balance diversity and computational feasibility, we cap the dataset size at 100 instances. Specifically, we use the first 100 manually constructed (synthetic) BPs from (Bon- gard, 1970). 100 problem samples from each of the Bongard HOI and Bongard-Open World datasets selected from their test splits through stratified sampling based on problem concept labels, and all 60 instances from Bongard-Ryy -WR.. Extended results are presented in Appendix C.
All in all, out of 60 real-world problems obtained (Bongard- RWR), 12 were generated fully automatically, 24 were gen- erated automatically and then adjusted manually (manual selection of the images), and 24 were constructed entirely manually, Appendix F presents the details.
Dataset variants. Bongard-RWR was designed to capture real-world complexity, including variations in aspect ratio, resolution, and layout. To complement the base dataset, we developed three additional variants: RWR-S (square im- ages), RWR-G (greyscale images), and RWR-SG (square greyscale images). In the square image variants, we manu- ally cropped all images to square format and rescaled them to 512 512 resolution. The BP matrix was resized to 1024px width while maintaining aspect ratio, resulting in height of 763px.
Binary classification. Fig. presents the results of binary classification tasks. In the Ground-Trui setting, most pro- prietary and some open-access models outperform random classifier baseline. In the Incorrect Label setting, since re- jecting incorrect concepts is a generally easier task, most models perform better than in the Ground- Truthh setup. How- ever, the consistently shifted performance of open-access models suggests potential bias toward agreeing or disagree- ing with provided concepts. In the Images to Sides task, proprietary models demonstrate strong performance, while
5. Experiments
To evaluate the AVR capabilities of MLLMs, we conduct experiments two main settings, involving binary clas-
6

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Table I: Language generation. The number of correct answers to 100 synthetic BPs, 100 selected BPs from each of Bongard HOI and Songard-OpenWeee World, and all 60 BPs from Bongard- RWR. Three main strategies: Direct, Descriptive, and Contrastive, denoted as Di, De, and Co, resp., are considered. The best result for a given strategy is marked in bold and the second best underlined. For Bongard- -Wir, we also report the percentage of correctly solved problems in parentheses. For comparison, the average human performance on Bongard-RWe is 39.2 correct answers (65%).
SYNTHETIC
HOI
OPENWORLD
BONGARD RONGARD-RWR
DI
DE
Co
Di
DE
CO
DI
DE
Co
DI
DE
Co
GPT-40 GPT-4 TURBO
17
17
10
35
42
18
40
46
19
5
(8.3%)
8 (13.3%)
2 (3.3%)
6
15
8
22
45
5
21
57
12
1
(1.7%)
5
{ 8..3%)
0 (0.0%)
GEMINI 1.5 PRO
21
17
23
40
15
13
32
11
3 (5.0%)
7 (11.7%)
1 (1.7%)
CLAUDE 3.5 SONNET
19
5
44
13
53
21
1
(1.7%)
13 (21.7%)
2 (3.3%)
INTERNVL.2-8B
0
12
2
2
11
18
7
0 (0.0%)
0 (0.0%)
0 (0.0%)
LLAVA-1.6 MISTRAL-7B
0
1
0
5
4
1
12
16
1
0 (0.0%)
0 [ (000%)
0 (0.0%)
PHI-3.5-VISION
0
2
0
1
4
2
7
12
0 (0.0%)
0 (0.0%)
0 (0.0%)
PIXTRAL 12B
4
28
27
7
33
34
14
1 (1.7%)
1 (17%)
(0.0%)
Pixtral stands out among open-access models. Neverthe- less, binary classification tasks do not fully reveal whether the solver truly grasped the presented concept or simply relied on surface-level similarities, leaving the space for other challenging evaluation setups.
access models. We hypothesize that this improvement is due to the Descriptive setting being more aligned with the model's training, where it primarily learns to caption indi- vidual images. However, this strategy doesn't 'tt leverage the additional information present in the joint BP image, and certain context-dependent visual features may be missed in captioning. We believe that with further advancements in reasoning over multi-part compositional images, mod- els in the Direct setting should eventually outperform the Descriptive strategy.
Generative capabilities in the Direct setting, As pre- sented in Table 1, model performance using the Direct gen- eration strategy is generally weak on synthetic BPs, with the best model, GPT-4o, solving only 17 out of 100 prob- lems. This indicates that the models struggle identify abstract, synthetic concepts and express them in natural language. The challenge is even more apparent on Bongard- RWR, where the best model, GPT-4o, solves only 5 out of 60 problems. Nevertheless, performance improves on Bongard HOI and Songard-OpenWorne World, with best results of 35/100 and 40/100, resp. Notably, while GPT-4o achieves the highest scores on these two datasets, Pixtral 12B ranks second (28/100 1000 and 33/10 100, resp.), showing that smaller open-access models can still be competitive in this setting. While the better performance on Bongard HOI and Bongard- Open World may be attributed to a higher ratio of real-world images in the training data, the weak results on Bongard- RWR suggest that the discrepancy more related to the specific underlying concepts than the visual domain as such (see Fig. 14 in Appendix F for the details).
Contrastive reasoning. Correct identification of concepts in BPs requires joint processing of the images from both sides of the problem. The Contrastive strategy evaluates the model ability to extract underlying differentiating concepts within such image pairs. Across all datasets, the models evaluated under the Contrastive strategy perform worse than with the Descriptive strategy (cf. Table 1). This points to the fundamental difference between human and machine ap- proaches to solving AVR tasks. Humans often rely on direct comparisons between image panels from different categories to highlight differences (Nüssli et al., 2009), whereas the tested methods perform better when making comparisons on text-based image descriptions, potentially disregarding critical visual details missed during image captioning. This discrepancy indicates the need for further modeling improve- ments to fully leverage the Contrastive strategy.
Independent image description. In the next experiment we analyse whether an iterative reasoning approach, in which the model first generates separate captions for each image and then combines them into a final answer, can improve performance. As shown in Table 1, compared to the Direct strategy, the Descriptive one improves the best results across all datasets: from 17 to 21 on synthetic BPs, from 35 to 45 on Bongard HOI, from 40 to 57 on Bongard-Open and from 5 to 13 on Bongard-RWR. A clear improvement can be observed individually for each proprictary model. The gain is less pronounced (if at all) for individual open-
Iterative reasoning. Next, we tested whether preserving responses from past turns in the dialog context could im- prove concept identification in both Descriptive and Con- trastive settings. As shown in Fig. 5, Descriptive-iteratie strategy visibly worsens the results compared its non- iterative counterpart across all datasets and models, except for negligible improvement of Intern VL.2-B on synthetic BPs and several cases of a complete failure (accuracy of 0) of both strategies. In contrast, Contrastive-iteraiie brings no improvement over Contrastive in only cases, 2 of syn-

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Descriptive
Contrastive
30
-direct
leralvvve
-direct
tterative
Syothetic HOI OpenWorld RWR
Synthetic HOl OpenWorld
Symthetic HOI OpenWorid
Synthetic HOI OpenWorid
10:
+5
+2
20
+16
+15
16
+12
+13
10
+9
+10
-15
+1
-30
-40
-5
Figure 5: The impact of -direct and -iterative variants. Bars in each group correspond to models in the following order: GPT-4o, GPT-4 Turbo, Gemini 1.5 Pro, Claude 3.5 Sonnet, InternVL2-8B,, LLaVA-1.6 Mistral-7B, Phi-3.5-Vision, and Pixtral 12B.
thetic BPs, and 3 of Opennddrd.. Despite these improvements, Contrastive-iterativ is generally inferior to Descriptive (see Table 4, Appendix C). Overall, the study indicates that contemporary models struggle to effectively utilize additional information from the context window.
improve multi-image reasoning capabilities of MLLMs for tasks that require reasoning across multiple images. Figs. 16 23 in Appendix G further show that altogether the consid- ered approaches solved 54, 89, 93, and 23 problems from synthetic BPs, Bongard HOI, Bongard-RWe RWWR, resp. This raises the question of whether an ensemble combining all proposed strategies could fur- ther enhance model reasoning performance. We leave the exploration of this emerging direction for future work.
World, and
Multimodal answer generation. In the final experiment, assessed whether incorporating image of the entire matrix at the answer generation step would improve the performance of the Descriptive and Contrastive strategies. As shown in Fig. 5, Descriptive-direct shows performance improvements over Descriptive in 12 out of 32 (dataset, model) cases. Contrastive-direet improves upon Contrastive in all (dataset, proprietary model) configurations, and ad- ditionally improves in certain (dataset, open-access model) settings. However, despite these gains, Contrastive-dires overall performs worse than Descriptive, except for GPT--4o and Intern VL2-8B on synthetic BPs, and Intern VL2-8B on Bongard HOI (see Table 4, Appendix C). This suggests that contemporary models are to some extent capable of utilizing additional visual inputs improve reasoning performance, in particular the newest GPT-4o, which displays improve- ment from incorporating the -direct extension in 7 out of 8 cases. Nevertheless, further work is needed to improve consistency across all models.
Proprietary vs. open-access models. Proprictary models generally outperform open-access ones, leading in 35 out of 40 (dataset, strategy) pairs (see Appendix C, Table 4). The black-box nature of proprietary models makes it challeng- ing to attribute their advantage specific aspects, whether it be the number of parameters, the size and composition of training data, or the pre- and post-processing methods. However, the recently released Pixtral 12B model performs competitively in multiple settings, occasionally surpassing proprictary models. This highlights the viability of devel- oping competitive MLLMs without sacrificing accessibility. At the same time, clear performance drop of Pixtral 12B synthetic BPs and Bongard- RWR suggests its intrinsic weakness in reasoning about abstract concepts, whether re- flected in synthetic or real-world manner, similarly to other open models.
Comparison of prompting strategies. Across all mod- els, the Descriptive strategy achieves the highest scores on Bongard-RWF and Bongard-Open World. In Bongard HOI, it ties with Descriptive-direct, while in synthetie BPs, it ranks just behind its -direct extension. As shown in Appendix G, altogether Descriptive strategies solve the same number of synthetic BPs as Contrastive strategies (44; Fig. 16), but lead in Bongard HOI (82 vs. 63; Fig. 18), (90 vs. 76; Fig. 20), and Bongard- RWR (20 vs. 11: Fig. 22). This overall advantage of De- scriptive over Contrastive strategies indicates that current MLLMs perform better with prompting strategies focused on processing single images. This also highlights the need to
Dataset variants. We evaluated the four open-access mod- els on the three binary classification setups presented in Section 3.3 using the additional Bongard-RWR variants. In the Ground- and Incorrect Label settings, all mod- els performed similarly in the original Bongard-RWR. and its three new variants. However, in the Images to Sides setting, model performance differed. On RWR-G, most models solved more problems compared to Bongard-RWR, e.g., Phi-3.5- Visiin (+1)), suggesting that removing color helped models focus on structural concepts. The difference was also present for RWR-S, S, e.g., for Intern VL2-8B 8B (+13), Pixtral 12B (+9), and Phi-3.5- --sion (+7), likely due to
8

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
cropping, which increased the signal-to-noise ratio by elimi- nating empty space between panels. RWR-SG followed similar pattern. Overall, these additional dataset variants not only make the task more approachable but also help measure robustness to matrix layout and sensitivity to image color. Extended results are provided in Appendix C.
image representations-aacieeee 64%. This suggests that MLLMs, which uniformly process images and text, outper- form decoupled two-stage approaches, which handle image captioning and
reasoning with different models. They also briefly consider a natural language generation task, where models describe concepts presented in the BP instance (Wu et al., 2024, Appendix E). They again use a two-step approach comprising fine-tuned BLIP-2 for image captioning and ChatGPT for concept generation. In contrast, we employ a single MLLM for both tasks. For evaluating free-form concept generation, they use automated text-based metrics, which provide a general measure of text similarity. We, however, employ a voting MLLM ensemble, offering a more direct assessment of solution correctness.
Concept selection. Similarly to the evaluation setup pro- posed in (Wüst et al., 2024), we conducted a multi-class clas- sification experiment in which the MLLM selects the correct concept from list of k E {2, 4, 8, 16} choices. Negative candidates were sourced from subsequent matrices, ensur- ing they were semantically different from the correct option. Across all four datasets, models achieved the highest accu- racy on Bongard HOT and Bongard- Openn World (exceeding 90% in most cases), while synthetic BPs and Bongard-RWi posed a greater challenge (e.g., for InternVL2-8B, 16% and 25% resp., for k- 16). These results suggest that real-world concepts, such as "Swing tennis racket vs. Not swing tennis racket", are easier for the models to recognize, likely because they can rely on surface-level object or action recognition to exclude incorrect options. In contrast, syn- thetic BPs and Bongard-RWR demand recognizing abstract concepts, such "Convex vs. Nonconvex", which require deeper understanding of the images. The gap between Bongard-RWi and synthetic BPs further highlights the ad- ditional challenge posed by representing abstract concepts in real-world images. The detailed results are provided in Appendix C.
Human results. We conducted a study on Bongard-RWR with 30 participants, as detailed in Appendix B. Participants were asked to describe BP concepts in an open-ended man- ner. Their responses were evaluated by expert annotators, who compared them to ground-truth labels and classified as correct or incorrect. Humans solved from 23 to 59 problems, with average of 39.2, achieving 65% accuracy. Notably, the lowest number of problems solved by a human participant (23) exceeded the number of problems solved by all models in total (22, see Fig. 23), which signifies clear performance gap.
6. Conclusions
Model scaling. We investigated the effect of model scaling on abstract reasoning capabilities by evaluating range of open-access and proprietary models of varying sizes across all four datasets using both the Direct and Descriptive strate- gies. Our results reveal that while larger models gener- ally perform better, especially among open-access models, smaller variants occasionally outperform their larger coun- terparts. Nevertheless, even smaller proprictary models (c.g.. GPT-4o mini) consistently outperformed the largest open- access alternatives, suggesting that scaling alone may be insufficient to achieve strong abstract reasoning capabilities. This highlights the need for explicit, dedicated efforts to improve abstract reasoning skils of MILMS in future work. Appendix D provides an in-depth analysis of these findings.
This paper investigates the reasoning capabilities of propri- etary and open-access MILMMS using BPs as a case study, Despite rapid progress, MLLMs still exhibit significant rea- soning limitations. Across all proposed answer generation strategies, the best-performing model solved only 22 out of 100 synthetic BPs. On the other hand, model performance improved moderately with real-world concepts, as shown by the results on Bongard HOI and Bongard-Open World. To delve deeper into the performance discrepancies between synthetic and real-world domains, we introduced Bongard- RWR, a new BP dataset designed represent concepts from synthetic BPs via real-world images. Focused experiments with this dataset suggest that the models' weak performance on synthetic BPs is not domain-specific but rather indicative of broader limitations in their reasoning abilities. Specifi- cally, MLLMs struggle with recognizing abstract concepts, fail to benefit from human- like multi-image reasoning ap- proach, demonstrate limitations in utilizing context window effectively, and require further work to consistently inte- grate text and vision modalities at the answer generation step. On a positive note, experiments conducted in three binary classification settings show that some models achieve encouraging results, suggesting that current limitations may be overcome with future advancements.
Comparison with state-of-the-art. A direct comparison with the results from (Wu al., 2024) is challenging due to the different ranges of test problems used in each study. With this caveat, we concentrate on key high-level observa- tions from both works. Wu et al. (2024) primarily focus on a binary classification setting corresponding to the Images to Sides setup in our work. On Sengard-Opee-WWee
our best performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieved 96% accuracy, while their top method, SNAIL-a meta-learning approach leveraging pre-trained OpenCLIP

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Acknowledgements
Bitton, Y., Yosef, R., Strugo, E., Shahaf, D., Schwartz, R., and Stanovsky, G. VASR: Visual analogies of situation recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 241-249, 2023.
This research was carried out with the support of the Labo- ratory of Bioinformatics and Computational Genomies and the High Performance Computing Center of the Faculty of Mathematics and Information Science Warsaw University of Technology. Mikolaj Malkiński was funded by the Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme.
Bongard, M. M. The recognition problem. Technical report, Foreign Technology Div Wright-Patterson AFB Ohio, 1968.
Bongard, M. M. Pattern Recognition. Spartan Books, 1970.
Impact Statement
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J.. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901, 2020.
We believe that the limitations of MILMs discussed in this work may eventually be overcome with the development of improved models and solution strategies. While such ad- vancements could lead to breakthroughs in abstract reason- ing, they also present potential risks of misuse. For instance, a method that excels in solving BPs could be exploited for solving online cognitive assessment tests, such IQ tests. This may allow individuals unfairly claim having higher cognitive abilities than they actually possess, leading un- ethical advantages in job recruitment, academic admissions, or other competitive contexts related to IQ scores.
Camposampiero, G., Houmard, L., Estermann, B., Mathys, J., and Wattenhofer, R. Abstract visual reasoning enabled by language. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 2643-2647, 2023.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404. 14219, 2024.
Cao, X., Lai, B., Ye, W., Ma, Y., Heintz, J., Chen, J., Cao, J., and Rehg. M. What the visual cognition gap between humans and multimodal LLMs? AxKiv22406.1442 2024.
Carbonell, J. G. Learning by analogy: Formulating and generalizing plans from past experience. In Machine learning, pp. 137-161. Elsevier, 1983.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I. Aleman, F. L., Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., et al. GPT-4 technical report. arXiv:2303.08774, 2023.
Chalmers, D. J., French, R. M., and Hofstadter, D. R. High- level perception, representation, and analogy: A critique of artificial intelligence methodology. Journal of Exper- imental & Theoretical Artificial Intelligence, 4(3):185- 211, 1992.
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. https: www.anthropic.comm caad//e 3-model-carde 2024. Accessed: 2025-01-30.
Chen, Y., Liu, Z., Xu, H., Darrell, T., and Wang, X. Meta- baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9062-9071, 2021.
Bai, J., Bai, S.. Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. VI: A versatile vision-language model for understanding, localization, text reading, and beyond. 2308. /2966, 2023.
Chen, Z. Wang, W., Cao, Y., Liu. Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Ex- panding performance boundaries of open-source mul- timodal models with model, data, and test-time scaling. arXiv:2412.05271, 2024a.
Barrett, D... Hill, F., Santoro, A., Morcos, A., and Lillicrap,
T. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 511- 520. PMLR, 2018.
Biscione, V., Yin, D., Malhotra, G., Dujmovic, M., Mon- tero, M. L., Puebla, G., Adolfi, F., Heaton, R. F., Hum- mel, J. E., Evans, B. D., et al. MindSet: Vision. A tool- box for testing DNNs on key psychological experiments. arXiv:2404.080 05290, 2024.
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to GPT-4V? Closing the gap to commercial multi- modal models with open-source suites. Science China Information Sciences, (112)22001, 2024b..
10

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Chen, Z.,, Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L.,, et al. V1.: Scaling up vision foundation models and aligning for generic visual- visual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E.,, Vinyals, O., Rae, J. W., and Sifre, L. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030, 2022.
Hofstadter, D. R. Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought. Basic books, 1995.
Chollet, F. On the measure of intelligence. 1rXiv:1911.01547 2019.
Depeweg, S., Rothkopf, C. A., and Jäkel, F. Solving bongard problems with visual language and pragmatic reasoning. arXiv:1804.04452, 2018.
Hofstadter, D. R. Gödel, Escher, Bach: an eternal golden braid. Basic books, 1999.
Holyoak, K. J. and Thagard, P. Analogical mapping by constraint satisfaction. Cognitive Science, 13(3):295-3355 1989.
Depeweg, S., Rothkopf, C. A., and Jäkel, F. Solving bon- gard problems with visual language and pragmatic con- straints. Cognitive Science, 48(5):e (e)5)e334332 13432, 2024.
Ichien, N., Liu, Q. Fu, S., Holyoak, K. J., Yuille, A., and Lu, H. Visual analogy: Deep learning versus compositional models. arXiv:2/05.0706, 2021.
Falkenhainer, B., Forbus, K. D., and Gentner, D. The structure mapping engine: Algorithm and examples. Ar- lificial Intelligence, 11(1):1-63, 1989.
Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7B (2023). arXiv:2310.06825 2023.
Fei-Fei, L., Fergus, R., and Perona, P. One-shot learning of object categories. IEEE Transactions on Puttern Analysis and Machine Intelligence, 28(4):594-6 61119 2006.
Fleuret, F., Li, T., Dubout, C., Wampler, E. K., Yantis, S., and Geman, D. Comparing machines and humans on visual categorization test. Proceedings of the National Academy of Sciences, 108(43):17621-17633 2011.
Jiang, H., Ma, X., Nie. W., Yu, Z., Zhu, Y., and Anandku- mar, A. Bongard- -ongard-HOI: Benchmarking few-shot visual reasoning for human-object interactions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19056-19065, 2022.
Foundalis, H. E. Phaeaco: cognitive architecture in- spired by Bongard's problems. Phl) dissertation, Indiana University, 2006a.
Jiang, Y., Sun, K., Sourati, Z., Ahrabian, K., Ma, K., Ilievski, F., Pujara, J., al. MARVEL: Multidimensional abstrac- tion and reasoning through visual evaluation and learning. In Advances in Neural Information Processing Systems, volume 37, pp. 46567-46592, 2024.
Foundalis, H. E. Index of bongard problems. http: wttpw.foundalls...o recee/re//tte wwwp/ees//bbeeeeeeee httppppppppphh.m..mp.ppp 2006b. Accessed: 2025-01-30.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv:2001.08366 2020.
Galatzer-Levy, I. R., Munday, D., McGiffin, J., Liu, X., Kar- mon, D., Labzovsky, I., Moroshko, R., Zait, A., and McDuff, D. The cognitive capabilities of generative AI: A comparative analysis with human benchmarks. arXiv:2410.07391, 2024.
Kharagorgiev, S. Solving bongard problems with deep learn- BowaanigoBoggrrr-rbbbbb... -iiee---ee--t learning/ February 2018. Accessed: 2025-01-30.
Gardner, M. and Richards, D. The colossal book of short puzzles and problems. Norton, 2006.
Gentner, D. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155-170) 1983.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
Hill, F., Santoro, A., Barrett, D., Morcos, A., and Lillicrap, T. Learning to make analogies by contrasting abstract rela- tional structure. In International Conference on Learning Representations, 2019.
Lee, K., Maji, S., Ravichandran, A., and Soatto, S. Meta- learning with differentiable convex optimization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 10657-10665, 2019.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E.,
Linhares, A. A glimpse at the metaphysics of bongard prob- lems. Artificial Intelligence, 121(1-2):251-271 2000.

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 34892-34916, 2023.
MistralAI. Announcing Pixtral 12B. https://
mistra a.a.iaaaii........ wtw.aaa.//////aeeeeee/ee http////wwww.atl/.p..... 2024. Ac-
cessed: 2025-01-30.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024.
Mitchell, M. Abstraction and amalogy-making in artificial intelligence, Annals of the New York Academy of Sciences, 1505(1):79-01, 2021.
Mitchell, M., Palmarini, A. B., and Moskvichev, A. K. Com- paring humans, GPT-4, and GPT-4v on abstraction and reasoning tasks. In AAAI 2024 Workshop on "Are Large Language Models Simply Causal Parrots?", 2023.
Lu, Y., Yang, X., Li, X., Wang, X. E., and Wang, W. Y. LLMScore: unveiling the power of large language models in text-to-image synthesis evaluation. In Advances in Neural Information Processing Systems, volume 36, pp- 23075-23093, 2023.
Moskvichev, A. K., Odouard, V. V., and Mitchell, M. The conceptARC benchmark: Evaluating understanding and generalization in the ARC domain. Transactions on Ma- chine Learning Research, 2023. ISSN 2835-8856.
Malkiński, M. and Mańdziuk, A review of emerging re- search directions in abstract visual reasoning. Information Fusion, 91:713-736, 2023.
Nie, W., Yu, Z., Mao, L., Patel, A. B., Zhu, Y., and Anandku- mar, A. Bongard- LOGo: A new benchmark for human- level concept learning and reasoning. In Advances in Neural Information Processing Systems, volume 33, pp. 16468-16480, 2020.
Matkiáski, M. and Maúdziuk, J. Multi-label contrastive learning for abstract visual reasoning. IEEE Transactions on Neural Networks and Learning Systems, 35(2):1941- 1953, 2024a.
Matkiáski, M. and Mañdziuk, J. One self- seff-configurable model to solve many abstract visual reasoning problems. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 14297-14305, 2024b.
Nüssli, M.-A., Jermann, P., Sangin, M., and Dillenbourg, P. Collaboration and abstract representations: towards predictive models based on raw speech and eye-tracking data. In CSCL '09: Proceedings of the 2009 conference on Computer support for collaborative learning. Interna- tional Society of the Learning Sciences, 2009.
Matkiński, M. and Maúdziuk, J. Deep learning methods for abstract visual reasoning: survey on raven's progressive matrices. ACM Computing Surveys, 57(7):1-36, 2025a.
Odouard, V. V. and Mitchell, M. Evaluating un- derstanding conceptual abstraction benchmarks. arXiv:2206.14111 2022.
Malkiáski, M. and Mańdziuk, J. Advancing generaliza- tion across a variety of abstract visual reasoning tasks. In Proceedings of the Thirty-Fourte Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, 2025b. (Accepted).
Patraucean, V., Smaira, L.,, Gupta, A.. Recasens, A., Mar- keeva, L.., Banarse, D., Koppula, S., Malinowski, M., Yang, Y., Doersch, C., et al. Perception Test: A diagnos- tic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, volume 36 pp. 42748 42761, 2023.
Malkiński, M. and Mandziuk, J. A-I-RAVEN and I-RAVEN- Mesh: Two new benchmarks for abstract visual reasoning. In Proceedings of the Thirty-Fre Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, 2025c. (Accepted).
Pexels. Pexels API. https:: http///wwppppwppp.e/op w.c.m//.ccccoo/.oooooo apii 2024. Accessed: 2025-01-30.
McCarthy, J., Minsky, M. I.., Rochester, N., and Shannon, C. E. A proposal for the dartmouth summer research project artificial intelligence, august 31, 1955. Al magazine, 27(4):12-12 2006.
Qi, Y., Zhang, K., Sain, A., and Song, Y.-Z. PQA: Percep- tual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12056-12064, 2021.
Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. In Proceedings of the 7th Conference on Robot Learning (CoRL), 2023.
Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations, 2020.
Raven, J. C. Mental tests used in genetic studies: The per- formance of related individuals on tests mainly educative and mainly reproductive. Master's 's thesis, University of London, 1936.
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018
12

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Raven, J. C. and Court, J. H. Raven's progressive matri- ces and vocabulary scales. Oxford pyschologists Press Oxford, England, 1998.
Webb, T., Dulberg, Z., Frankland, S., Petrov, A., O' O'Reilly, R., and Cohen, J. Learning representations that support extrapolation. In International Conference Machine Learning, Pp. 101166 PMLR, 2020.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac. J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser,, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. 22Xiv:2403.055300 2024.
Webb, T., Holyoak, K. J., and Lu, H. Emergent analogi- cal reasoning in large language models. Nature Human Behaviour, 7(9):1526-1541, 2023.
Wechsler, D. Scoring Manual. PsychCorp. 2008.
WAIS-IV Administration and
Ren, S., He, K., Girshick, R., and Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28, pp. 91-99, 2015.
URL
BE-DSWEACARJ.
Winston, P. H. Learning new principles from precedents and exercises. Artificial intelligence. 19(3):321-33) 1982.
Saito, K. and Nakano, R. A concept learning algorithm with adaptive search. In Machine intelligence 14: applied machine intelligence, pp. 347-363, 1996.
Wu, J., Gan, W., Chen, Z., Wan, S., and Philip, S. Y. Multi- modal large language models: A survey. In 2023 IEEE International Conference on Big Data, pp. 2247-2256. IEEE, 2023.
Santoro, A.. Raposo, D.. Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. A simple neu- ral network module for relational reasoning. In Advances in Neural Information Processing Systems, volume 30, pp. 4967-4976, 2017.
Wu, R., Ma, X.,, Zhang, Z., Wang. W., Li, Q., Zhu, S.-C., and Wang, Y. Hongard-Opent World: Few-shot reason- ing for free-form visual concepts in the real world. In International Conference on Learning Representations. 2024.
Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30. pp. 4080 4090, 2017.
Wüst, A., Tobiasch, T., Helff, L., lbs, I., Stammer, W., Dhami, D. S., Rothkopf, C. A., and Kersting, K. Bongard in wonderland: Visual puzzles that still make Al go mad? In NeurIPS 2024 Workshop on "System-2 Reasoning at Scale". 2024. Preprint available at: arXiv:2410.19544
Sonwane, A., Chitlangia, S., Dash, T. Vig, L., Shroff, G., and Srinivasan, A. Using program synthesis and inductive logic programming to solve bongard problems. arXiv:21 10.09947, 2021.
Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. LLMs and the abstraction and reasoning corpus: Suc- cesses, failures, and the importance of object-based repre- sentations. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Stabinger, S., Peer, D., Piater, J., and Rodríguez-Sānccher A. Evaluating the progress of deep learning for visual relational concepts. Journal of Vision, 21(11):8-88 8, 2021.
Teney, D., Wang, P., Cao, J., Liu, L., Shen, C., and van den Hengel, A. V-PROM: A benchmark for visual reasoning using visual progressive matrices. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 34, pp. 12071-12077 2020.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. National Science Review, 11(12), 2024.
Yiu, E., Qraitem, M., Majhi, A. N., Wong, C., Bai, Y., Gi- nosar, S., Gopnik, A., and Saenko, K. KiVA: Kid-inspired visual analogies for testing large multimodal models. In International Conference on Learning Representations, 2025.
van der Maas, H. L., Snoek, L., and Stevenson, C. E. How much intelligence is there in artificial intelligence? a 2020 update. Intelligence, 10110144, 2021.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, 7., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, Qwen2-VI. Enhancing vision-language model's s per- ception of the world at any resolution. arXiv:2409.1219. 2024.
Zerroug. A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. In Advances in Neural Information Processing Systems, volume 35, pp. 29776-29788, 2022.
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. RAVEN: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. Generalizing from a few examples: A survey on shot learning. ACM Computing Surveys, 53(3).1334, 2020.
13

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
A. Limitations and Future Work
Going beyond Bongard Problems. BPs fundamentally require solvers to articulate answers in natural language, making them a valuable testbed for assessing the reasoning capabilities of MLLMs. However, to comprehensively explore the challenges posed by the AVR domain, it is crucial to consider a broader range of problems.
For instance, VCog-Bench (Cao et al., 2024) is a benchmark designed to evaluate the reasoning capabilities of MLLMs across 3 datasets: 560 problem instances from RAVEN (Zhang et al., 2019), 309 from CVR (Zerroug et al., 2022), and 480 from MaRs- These datasets present multi-class classification tasks, offering between 4 to 8 options per problem. While the classification setting in VCog-Bend differs from our focus on natural language generation, both studies echo a shared conclusion - MILMs struggle in complex, multi-image reasoning tasks. We argue that generative problem formulations, such as those used in our study, pose a more substantial challenge than discriminative tasks, in which the solution may be induced from correlations or by making educated guesses. Further advances in abstract reasoning may require the development of new AVR benchmarks with generative evaluation settings.
A compelling example of a generative problem formulation is the Abstraction and Reasoning Corpus ( (ARC) (Chollet, 2019), in which each instance involves transforming a source grid into target grid based on an induced transformation rule. Each instance is accompanied by a few demonstrations to guide the solver. (Mitchell et al., 2023) explored multi- reasoning of GPT-4V on ConceptARC (Odouard & Mitchell, 2022), a variant of ARC categorizing tasks into distinct types. The study employed prompting strategies: presenting all demonstrations in single image, using separate images for each source and target grid pair, and separating each grid pair into distinct images. These settings are related to the Direct, Contrastive, and Descriptive strategies from our study, resp. The model performed best with the last approach, which aligns with the leading performance of the Descriptive strategy in our paper. Their study revealed that ARC tasks pose significant challenge for MLLMs, aligning with our results. Similar to our findings, GPT-4V evaluation on ConceptARC demonstrated that generative problem formulations pose a significant challenge for contemporary MLLMS.
Fine-grained analysis of MLLM perception. Related studies emphasize the importance of evaluating fine-grained aspects of model performance in visual reasoning tasks. Notably, (Biscione et al., 2024) propose the MindSet: Vision toolbox, which categorizes tasks into three main domains: low- and mid-level vision, visual illusions, and shape and object recognition. This benchmark is specifically designed to test models on 30 psychological findings inspired by human visual perception, providing a framework for understanding similarities and differences in human and machine vision. Preliminary evaluations using ResNet- 152 and GPT-4 on selected tasks revealed notable differences in perception between humans and machines. Applying MLLMs and the reasoning strategies proposed in our work to the MindSet: Vision toolbox opens a promising direction for future research, which could offer deeper insights into the perceptual capabilities of MLLMs.
Incorporating proposed strategies to enhance abstract reasoning abilities. (Galatzer-Levy et al., 2024) compared the cognitive abilities of MILMs to humans using the Wechsler Adult Intelligence Scale (WAIS-IV) (Wechsler, 2008). Their findings reveal that while MLLMs excel in tasks related to verbal comprehension and working memory, they significantly underperform in perceptual reasoning tasks. The evaluation setting used in this study involved presenting models with an image of an abstract reasoning matrix alongside text prompt describing the task, closely aligning with the Direct strategy employed in our work. However, as discussed in Section 5, the Direct strategy poses notable challenges for MILMs. Our experiments show that models consistently achieve better performance with alternative approaches, such as the Descriptive strategy. This highlights the importance of selecting appropriate strategies when evaluating MLLMs on abstract reasoning tasks. We believe that the diverse suite of strategies proposed in our work can be extended to other studies in abstract reasoning to fully capitalize on MLLM reasoning capabilities.
Cross-domain analysis of MLLM perception. A possible hypothesis for the subpar performance of MLLMS on AVR tasks involving synthetic datasets, such as VCog-Bent -ench or ARC, is the limited representation of synthetic images in their training data. This assumption is supported by the observed performance gap between synthetic BPs and real-world image BPs, such as those in Bongard HOI and -onnaard-Openoolfie which might suggest that MLLMs perform better at abstract reasoning with real-world images. However, our experiments with Bongard-RWR challenge this notion. Despite using real-world images, Bongard-RWi RWR demonstrates that MLLMs still struggle with abstract reasoning, indicating that the performance gap cannot be solely attributed to differences in data domains. Instead, this suggests more fundamental challenges in visual reasoning. Future work could extend this research line by leveraging datasets that include both synthetic and world images, such as Raven's s Progressive Matrices (Zhang et al., 2019; Teney et al., 2020) or Visual Analogy
14

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 1: Text used as a brief introduction in human testing.
The presented problems represent type logic puzzle. Each problem consists of sides separated by vertical line. Each side contains six images. The task is to find characteristic that applies to all the images on the left side but does not apply to those on the right side.
Some problems may be less obvious and require a broader perspective or focus on details. Simply comparing the general content of the images might not be enough. Answers may repeat.
Problems (Hill et al., 2019; Bitton et al., 2023). Contrasting MLLM performance on such dataset pairs may provide valuable insights into whether their limitations are rooted in data domain or in broader domain-free reasoning challenges.
B. Human Performance on Bongard-RWR
Foundation of the study. Our tests on MLLMs using the Bongard-RWR dataset revealed their poor performance in solving synthetic concepts depicted in real-world images. However, the difficulty and reliability of this new dataset remains an open question. To address this issue, we decided to assess human capabilities in solving these problems.
Methodology. We compiled all Bongard- RRW problems into single document, including brief introduction that explains what BPs are (see Prompt 1), along with a few detailed examples. The examples included one problem from the original BPs (# 133), one from Bongard-Oper World, and an additional BP (#336) manually translated to the real-world domain. Bongard-RWy RWR problem instances were positioned randomly in the document and were posed in an open open-ended manner, allowing participants to provide any response they deemed appropriate.
Participants in our human evaluation predominantly belonged the academic community, including Master students and (ocassionally) faculty members and PhD students, primarily due accessibility. This demographic selected based on the case of reaching and engaging with individuals who are readily available in academic settings.
All answers were collected using an online form, ensuring streamlined and efficient process for submission. Each participant was allowed to make only a single submission, to maintain the integrity and reliability of the data. In addition, the form contained a few more questions to gather basic statistics on our new dataset and the quality submissions:
1. How would you assess the readability of the images included in the problems? (Scale - 10)
2. How would you assess the difficulty of the tasks you received? (Scale - 10)
3. What your level of education? (Primary, Secondary, Higher, prefer not to say)
4. How much time did you spend solving the tasks? (Less than 30 minutes, From 30 minutes to an hour, From one to two hours, More than two hours)
Answers evaluation. In contrast to the evaluation of MLLM solutions, human responses were evaluated entirely manually. Initially, two humans reviewed the complete of answers independently, achieving 94.5% agreement on the correctness of the responses. The discrepancies were then discussed and consensus was reached leading to a single, unified evaluation.
Respondent overview. Overall, we successfully collected 30 responses, with 26.7% of participants having secondary education and 73.3% having higher education. Although the sample of respondents was small, we observed no significant discrepancies between the results of individuals with higher education and those with secondary education, both in the responses to additional questions (Fig. 6) and in the problem-solvine results (Fig. 8).
Findings from respondent responses. In Fig. 8, we present the distribution of responses to the Bongard-RWi dataset. Every problem was solved by at least one respondent, confirming the solvability of the dataset. The results are consistent across respondents (see Fig. 7), with the number of solved problems ranging from 23 59. Moreover, the findings
15

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
lliger Secodaay
20
Fiiaary
Secastoy
Highe
Image Reabbbility
Difliculty
lightr
Secoodary
Isage Radakilty
Takls DMfic
Figure 6: The respondent's answers to the questions attached in the form. Even though the majority of respondents had higher education, they, on average, spent more than one hour solving the problems. Moreover, only one person rated the problems as relatively easy (giving them a difficulty score of 4 out of 10).
demonstrate the superiority of humans over MILMs in tackling this type of task. Notably, the lowest human score exceeded the combined score of all the models. Half of the respondents solved more than 40 problems, with mean and median equal to 39.2 and 40.5, resp., resulting in 65% average accuracy.
The difficulty of each problem can be estimated based on the number of respondents who successfully solved As shown in Fig. 8, the problems exhibit varying levels of difficulty: 22 of them were solved by at least 25 respondents, while 10 were solved by fewer than 10 respondents. In addition, three problems-11 88 and 100-were solved by all respondents. Overall, the dataset was rated quite difficult, with an average difficulty score of 7.6 across all respondents.
Overall, the results demonstrate the robustness and applicability of Bongard- novel dataset for investigating the performance differences between human and model-based visual reasoning.
C. Extended Results
Table 4 presents results across all models, strategies and datasets discussed in Section the following paragraphs, we extend the discussion concerning binary and multi-class classification settings.
Binary classification (Ground-Truth) We assessed whether MILMs determine if given concept matches problem instance. On synthetic BPs, 3 proprietary (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) and 2 open-access (LLaVA-1.5 Mistral-7B, Pixtral 12B) models outperform a random classifier by notable margin. On Bongard HOI, 3 proprietary (GPT-40, GPT-4 Turbo, Gemini 1.5 Pro) and the same open-access models also surpass random guessing. Notably, Pixtral 12B attained a perfect score on this dataset. On Bongard- Nongard-OpenWWone World GPT-4o, Gemini 1.5 Pro, LLaVA-1.6 Mistral-7B, and Pixtral 12B achieve reasonable results. Again, the leading model Pixtral 12B with the outstanding 99% outcome. Model
16

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
60
Higher Secondary
50
Numbher of respondents
35
23
27
31
43
47
Humas results
Nutber
Figure 7: The number of problems solved by humans. The histogram illustrates the consistency across multiple respondents solving the Bongard-RWR problems. Notably, half of the respondents solved more than 40 problems, with none of them solving fewer than 23 ones. In the histogram, the lower bounds the bins are inclusive, and the upper bounds are exclusive, except for the last bin, which is [55, 59].
Solved by humans
4
9
10
Solved by any model
Not translated
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
42
43
44
45
47
48
49
50
N Number solved problems
0.0
2.5
5.0
7.5
10.0
12.5
53
54
55
58
59
60
61
62
63
64
65
66
68
69
70
22
71
72
73
74
75
76
77
78
79
80
Namber of
81
82
83
84
85
86
87
88
89
90
10
91
92
93
94
95
97
98
99
100
Figure 8: Human performance on Bongard-RWR. shown in the plot, the models struggled with many problems that humans found relatively easy solve. On the other hand, the models were able to solve problem 487 that appeared be relatively demanding for human solvers. In the histogram, the lower bounds of the bins are inclusive, and the upper bounds are exclusive.
17

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Table 2: Binary classification on Bongard- variants. The number of problems solved by open-access models in the
Images to Sides task from Bongard- RWW and its three variants, with the difference relative to Bongard-Ryy RWie in parentheses.
BONGARD-RWR
RWR-G
RWR-S
RWR-SG
IL 8B
16
17 1)
29
(+13)
27 (+11)
INTERNVL.2.5 38B
13
13 (4 0)
25 (412)
23 (+10)
INTERNVL.2.5 78B
15
17 17(+ 2)
24 9)
23 (+ 8)
QWEN2VI. 72B-INSTRUCT
20
25 (+ 5)
20
20
(4
LLAVA-1.6 1.6 MISTRAL 7B
0
3 3)
0 0]
0 0)
LLAVA-NEXT 110B
24
29 (+ 5)
29
29
PHI-3.5- -ISSION
1
16 (+15)
(( 7)
18 (+17)
PIXTRAL 12B
26
21 1
35
(+ 9)
33 (+
accuracy drops significantly on Bongard-RWR, where only LLaVA-1.6 Mistral-7B and Pixtral 12B outperform a random classifier. This suggests that correctly identifying concepts expressed in Bongard-RWK likely requires more advanced reasoning abilities, even the relatively simpler binary classification setting.
Binary classification (Incorrect Label). Rejecting possible solution is intuitively simpler than confirming its correctness, as it boils down to finding least one image that doesn't 't match the provided concept. Accordingly, 6 models perform better in the Incorrect Label setting than in Ground-Truth, with 7 perfect scores, 2 of them on Bongard-RWR, The exceptions are LLaVA-1. 1.6 Mistral-7B, and Pixtral 12B which are below a random guessing threshold for all four datasets, despite being above this threshold in Ground- Truth. This suggests that their strong performance in the Ground- Truth setting may be due to potential bias toward agreeing with the provided concept.
Binary classification (Images to Sides). We also evaluate the models' ability to correctly assign two test images to the appropriate sides of the problem. A problem is considered solved if both images are correctly assigned to the respective sides. Proprictary models perform well in this task across synthetic BPs, Bongard HOI and Bongard-Opent World. Conversely, among open-access models, only Pixtral 12B consistently achieves strong results. Notably, on Bongard-RWi Pixtral 12B solves 26/00 problems, outperforming all proprietary models, however, all models perform below the level of random guessing. The remaining open-access models show poor results in this setting. Notably, weak results of LLaVA-1 Mistral-7B on real-world datasets are primarily attributed to its incorrect generation of JSON output required to format the result.
Binary classification on RWR variants. In the Ground- -ruth and Incorrect Label settings, models performed similarly across the Bongard- Bongard-RWR RWR dataset and its three variants (RWR-G, G, RWR-S, RWR-SG SG), indicating that color removal and cropping did not substantially affect performance in these settings. However, in the Images to Sides setup (Table 2). several models showed clear differences. For example. Phi-3.5 --Viin improved from solving 1 problem on RWR to 16 on RWR-G GG 8 on RWR-S, and 18 on RWR-SG, suggesting that color removal and cropping reduce unnecessary variation, and shift the focus of the model toward structural patterns. Encouraged by these improvements, we evaluated larger model variants, including Intern VL2.5 VL2.5 38B and 78B (Chen et al., 2024a;c), Qwen2VL 72B-Instruee (Bai et al., 2023; Wang et al., 2024), and I.LaVA-NeXT 1 10B, which showed similar trends. Overall, the RWR-G variant highlighted models' sensitivity to color, while RWR-S and RWR-SG revealed how cropping affects performance by reducing empty space between panels. Nevertheless, in most cases the models were unable to surpass the random guess threshold, highlighting their inherent limitations in recognizing Bongard-RRy -WWR concepts.
Multi-class classification (concept selection). Fig. 9 and Table 3 present detailed results of the multi-class classification experiments, where models were tasked with selecting the correct concept from & E {2, 4, 8, 16) choices across all four datasets. First, the performance improves from InternVL2 8B VL25 8B, reflecting improvements in training data and model architecture in newer model versions. Next, moving from Intern VL2.5 8B Intern VL2.5 38B and InterVVL2.5 78B shows further gains, highlighting the benefits of scaling up model size, which explore further in Appendix D. Comparing Intern VI.2.5 78B with Qwen2VI. Instruutt reveals that state-of-the-art models also from other families face challenges on this task, particularly on Bongard-RWi RWWR and synthetic BPs. It's worth noting that the average human accuracy of 65% on Bongard- RWWR observed in the free-form setup is similar to the performance of best models in the much simpler multiple-choice setup with only l = 4 candidate answers. This observation underscores the limitations of current models compared to humans.
18

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Bongard HOI
Sangard-OpenWorld
Bongard Bongard-RWR RWR
Synthetic BPs
InternVL2 8B
InternVL2.5 8B
IntemVL2.5 38B
InterVL2.55 78B
Qwen2VL 72B
100
Accuracy (%)
75
50
25
k 16
2
16
16
2
16
4
8
16
k
Figure 9: Concept selection, Multi-class classification accuracy of MILMs across datasets and E (2,4, 8, 16) choices. Detailed results are presented Table
Table 3: Concept selection. Multi-class classification accuracy of MLLMS across datasets and k E 4,8, 16) choices. VL2 is denoted IVL2, Intern VL2.5 as IVL2.5, and Qwen2VL Instruct Q2VL. High-level trends are illustrated in Fig. 9.
BONGARD HOT
B- B-OPENWORLD
BONGARD-RWR
SYNTHETIC BPs
16
2
16
8
16
2
IVL2
0.95
0.97
0.90
0.90
0.92
0.89
0.92
0.81
0.48
0.50
0.25
0.25
0.61
0.49
0.30
0.16
IVL2.5 8B
0.97
0.96
0.92
0.93
0.96
0.96
0.94
0.87
0.82
0.52
0.45
0.27
0.73
0.56
0.41
0.30
IVL2.5 38B
1.00
0.97
0.99
0.97
1.00
1.00
0.99
0.98
0.73
0.65
0.52
0.32
0.82
0.71
0.66
0.46
IVL2.5 78B
1.00
0.99
0.98
0.99
0.99
1.00
0.99
1.00
0.88
0.65
0.68
0.38
0.86
0.83
0.75
0.70
Q2VL 72B
1.00
0.99
0.95
0.96
0.98
0.98
0.99
0.97
0.85
0.67
0.57
0.32
0.91
0.69
0.59
0.41
D. The Impact of Model Scaling on Abstract Reasoning Abilities
Performance of MLLMs on downstream tasks is often correlated with the number of model parameters and the size of training datasets (Kaplan et al., 2020; Hoffmaan et al., 2022). To investigate the relationship between model scaling and abstract reasoning performance, we conducted experiments with diverse set of model sizes across proprictary and open-access MLLMs. To this end, we evaluated both smaller and larger variants of the selected models. Specifically, we considered GPT-4o mini and Gemini 1.5 Flash as smaller counterparts to GPT-4o and Gemini 1.5 Pro, resp. Also, tested multiple configurations of Intern VL2 and LLaVA-NeXT model families including Intern VL2-8B, Intern VL2-26B, InternVL2-44 LLaVA-v1.6 Vicuna- 13B, LLaVA-v1.66 34B, LLaVA-NeXT 72B, and LLaVA-NeXTI 110B. We conducted experiments on all 4 datasets using two solution strategies, including Direct, which is an intuitive baseline, and Descriptive, the most effective strategy identified in the main experiments.
The results are presented in Fig. 10. general, larger proprictary models outperformed their smaller counterparts in 10 out of 16 cases. However, smaller variants sometimes performed better than larger ones. For instance, on Bongard HOI with the Direct strategy. GPT-4o mini and Gemini 1.5 Flash surpassed their larger alternatives. This suggests that smaller models can achieve competitive performance in abstract reasoning.
For open-access models, performance consistently improved with model size. For example, the results of Intern VL2 on Bongard HOI increased from 12 to 25 and from to 29 for Direct and Descriptive strategies, resp. Similarly, on Bongard HOI, the performance of LLaVA-NeXT improved from to 27 and from to 27 for the two strategies. Analogous improvements were observed on Songard-OpenWone highlighting the potential benefits of model scaling.
Despite these significant improvements open-access models, proprietary models consistently outperformed them. In particular, GPT-4o mini achieved worse results than the best open- model in a single case only, i.e., Bongard HOI using the Descriptive strategy (26 vs. 29). Although model scaling demonstrates its potential to enhance abstract reasoning, as shown by the open-access models, the relatively strong performance of GPT-4o mini shows that a large parameter count is not necessarily critical for excelling in abstract reasoning tasks. Consequently, these results suggest that simply scaling model size may be insufficient to achieve stronger abstract reasoning capabilities and future efforts should explicitly address this aspect, e.g., by incorporating AVR datasets into model training.
19

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Table 4: Evaluation results. The number of correct answers to the first 100 synthetic BPs, 100 selected BPs from Bongard HOI and Bongard-Oper OpenWorld, World, and all 60 BPs from Bongard- RWR. The best result for a given strategy marked in bold, and the second best underlined.
GPT-4
GPT-4 TURBO
GEMINI
CLAUDE
INTERNVL2 8B
LLAVA-1.6
PIXTRAL
SYNTHETIC BPS
0
3.5 SONNET
MISTRAL 7B
3.5V
12B
GROUND-TRUTH
87
48
78
84
30
70
22
92
INCORRECT LABEL
79
89
70
78
24
27
IMAGES TO SIDES
72
56
69
75
32
33
57
DIRECT
17
6
7
13
0
0
0
1
DESCRIPTIVE
17
15
21
19
0
1
2
4
DASCKIPTIV--TTERATI
9
7
4
8
1
0
2
DESCRIPTIVE-DIRECT
22
13
20
22
0
5
CONTRASTIVE
10
8
17
15
0
0
1
0
CONTRASTIVE -IVERAATIVVEEEEEIIIVVEE
20
4
CONTRASTIVE-
19
20
18
GPT-4
GPT-4
GEMINI
CLAUDE 3.5 SONNET
INTERNVL2
LLAVA-1.6 MISTRAL 7B
PHI
PIXTRAL
BONGARD HOI
0
TURBO
1.5
8B
3.5V
12B
GROUND-TRUTH
73
74
60
32
41
70
49
100
INCORRECT LABEL
100
100
96
100
90
32
33
IMAGES SIDES
99
95
0
86
DIRECT
35
22
23
5
12
5
28
DESCRIPTIVE
42
45
40
44
2
4
4
27
32
23
6
6
0
1
2
11
DESCRIPTIVE-DIRECT
45
40
30
29
7
3
7
25
CONTRASTIVE
18
5
15
13
2
7
CONTRASTIVE--TERRNA
22
20
15
2
14
CONTRASTIVE-DIREC
25
27
15
7
GPT-4
GPT-4
GEMINI
CLAUDE 3.5 SONNET
INTERNVL2
LLAVA-1.6
PHI
PIXTRAL
BONGARD-OPENWORLD
TURBO
1.5
8B
MISTRAL 7B
3.5V
12B
GROUND-TRUTH
80
52
59
37
49
72
31
99
INCORRECT LABEL
100
100
98
99
86
41
83
29
IMAGES SIDES
94
86
19
2
87
DIRECT
21
13
10
11
12
33
DESCRIPTIVE
46
57
32
53
18
16
12
34
DESCRIPTIVE-ITER.AA
31
24
6
13
4
2
2
11
DESCRIFTVE-DIRECE
52
52
25
46
24
8
38
CONTRASTIVE
19
12
11
21
7
1
14
CONTRASTTIVE-ITEAATI
25
21
34
5
18
CONTRASTIVE-DDRECT
35
25
17
22
12
4
27
GPT-4
GPT-4
GEMINI
CLAUDE
INTERNVL2
LLAVA- 1.6
PHI
PIXTRAL
BONGARD-RWR
TURBO
1.5 PRO
3.5 SONNET
8B
MISTRAL 7B
3.5V
12B
GROUND-TRUTH
22
5
26
10
13
38
21
58
INCORRECT LABEL
59
60
52
60
53
21
47
9
IMAGES SIDES
15
24
16
26
DIRECT
0
DESCRIPTIVE
8
5
7
13
0
0
1
DESCRIPTIVE-ITERAATI
4
1
O
3
0
0
0
0
DISCRIPTIVEDDIREO
5
7
5
6
O
0
0
2
CONTRASTIVE
2
O
2
O
0
0
0
CONTRASTIVE--TERRATE
5
0
CONTRASTIVE
4
4
0
E. Evaluation of MILMs Answers
Preliminary experiments revealed that proprietary MILMSs are generally much more effective in solving Bongard Problems than open, publicly-availaae MLLMs. Therefore, all efforts devoted to optimizing the final scores, in particular tuning the evaluation prompt were performed using these 4 commercial MLLMs.
Open-ended characteristics of BPs stemming from textual form of an answer, and the number of considered models (8).
20

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 2: Initial prompt used in MLLM answer evaluation. We focused on its clarity and simplicity.
You are a logic module designed to provide accurate answers. In Bongard Problem the objective is to spo: the difference between the contents of images located on the two opposite sides of the problem. You are given correct labels of these sides and must decide whether the answer provided by the user is correct and matches with those labels. Answer with 'OK' or 'WRONG'.
LEFT SIDE LABEL: <left_label>
RIGHT SIDE LABEL: <right_label>
USER ANSWER: <model_answer>
generation strategies (7), datasets (4), and BP instances per dataset (60 in Bongard-RWR and 100 in the remaining cases) require the use of an automated NLP-based evaluation of the model's answers. For this task we employed MLLMs with a specially designed prompt. The initial version of the evaluation prompt (see Prompt was intentionally relatively simple- model received answer to be evaluated well as the ground-truth labels, and was requested to output binary yes/no answer. This prompt formulation turned out to be too simplistic. While the level of agreement between all models was relatively high (87% of responses were rated unanimously by all models), as illustrated in Fig. 1 la, manual inspection of the selected answers revealed that the assessment was generally too optimistic and relatively many evaluations wrongly pointed to correct answers.
E.1. Evaluation Prompt Optimization
Due to the above evaluation disagreement, we made an attempt to optimize the prompt based on the GPT-40 solutions for the additional 20 BPs (# 101 - # 120) that were not used in the main experiments. First, following the few-shot prompting technique, we expanded the prompt include two examples showing a possible logical difference between correct and incorrect answers. Furthermore, added sentence which requested strict logical compliance with the provided labels. However, this refinement appeared to be too strong, as 2 (out of 4) models didn't 't evaluate any of the solutions as correct.
To impose some flexibility, we changed the word strictly to logically, but this resulted in an increased rate of false- possiivve evaluations. Finally, we combined these two prompts, obtaining the outcome closest to the manual (our human) evaluation. The final version of our evaluation prompt is listed in Prompt 3. Additionally, we attempted to attach the image of the evaluated BP instance to each version of our prompt, but this actually confused the models rather than improving their results, so we ultimately abandoned this option and stuck with the fully text-based prompt.
Although the consistency of results regarded as the number of unanimous assessments stayed at the same level (87%) (see Fig. 1lb), the number of answers rated as correct significantly decreased, which was in accordance with our random manual verification.
Despite lowering the results variation, there were still BPs for which the assessment varied. Therefore, we eventually decided to use hard voting to ensemble all models' evaluations. We marked a solution as correct at least of the 4 models evaluated as correct. This approach brought better results than the majority voting.
E.2. Manual Verification of the Evaluation Performance of MLLMs
In order to finally assess the efficacy of Prompt 3 manually checked the models' evaluation performance the 100 BPs solved by GPT-4o using the Descriptive strategy. As shown in Table 4 in the main paper, all proprietary models achieved better scores on incorrect labels classification. For this reason, we decided to manually verify only the problems evaluated as correct by at least one MILM, assuming that those incorrect are generally evaluated properly. The comparison between the evaluation performance of the initial and final prompts and our manual evaluation is presented in Table 5. All models denotes evaluations where solution is marked as correct only if all models evaluate it as correct. Similarly, any model
21

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 3: The final version of the evaluation prompt. It is enriched with the few-shot prompting technique and imposes a logical compliance with provided labels.
You are a logic module designed to provide accurate answers.
In Bongard Problem the objective is to spet the difference between the contents of images located on the two opposite sides of the proplem.
You are given correct labels of these sides and must decide whether the answer provided
+ by the user is correct and matches with those labels. Answer with 'OK' or 'WRONG'.
The user's answer has to strictly logically match the labels, as shown examples.
FIRST EXAMPLE:
LEFT SIDE LABEL: All shapes small.
RIGHT SIDE LABEL: All shapes big.
USER ANSWER: On the left side, one of the shapes is small. 1. On the right side, all of the shapes are big.
EVALUATION; WRONG
END OF FIRST EXAMPLE.
SECOND EXAMPLE:
LEFT SIDE LABEL: All shapes are small.
RIGHT
LABEL: All shapes big.
USER ANSWER: On the left side, all of the shapes are small. On the right side, all of the shapes are big.
EVALUATION: OK
END OF SECOND EXAMPLE.
LEFT SIDE LABEL: <Left_labels
RIGHT SIDE LABEL: <right_label>
USER ANSWER: <model answer>
Table 5: Consensus with manual evaluation synthetic BPs. The percentage of the solutions evaluated the in our manual evaluation in BP instances #1 100 (Bongard, 1970). The assessed solutions were obtained by GPT-4o using the Descriptive prompting strategy.
INITIAL PROMPT
FINAL PROMPT
ALL MODELS
ANY MODEL
VOTING
ALL MODELS
ANY MODEL
VOTING
0.93
0.90
0.94
0.90
0.96
0.99
refers to the cases where solution marked as correct if at least one model evaluates it as correct. Voting refers to the hard-voting scheme described section E. It important to observe that the chosen voting evaluation method differed from the manual evaluation in only one specific problem, which depicted in Fig. 12.
In addition, checked the performance of enhanced evaluation prompt 20 new, not used in other experiments, manually evaluated Bongard-Open problems solved by GPT-4o using the Descriptive strategy. Again, the use of Prompt 3 visibly increased the consensus with manual evaluation (see Table 6). The difference between our manual evaluation and the voting scheme occurred only in 2 problem solutions whose correctness is disputable (see Fig. 13).
While the choice examples shown in the prompt may in some cases impact the evaluation performance, the finally proposed evaluation prompt seems be well suited to both domains: synthetic and real-world, and should potentially be effective other similar datasets and solving strategies.
F. Bongard-RWR Dataset
The dataset generation algorithm is presented in Algorithm 1 using notation introduced in Section 4.
Furthermore, Fig. 15 provides additional examples the proposed Bongard- Bongar--WWR RWR dataset. Each subfigure presents a comparison between the synthetic Bongard problem and its respective real-world translation in Bongard-RWi Examples
22

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Table 6: Consensus with manual evaluation on Bongard-Open World. The percentage of the solutions evaluated the same as in our manual evaluation in the additional 20 Longard-Open World instances # 101 - # #20 (Wu et al., 2024), not used in the main experiment. The solutions were obtained by GPT-4o using the Descriptive prompting strategy.
INITIAL PROMPT
FINAL PROMPT
ALL MODELS
ANY MODEL
VOTING
ALL MODELS
ANY MODEL
VOTING
0.75
0.70
0.70
0.65
0.85
0.90
15a and 15b were translated automatically, whereas 15c and 15d were constructed fully manually, including building an appropriate scene and taking picture. Additionally, Fig. 14 shows a particular approach taken when translating given synthetic BP to its Bongard- -War counterpart (problems not translated and those rejected after translation are combined into one category).
G. Coverage of Bongard-RWR Instances
Even though the final results of individual models and strategies solving Bongard-RWR somewhat unsatisfactory, especially in the case of open language response generation, it is worth to examine the degree of overlap of correct answers across the tested models. On the one hand, the results of ground-truth classification and model's ' disagreement on the solution evaluation clearly confirm inability of any single model to solving all problems from the Bongard-RWR dataset. On the other hand, it is likely that the overlap is not complete, and it is possible to expand the solution coverage by appropriate mixture-of-expertt approach. This indeed the case in our experiments, as illustrated in Figs. 16 - 23. While single proprietary MLLMs solved between 9 and 14 instances, the number of instances solved by any MLLM equaled 23. We leave exploration of this path for future research.
H. Comparison of Synthetic Bongard vs. Bongard-RWR results
Our research shows that all of the tested models have difficulty solving synthetic concepts when applied to real-world images. Comparing the results for both datasets (see Figs. 17 and 23) we identified some discrepancies. Four problems that domain of the Bongard-RMy dataset:
remained unsolved in the synthetic BPs were successfully solved in the #56, #87, #88, and #98. However, three of these problems differ slightly from their synthetic counterparts. The images in #56 from Bongard- RWR feature variety of colors instead of the usual Blcc-and--a and-white figures. Furthermore, in real-world version of problem #87 more images feature disjoint elements instead of multi-part objects, which may have nudged the model toward the correct answer. Additionally, in problem #98, the figures are shown against hatched texture, which was not accounted for in the real-world translation.
Conversely, 22 problems that were solved the synthetic BPs were not solved in the Bongard- dataset. In most cases, the models focused on general associations that did not apply to all the images. For example, the concept presented in problem #3 is: "LEFT: Outline figures, RIGHT: Solid figures." Claude 3.5 Sonnet, using the Descriptive strategy, responded: "LEFT: focuses practical, everyday objects scenes, RIGHT: emphasizes aesthetic, artistic, decorative elements that serve more for visual appeal than utility". Nevertheless, both sides feature red coffee cup with saucer, which matches both generated descriptions. The key difference lies in the color of the saucer's rim, which determines whether it should be considered as an outline or solid figure.
I. MILM Prompts
L.1. Prompts for Bongard-RWF Generation
At first, Prompt 4 was employed generate real-world representations of synthetic concepts. After downloading the images corresponding to these textual descriptions, Prompt 5 was used to select those images that correctly represent given concept translation. In addition to the left and right concepts, we also provided prompts briefly explaining the context that the image should match. These prompts were generated during the translation stage of our algorithm (see the fourth line in Algorithm 1).
23

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Algorithm 1 The Bongard-RWR dataset generation.
I: Input: A of synthetic concepts Y
Output: set of generated instances RWR
3: M + 15, N+ 10, T
4:
5: for xx E do
6:
D* GeneraaeeTransaaiionnny" N)
1x 0. P
8:
9:
for D.N E D* do
10:
for m <- 1 to M do
11:
for [T,, R) do
12:
1 +- Download ( m)
13:
if is accepted by model then T₂ 7 U (T)
14:
15:
end
16:
end for
17:
18:
if
1 2 and P (i)
11, || 2 then
20:
break
21:
end
22:
end for
23:
[I| 2 T then
25:
Break end if
26:
27:
end for
28:
29: _ +- 0. R +
30:
|| 2 T then
31:
for +- 1 to 6 do
32:
p Plk mod T j { k
33:
34:
for
S L. R) do
35:
5*+ S* (+)*(0))
36:
end for
37:
end for
38:
39:
RWR* < (C* R*}
40: RWR < RWR U (RWR*)
41:
end if
42:
end for
1.2. Prompt Describing the Bongard Problem
Prompt describing the BP task has been placed at the beginning of each solving strategy introduced in Section 3.1.
I.3. Prompts for Classification Strategies
Prompt 8 was used to assess solution correctness (see Section 3.2). Prompt 7 was used to assign images to sides (see Section 3.3).
24

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 4: Initial concept prompt used in construction of Bongard-RYy
Your goal is to translate a comparison concept from the gecmetric domain to the + real-world domain. Your cranslations should be expressible as images.
Example: Geometric domain: triangles squares
"left"; "concept": "pyramida"
"right": {
"concept"; "rectangular buildings"
Give <number> unique translations for the following concept raw JSON array of objects (same in the example above). <concept>
Prompt 5: Prompt used for the selection of proper images for translated concept.
You translated concept comparison geometric domain to the real-world domain as follows:
Geometric domain: <left_geommtriccccoeepp <right_geometric_comep.. Real world domain:
"left":
"concept"; cleft_conceptt]
"prompt"; <left_ccocopt__decripii..
"right": "concept": bright__onceppii
"prompt"; teljb__concppt_eeerrrpp..oo
Now, you need to check if the queried image matches your translation and provides enough information to distinguish it from the other concept. Don't focus too much on the prompt. it's just hint for you to understand the concept better. Provided image represents <side_concept>
Give your EVALUATION; OK
in the following format:
EXPLANATION: <here you can provide additional information
or RVALUAT REJECTED
EXPLANATION: <here you can provide additional information>
Prompt 6: Prompt explaining Bongard problems an MLLM.
A Bongard Problem is composed of left and right sides separated by line. Each side contains six images. All images belonging to one side present a common concept, which is lacking in all images from the other side. The goal is to describe the rule that fits all images on the left side, but none on the right, and, conversely, the rule that fits all images on the right side, but none on the left. The description of the rule should be simple and concise.
Example 1: All shapes on left are smail. All shapes on right are big. Example 2: The left side contains circles. The right side contains triangles.
25

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 7: Prompt used for images side classification (see Fig. c). Two examples were provided not bias the results of the model.
You are vision underatanding module designed to provide short, clear and accurate answers. Your goal is to classify two test images to the corresponding side of the Bongard Problem, LEFT or RIGHT. Each image belongs to exactly one class. The test images belong to different clasecs.
The images are always provided correctly. Respond only the specific request. Respond in Json using the following format.
FIRST EXAMPLE:
Left images: <small shapes>
Right images: <big shapes>
First test image: <small shape>
Second test image: <big shape>
Response:
"first":
:oxplanation"; "The test image shows small shape, similarly all images the left side. Conversely, the images on the right side feature big shapes.",
"concept" "small vs big", b "answer"; "LEFT" "second";
"explanation"; "The test image shows big shape, similarly as all images on right. The images on left, the other hand, feature small shapes.", "concept": "small vs big", "answer"; "RIGHT"
END OF FIRST EXAMPLE
SECOND EXAMPLE:
Left images: <eircles>
Right images: <triangles>
First test image: <triangle>
Second test image: <circle>
Response:
"first": {
"explanation"; "The test image shows a triangle, which matches all images on right. In contrast, left side images feature circles.", "concept"; "circles triangles", "answer": "RIGHT"
"second":
"explanation"; "The test image shows circle, which matches all images en left. 4 Conversely, the right side images feature triangles.", "concept": "circles vs triangles", "answer"; "LEFT"
END OF SECOND EXAMPLE
Prompt 8: Prompt used for the solution correctness assessment (see Fig. (b).
You are a vision understanding module gned to provide short, clear and accurate answers. Your goal is to evaluate the correctness of the provided answer to the given Bongard Problem. All images are provided correctly. De not explain the answer, just evaluate it. Respond 'OK" it the answer is correct, otherwise respond 'WRONG'.
User answer: aiser_answer>
26

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt 9: Prompt used for the Direct strategy. (see Fig. a).
You a vision understanding module designed provide short, clear and accurate answers. Your goal is to solve the provided Bongard Problem. What is the difference between the two sides of the problem?
Prompt 10: Prompt used to obtain the image descriptions in the Descriptive strategy (see Fig. b).
The provided image is a part of an abstract visual reasoning problem. Describe all crucial properties of the image. Your descript should be as concise as possible. Focus on the most important details. The image provided correctly. Respond only with deseriptions.
L.4. Prompts for Natural Language Answer Generation Strategies
Prompts 9-16 were used for natural language answer generation (see Section 3.1).
Prompt 11: Prompt used for the Descriptive and Descriptive-direct strategies (see Fig. b).
You are a vision understanding module desi gned to provide short, clear and accurate
answers. Your goal is to solve the provided Bongard Problem using descriptions of its images.
LEFT IMAGES: <left_descriptionss
RIGHT IMAGES: <right_descripp inserripioon>
What is the difference between the two sides of the problem?
27

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Prompt used to obtain the image descriptions in the Descriptive-teraatie strategy (see Fig. c).. After the last image, we used the prompt: "That was the last image. Now provide your final answer."
Prompt 12:
You'll receive a sequence
images that are part
single side of Bongard find
Problem. The images will be provided one by one. Your geal concept presented in all images. Your description should be concise possible. Focus the most important details. Try to enhance the description of the concept after each image.
The image is al always provided correctly. Respond only to the speci fic request. The first image will be provided in the next message.
Prompt 13: Prompt used for the rescriptiiv--iteratii strategy (see Fig. Cc).
You are a vision understanding module designed to provide short, clear and accurate + answers. Your goal is to solve the provided Hongard Problem using descriptions of two sides of the problem.
LEFT SIDE DESCRIPTION: cleft_description
RIGHT SIDE DESCRIPTION:
cright_description
What is the difference between the two sides of the problem?
Prompt 14: Prompt used to obtain the comparison between the left and right image in the Contrastive strategy (see Fig. d), After the last image, we used the prompt: "That was the last image. Now provide your final answer."
You are given two images extracted from the left and right side Bongard Problem, espectively. Your goal is to compare the images. Your comparison should be as concise possible.
Prompt 15: Prompt used for the Contrastive and Contrastive-direct strategics (see Fig. 2d).
You are vision understanding module designed to provide short, clear and accurate answers. Your goal is to solve the provides, ded Bongard Problem using compari sons between pairs of images. Each pair contains one image from the left and one from the right side of the problem.
COMPARISONS:
<comparison >
What is the difference between the sides of the problem?
Prompt 16:
Prompt used for the Contrastive-iterative strategy (see Fig. Te. After the last pair of images, we used the prompt: "It was the last pair of images. What the difference between the two sides the problem?"
You
vision understanding module designed to provide short, clear and accurate answers. Your goal is to solve the provided Bongard Problem. You'l1 receive sequence of image pairs. Each pair contains one image from the left and one from the right side of the problem. In each step compare the two images and refine the definitions of concepts that describe left and right sides of the problem. Your description should be as concise as possible. Focus on the most important details. The first pair will be provided the next message.
28

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
GPT-
Gomini 1.5
IntermLL
LLaVA-NeXT
mini
Flash
Pro
8B
26B
40B
76B
7B
34B
72B
110B
Direct
Synthetic BPs
Descriptive
30
curnec annwers
30
21
17
17
17
12
10
5
0
0
GPT-4 Gemini 1.5
InterVLL2
LLaVA-NeXT
GPT-4 Gemini 1.5
InterVLL2
LLaVA-NeXT
Direct
Bongard HO1
Descriptive
40
38
currect anwwers
35
34
29
27
26
27
23
20
18
14
12
11
10
GPT-4
Gemini 1.5
InterVVL2
LLaVA-NeXT
GPT-4 Gemini 1,5
InterVLL2
LLaVA-NeXT
Direct
Bongard-Ope Open World
Descriptive
46
40
clorect answers
31
32
26
23
20
18
18
18
12
10
GPT-4
Gemini 1.5
InterVVL2
LLaVA-NeXT
GPT-4 Gemini 1.5
IntemVL2
LLaVA-NeXT
Direct
Bongard-RWR
Descriptive
correct answers
10
3
3
3
0
GPT-4 Gemini 1.5
InterVVL2
LLaVA-NeXT
GPT-4 Gemini 1.5
IntemnL22
LLaVA-NeXT
Figure 10: The impact of model scaling on abstract reasoning abilities. We considered diverse of model sizes across proprietary and open-access MLLMs. The experiments cover all 4 datasets using the Direct and Descriptive solution strategies.
29

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Marked
Marked
77
wrong by all models
wrong by models
conrect by models
28
conrect by all models
20
by
GPTT4 Turbo correct
27
19
by GPT-4
by Oemini 1.5 Pro
25
Gemini 1.5 Pro
15
correct by Claude 3.5 Sonnet
connect by Cloude 3.5 Sonnet
20
18
10
2
4
6
9
10
2
4
5
7
10
14
19
21
22
23
24
26
27
28
29
22
23
25
27
28
29
30
31
32
34
36
37
38
39
32
33
35
37
38
39
40
44
42
47
51
52
53
54
55
56
57
58
59
60
51
92
53
54
55
56
57
58
59
60
64
69
65
72
73
74
76
77
78
79
71
72
73
74
75
76
77
78
79
80
82
84
86
887
89
82
83
85
87
88
89
90
96
92
97
100
(a) Results with the initial evaluation prompt.
(b) Results with the final evaluation prompt.
Figure 11: Models' agreement the evaluation of BPs. The assessed solutions generated by GPT-4o with the Descriptive strategy. The numbers refer to the BP tasks from (Bongard, 1968). Green indicates tasks unanimously evaluated as correctly solved by all models, while white indicates unanimous incorrect evaluations. Other colors highlight tasks marked as correctly solved by individual models.
30

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Z
M
A
Figure 12: Synthetic BP #85. This is the only BP instance for which the voting- voting-based evaluation of MLLM solutions differed from our manual evaluation. Left: Three parts. Right: Five parts. GPT-4o answer: Left: All images are composed of exactly three lines. Right: All images are composed of more than three lines. Voting marked it as incorrect, whereas in manual evaluation it was marked as correct. The first difference lies in the meaning of the words lines and parts, which, in this visual context, seems identical. The second difference stems from the number of the parts on the right side of the problem. The answer seems to be correct, obviously, five is more than three. However, could argue that the answer is incomplete, each of the squares on the right side clearly depicts exactly five parts.
W Canstack
(a) Left: Underground tunnels beneath the city. Right: NOT Un- derground tunnels beneath the city. GPT-4o solution: Left; All images depict scenes that are primarily indoors or underground. Right: All images depict scenes that are primarily outdoors. We evaluated it as correct, while the voting marked it as incorrect. The difference arises from how the left. concept is perceived. Although the images on the left depict an underground setting, they do not appear to represent an indoor scene. Nevertheless, one of the statements is still true.
(b) Left: A woman wearing a white wedding dress, Right: NOT A woman wearing white wedding dress. GPT-4o solution: Left: All images feature women white wedding dresses or wedding- related scenes. Right: All images feature women non-wedding attire, wearing dresses suits of various colors other than white. We evaluated it incorrect, while the voting marked it correct. The difference stems from small detail in the right concept. One of the images on the right depicts a woman in a white suit, which conflicts with the model's 's answer.
Figure 13: The only two Bongard-Open World problems (out of the selected 20) for which the voting evaluation differed from our manual evaluation. The correctness of GPT-4o's solutions to these problems is disputable.
31

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Bongard-RWR instances that were:
2
3
5
6
8
Not translated removed
11
12
13
14
15
16
17
18
19
20
Generated fully automatically
Generated automatically and corrected manually
22
23
25
26
28
29
Generated manually from existing pictures
Generated manually from photos manually built scenes
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
52
53
55
58
59
61
62
63
64
65
66
67
68
69
70
24
71
72
73
74
75
76
77
78
79
80
17
82
83
85
86
88
89
90
91
92
93
94
95
96
97
98
99
100
Figure 14: The structure of the Bongard-RWR dataset. Each color denotes the genesis of the translation of the respective problem. Problems not translated and those rejected after translation are combined into one category. Red color outlines denote problems which were solved by any model using any strategy. There is no visible correlation between the set of solved problem and the methods used for their generation.
32

---

Reasoning Limitations of Multinodal Large Language Models. A Case Study of Bongard Problems
D
0
X
(a) Synthetic BP #8 with automatically translated Bongard-RNe RWR version. Left: Figures on the right side. Right: Figures on the side.
OPEN
4
(b) Synthetic BP # 10 with its automatically translated Bongard-RWR version. Left: Triangles. Right: Quadrangles.
(c) Synthetic #41 with it manually constructed Bongard-RWR version. Left: Outline cireles on one straight line. Right: Outline circles not on one straight line.
V
A
A
A
O
A
(d) Synthetic BP #-17 with its manually constructed Bongard-RWR version. Left: Triangle top of the circle. Right: Circle top of the triangle.
Figure 15: Additional examples of Bongard-RWF instances.
33

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Not raakeel
No: rosheel
Salvel all stategle
Sakee strategles Sahvd strongy
Salved strttgy
35
Covtaallee-testiie
Desoritive stranng
Dioer erastegy
Descriptive
Deeriptineiteatiie
Desriptivediett
8
10
29
33
34
36
38
30
33
34
37
40
44
48
49
42
58
84
73
74
75
77
78
79
72
73
74
77
82
85
57
38
89
52
63
84
87
Figure 16: Overall result each strategy on synthethic BPs. Colormaps depict all problems solved by any tested model using the respective prompting strategy. The right figure aggregates strategies into corresponding groups for better coverage exposure.
resolved
8
Solved models
Solved model
GPT-4 Turbo
12
13
14
15
16
17
18
19
20
GPT-4o
Gemini 1.5 Pro
21
22
23
24
25
26
27
28
29
30
Claude 3,5 Sonnet
InsernVL2 8B
LiaVa 1.6 Mistral 7B
34
38
Phi 3.5V
Pixtral 128
48
51
53
54
57
58
59
53
61
62
63
64
65
66
67
68
69
70
39
73
74
77
78
79
33
26
84
88
93
94
95
98
99
100
Figure 17: Final summary all experiments on the synthetic Bongard Problems dataset. The colormap aggregates all problems solved by any tested model using any generation prompting strategy. Overall, the models collectively managed solve 53 problems.
34

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Not raaleel
No: resheel
Salvel all stategle
Sakee strategles Sahvd stroney
86
Salved strttey
Covtaallee-testiie
Desoritive stranng Diowr erastegy
Descriptive
Desriptineitratiee
Desriptiisdreet
3
29
34
37
30
32
33
37
40
10
42
90
58
57
74 76
72
73
74
76
77
78
80
84 57
38
52
83
87
88
90
Figure 18: Overall result of each strategy on Bongard HOL Colormaps depict all problems solved by any tested model using the respective prompting strategy. The right figure aggregates strategies into corresponding groups for better coverage exposure.
resolved
8
Solved models
Salved model
GPT-4 Turbo
12
13
14
16
17
18
19
20
GPT-4o
Gemini 1.5 Pro
21
22
23
24
25
26
27
28
29
30
Claude 3,5 Sonnet
InsernVL2 8B
LiaVa 1.6 Mistral 7B
39
34
35
38
Phi 3.5V
Pixtral 128
51
53
54
55
57
58
59
67
61
62
63
64
65
66
67
68
69
70
61
60
58
73
744
75
77
78
79
88
12
93
94
95
98
99
100
Figure 19: Final summary all experiments on the Bongard HOI dataset. The colormap aggregates problems solved by any tested model using any generation prompting strategy. Overall, the models collectively managed solve 88 problems.
35

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Not raaleel
No: rosheel
Salvel all stategle
Sakee strategles Sahvd stronay
93
Salved strttey
74
90
Covtaallee-testiie
Desoritive stranng Diowr erastegy
Descriptive
55
Deeriptineiteatiee
Desriptiosdrett
3
10
33
34
30
33
34
37
38
40
46
48
43
48
50
71
77
78
72
73
74
76
77
78
82
55
87
38
52
83
84
67
90
Figure 20: Overall result each strategy on Bongard- -ppnn World. Colormaps depict all problems solved by any tested model using the respective prompting strategy. The right figure aggregates strategies into corresponding groups for better coverage exposure.
resolved
8
Solved models
Salved model
GPT-4 Turbo
12
13
14
15
16
17
18
19
20
GPT-4o
Gemini 1.5 Pro
21
22
23
24
25
26
27
28
29
30
Claude 3,5 Sonnet
InsernVL2 8B
LiaVa 1.6 Mistral 7B
34
37
38
Phi 3.5V
Pixtral 128
48
51
53
54
55
57
58
59
93
61
62
63
64
65
66
67
68
69
70
69
66
73
74
75
77
78
79
88
30
93
94
95
98
99
100
Figure 21: Final summary of experiments on the Bongard-Open World dataset. The colormap aggregates all problems solved by any tested model using any generation prompting strategy. Overall, the models collectively managed solve 93 problems.
36

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
Not raokeel
No: rosheel
Salved stategle
Sikee strategles
40
Salved
Salvd strongy
tnroissdd
Contrastive strotagies
Deccrption aboongge
Corerattieddmet
Dinect stistegy
Descrptive
29
27
31
33
34
35
37
38
30
33
34
36
37
38
40
43
44
45
47
48
49
42
43
50
58
53
84
73
74
75
77
78
79
72
73
74
76
77
78
83
84
85
87
59
52
83
67 88
90
Figure 22: Overall result each strategy on Bongard-RYY RWR. Colormaps depict all problems solved by any tested model using the respective prompting strategy. The right figure aggregates strategies into corresponding groups for better coverage exposure.
8
Solved models
Solved model
Not translated
12
13
14
15
16
17
18
19
20
GPT-4 Turbo
GPT-4o
21
22
23
24
25
26
27
28
29
Gemini 1.5 Pro
Claude 3.5 Sonnet
InternVL2 8B
34
38
LiaV 1.6 Mistral 78
Phi 3.5V
Pixtral 12B
48
53
54
57
58
59
38
61
62
63
64
65
66
67
68
69
70
22
144
73
74
75
77
78
79
93
94
97
88
100
Figure 23: Final summary of all experiments on the Bongard-RWR dataset. The colormap aggregates problems solved by any tested model using any generation prompting strategy. Overall, the models collectively managed solve 22 problems.
37

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
A
A
A
A
A
O
V
A
O
O
(a) Model's answer: "Left: All shapes are filled. Right: At least one shape is unfilled"". Evaluated as incorrect.
(b) Model's answer: "Left: Images are monochromatic (containing only shades of a single color). Right: All images contain at least one hollow (unfilled) shape". Evaluated as correct.
X
1
Y
W
(a) Model's answer: "Left; All shapes are composed of straight lines that form only acute right angles. Right: All shapes contain least instance of obtuse angles curves (e.g.. pentagon, "H" shape, radial pattern"". Evaluated as incorrect.
(b) Model's answer: "Left; All images contain exactly four of some- thing: four orange slices, Four cake slices, four lobes of the four-leaf clover, four pineapples, four fingers with painted nails (thumb visible), four visible elements (key, keychain, watch/compass, leather strap). Right: Images do not contain exactly four of any element: single four-leaf clover. multiple keys on ring, single street lamp with multiple lights. single ladybug. multiple oranges (more than four). stack of five cookies". Evaluated as correct.
Figure 25: Synthetic BP #87 with its manually translated Bongard- version. Correct answer: "Left: Four parts. Right: Five parts". Provided answers belong to Claude 3.5 Sonnet using the Descriptive strategy, as it was the only combination that solved this Bongard-RY -RR instance correctly.
38

---

Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
110
110
0
0
O
8
0
x
(a) Model's answer: "Left: Each image contains an even number of shapes. Right: Each image contains an odd number of shapes". Evaluated as incorrect.
(b) Model's answer: "Left: Contains groups three items. Right: Contains groups of five items". Evaluated as correct.
Figure 26: Synthetic BP #88 with its manually translated Bongard-RWF version. Correct answer: "Left: Three parts. Right: Five parts". Provided answers belong to GPT-4o using the Direct strategy. Overall this problem was solved by three models and four strategies.
(a) Model's 's answer: "Left: All images contain geometric shapes patterns that oriented have components that predominantly point the right. Right: images contain geometric shapes patterns that are either static (not pointing in any specific direction) or have components that do not predominantly point to the right". Evaluated as incorrect.
(b) Model's answer: "Left: All images the left side feature objects that primarily triangular in shape. This includes the slices of cake and pizza, which are cut into triangular pieces, the sailboats, which have triangular sails. Right: All images on the right side feature objects or scenes that are primarily rectangular square shape. This includes the modern buildings, the sandwich, the grid-like urban area, the arrangement of kites, and the plate toasted bread, all which emphasize rectangular or square forms". Evaluated as correct.
Figure 27: Synthetic BP #98 with its manually corrected Bongard-RW3. version. Correct answer: "Left: Three parts. Right: Five parts". Provided answers belong to GPT-4 Turbo using the Descriptive-direct strategy. Overall this problem was solved by two models using different strategies.
39


Reviews:
[{"review_id": "qY7FSRhxNl", "reviewer": "Reviewer_NTSJ", "summary": "In this work, the authors aim to test the abstract visual reasoning performance of MLLMs. To achieve this, they create a testing set that resembles abstract concepts but uses real images, a component lacking in standard Bongard datasets. They develop various prompt strategies and assess the puzzle-solving performance of both open-sourced and proprietary MLLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> The proposed test set is limited to 60 examples.\n\nPlease, see our response to Reviewer 2kYM (*“The sample sizes are relatively small …”*).\n\n> Previous research has explored a similar task [1], …\n\nPlease, see our response to Reviewer Aobr (*“… concurrent work, …”*).\n\n> The dataset generation and evaluation heavily rely on MLLMs, …\n\nWe appreciate the Reviewer’s concern regarding potential MLLM hallucinations and inaccuracies. We recognize this as an important challenge and have taken several steps to mitigate its impact on both dataset generation and evaluation.\n\nFor dataset generation, we implemented a rigorous manual verification process to ensure correctness of all problem instances, as detailed in Section 4 and Appendix F. Of the 100 synthetic BPs considered, only 12 generated instances were incorporated into Bongard-RWR without modification, while 24 required manual (human) adjustments, and 24 were constructed manually, entirely from scratch. Given this manual review of each Bongard-RWR matrix, we are confident that hallucinations and inaccuracies do not affect the final dataset quality.\n\nGiven the scale of our experiments (the number of datasets, problem instances, models, and generation strategies), manual assessment of every response is infeasible. For automatic evaluation, we acknowledge that relying on MLLMs introduces challenges. Hence, in Section 3.2 we proposed an automatic assessment strategy. To improve its robustness, we carefully selected the best-performing evaluation models based on a manual review of selected evaluations and tuned the evaluation prompt to align results with human assessments (Appendix E). While open-ended answer evaluation remains an open problem in NLP research, Tables 3 and 4 in Appendix E show high consistency between our final evaluation pipeline and scores of human experts.\n\nFurthermore, in the evaluation process we did not rely on a single model judgement, but instead evaluated the answers in the mixture-of-experts manner, with the voting scheme of 4 MLLMs manually optimized in preliminary experiments.\n\n> Another concerning factor is that human performance …\n\nWe would like to clarify that the reported 39.2 out of 60 refers to the average number of problems solved, which corresponds to 65% accuracy. Given the open-ended nature of the task, we believe this to be a reasonable score.\n\nAs detailed in Appendix B, every Bongard-RWR instance was solved by at least one human participant, with the best-performing individual solving all but one problem instance (59 out of 60). Furthermore, half of the respondents solved more than 40 problems (two-third), and no participant solved fewer than 23 instances (Figure 7). The problems exhibit varying levels of difficulty (Figure 8) and participants rated it as rather difficult (Figure 6, bottom-right), with an average difficulty score of 7.6 out of 10.\n\nWhile we acknowledge that Bongard-RWR poses a challenge even for humans, we argue that this level of difficulty is desirable. With the rapid progress in MLLM reasoning capabilities, it is crucial to develop benchmarks that push the boundaries of current models. \n\n> … include standard NLP reference-based metrics ...\n\nThank you for this suggestion. We computed BLEU, METEOR, ROUGE_L, CIDEr,  BERTScore metrics for all tested models under Descriptive strategy. However, since these metrics primarily rely on lexical similarity, their usefulness in this setting is not obvious. For instance, the highest CIDEr score was achieved by Gemini-1.5 Pro on OpenWorld (0.13), despite underperforming (solving 32 problems in Table 1). In contrast, GPT-4 Turbo, which solved the most instances (57), had a lower CIDEr score (0.09). \nDue to space limits, we could not all results in this response. We will add and discuss them in detail the next review round.\n\n> For the descriptive dialog strategy, …\n\nIn the Descriptive strategy, we evaluate only the final MLLM response, as depicted in Figure 2b. We will update the corresponding paragraph in Section 3.1 to make this more clear.\n\n> What is the overlap …\n\nWe analyzed this relationship on Bongard-RWR, where we had access to instance-level human responses. Figure 8 (Appendix B) presents a heatmap with the number of human evaluators who successfully solved each problem, with red boundaries marking problems solved by at least one model. Our findings reveal notable discrepancies between human and model performance. Some problems that were relatively easy for humans (e.g. #16, #30, #47) posed significant challenges for the models. Conversely, certain problems that appeared more demanding for some human participants were successfully solved by a model (e.g., #87). This highlights certain differences in the reasoning patterns of humans and MLLMs, which we believe warrants further investigation in future works.", "claims_and_evidence": "Yes, claims are clear.", "methods_and_evaluation_criteria": "Yes, they make sense.", "theoretical_claims": "No proofs or theoretical claims are found in the paper..", "experimental_designs_or_analyses": "The experiments section was checked.", "relation_to_broader_scientific_literature": "The contributions of this work challenge the question of whether the MLLM's decline in abstract visual reasoning results from the synthetic images or the task itself. The authors develop a new test set to address this question.", "essential_references_not_discussed": "[1] analyzes the performance of MLLMs on Bongard problems, akin to this study. The authors are encouraged to reference and compare this work.\n[1] Wüst, Antonia, et al. \"Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?.\" arXiv preprint arXiv:2410.19546 (2024).", "other_strengths_and_weaknesses": "Strengths\n+ This study seeks to address a significant question regarding the reasoning capabilities of multimodal LLMs.\n+ The compilation of test examples aims to evaluate the performance of trained MLLMs.\n+ An examination of various prompting strategies is conducted to solve BPs using MLLMs.\n\nWeaknesses\n+ The proposed test set is limited to 60 examples.\n+ Previous research has explored a similar task [1], which limits the contribution of this work.\n+ The dataset generation and evaluation heavily rely on MLLMs, making it vulnerable to hallucinations and inaccuracies.\n+ Another concerning factor is that human performance on the proposed test set is quite low (39.2%). This raises the question: Is the dataset excessively challenging or flawed? The reviewer recommends that the authors thoroughly examine the examples created and the human responses to eliminate potentially poor samples.\n\n\n[1] Wüst, Antonia, et al. \"Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?.\" arXiv preprint arXiv:2410.19546 (2024).", "other_comments_or_suggestions": "+ Can the authors include standard NLP reference-based metrics (i.e. meteor, spice, cider, bert-score) to address the performance of the models.", "questions_for_authors": "- For the descriptive dialog strategy, the comparison is made with all generated text or just the final MLLM response.  \n- What is the overlap between the puzzles that humans and MLLMs answered correctly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jWsDN4V4np", "reviewer": "Reviewer_Aobr", "summary": "The paper investigates the abstract visual reasoning capabilities of contemporary multimodal large language models (MLLMs), a problem that is aptly motivated. Firstly, the classic Bongard problems are posted to MLMMs in various forms. The machine learning models show performance that is significantly inferior to human baselines. Next, a real-world analogy, \"Bongard-RWR,\" is designed, where experimental evaluation highlights comparable shortcomings of the MLLMs. The key result is that these models fail at Bongard problems not because of the artificial nature of the images but because of a lack of detecting and reasoning over abstract concepts altogether.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> … concurrent work, …\n\n(Wüst et al., 2024) := [1]\n\nThank you for bringing this highly relevant work to our attention. At the time of our submission, we were unaware of [1], as it was presented at a NeurIPS workshop in December 2024, just a month before the ICML submission deadline. We will include [1] in Section 2, as we agree it’s a relevant contribution.\n\nWhile [1] explore a related topic, our work differs in several important ways. Their study evaluates vision language models (VLMs) on the first 100 synthetic BPs in two setups:\n\nOpen-ended language generation, which aligns with our Direct strategy. Likewise, they use an LLM-based judge to assess responses. However, their solution relies solely on GPT-4o, whereas we employ an MLLM ensemble with hard voting (Section 3.2). Additionally, we tune our approach to improve consistency with human evaluation (Appendix E). We believe that these two key points make our evaluation methodology more robust.\nMulti-class classification, where the VLM selects the correct concept from a predefined list. \n\nBeyond these differences, our study extends [1] in multiple directions. We propose 10 distinct solution strategies, compared to 2 in [1]. We conduct experiments on 3 additional datasets, including Bongard HOI, Bongard-OpenWorld, and Bongard-RWR, the latter being a novel contribution of our work. Our main experiments cover 8 models, compared to 5 in [1]. We analyze the impact of model scaling on abstract reasoning capabilities (Appendix D), and examine model performance differences between synthetic and real-world BPs (Appendix H).\n\nIn summary, we believe that due to the above-discussed differences between [1] and our paper our work should be of value to the research community, despite the recent publication of [1].\n\nThanks to this comment, we acknowledged that considering the multi-class classification setup could be a valuable addition to our experiments. During rebuttal, we implemented a corresponding multiclass classification strategy and conducted experiments on all 4 datasets for direct comparison. Preliminary experiments show that models perform best on HOI and OpenWorld (acc above 90%), while Synthetic BPs and RWR pose a bigger challenge (e.g., accuracy equal to 16% and 25% resp. for InternVL2-8B with a total of 16 concepts to choose from). Due to space limitations in this rebuttal, we’d be happy to include the complete results in the next review round.\n\n> Fig. 2: …\n\nIf the confusion stems from the bottom row, where the left-most subfigure is (e) and the right-most is (d), then this arrangement was chosen to optimize space. In that case, we can revise the figure by swapping the order of (d) and (e) to improve the reading flow. If the issue lies elsewhere, we would appreciate any additional details from the Reviewer so that we can address it effectively.\n\n> Fig. 4: …\n\nTo improve clarity, we will add the following sentence to the figure caption: “The top row presents results for the Ground-truth and Incorrect Label strategies, while the bottom row corresponds to the Images to Sides strategy.” Additionally, we will increase the margin between the top and bottom rows for better separation.\n\n> Q1: …\n\nWe would like to clarify the details of our human study. Each participant was explicitly asked to name the rule in natural language, rather than solve a binary classification task. As described in the “Methodology” paragraph in Appendix B: “Bongard-RWR problem instances [...] were posed in an open-ended manner, allowing participants to provide any response they deemed appropriate.” Human responses were then evaluated by expert annotators, who compared them to ground-truth labels and classified them as correct or incorrect. This final binary assessment was used to compute accuracy.\n\nThank you for bringing this point to our attention. We agree that this important aspect of the experimental setup should be included in the main body of the paper. To improve clarity, we will add the following sentence to the “Human results” paragraph in Section 5: “Participants were asked to describe BP concepts in an open-ended manner.”\n\n> Fig. 4/Tab. 1/Fig. 5: …\n\nWe appreciate this valuable suggestion. Since human results were obtained in an open-ended setup, as explained above, they are not directly comparable to the binary classification setup presented in Fig. 4. Additionally, incorporating them into Fig. 5 would be challenging, as this figure illustrates the effects of applying -direct and -iterative variants to the Descriptive and Contrastive strategies. We fully agree that including human performance in the right-most section of Table 1, which concerns Bongard-RWR, would provide a useful reference information for the readers. We will incorporate this addition in the revised version of the paper.", "claims_and_evidence": "The claims are appropriately supported by extensive experiments.", "methods_and_evaluation_criteria": "The possible design space of abstract visual reasoning datasets is very large. Yet, the proposed dataset is a mostly convincing construction that directly serves the dedicated purpose of assessing why MLLMs fail at Bongard-like problems. See the \"Questions to Authors\" below.", "theoretical_claims": "The paper does not contain mathematical theorems or proofs.", "experimental_designs_or_analyses": "The experimental design seems comprehensive. In particular, the selection of MLMMs and the different approaches to querying MLLMs are appropriate.", "relation_to_broader_scientific_literature": "The relation to the wider body of work on the topic seems comprehensive.", "essential_references_not_discussed": "There appears to be concurrent work, such as the following, which should be referenced in a camera-ready version.\n\nWüst, Antonia, Tim Tobiasch, Lukas Helff, Devendra S. Dhami, Constantin A. Rothkopf, Kristian Kersting (2024): *Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?* [arXiv preprint 2410.19546v1](https://arxiv.org/abs/2410.19546).", "other_strengths_and_weaknesses": "The problem can be rather directly derived from existing work and is very significant. The overall quality and clarity are high.", "other_comments_or_suggestions": "Minor Comments:\n- Fig. 2: The reading flow and differing visual ordering confused me.\n- Fig. 4: The results are tricky to map to the experiments. For instance, it took me a bit to understand that the \"Images to Sides\" refers to the bottom row, not the top one.\n- Fig. 4/Tab. 1/Fig. 5: One should also include human performance on these problems in addition to a random agent.", "questions_for_authors": "My questions center around the new dataset's solvability. The concepts in the original BPs are very clear in that there is little clutter and space for ambiguity in the clean synthetic images. However, in Bongard-RWR, the case is somewhat more ambiguous. For instance, in Fig. 1d, the triangle in the image with the candle is hard to make out as such, or in Fig. 1b, one can arguably make out more prominent lines that could count. So:\n\n**Q1**: Did you assess to which degree humans can recognize these concepts? It appears that humans struggle significantly with the task, as indicated by the results in the last paragraph of Sec. 5. Here, humans reach 65% accuracy, whereas a random agent would already reach 30%. How do humans respond when not prompted to classify but name the rule in natural language?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnIdmYqH6E", "reviewer": "Reviewer_pAAo", "summary": "The submitted work discusses the Abstract Visual Reasoning (AVR) capabilities of Multimodal Large Language Models (MLLM) with a focus on Bongard Problems (BP). To explore the gap between AVR capabilities on synthetic and real-world images, they introduce Bongard-RWR. This dataset aims to be more comparable to synthetic BPs than the state-of-the-art (Bongard-HOI and Bongard-OpenWorld) by consisting of real-world images that mirror the concepts in synthetic BPs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> … some easy adjustments to the dataset …\n\nThank you for these excellent suggestions regarding the dataset improvement. One of the key motivations behind Bongard-RWR is to test how models handle real-world complexity, which naturally includes variations in aspect ratio, resolution, and layout. Unlike synthetic BPs, real-world images are not uniformly structured and our dataset reflects this challenge. For instance, unifying image aspect ratios would require forceful rescaling, potentially introducing distortions or loss of content, which we aimed to avoid. Similarly, we refrained from cropping images, as doing so could obscure crucial visual features necessary for identifying abstract concepts. When arranging BP images into a joint matrix, we optimized the layout to minimize empty space between images.\n\nIn the Descriptive, Descriptive-iterative, Contrastive and Contrastive-iterative strategies, images are processed in their original resolution. In these cases, differences in resolution do not pose an issue, as the tested models are naturally capable of processing images of varying size. Also, differences in separators are not relevant, as the separator is not required in these settings. In the remaining strategies, we acknowledge that constructing joint BP images introduces some structural differences between Bongard-RWR and synthetic BPs. We believe, however, that variable aspect ratios, resolutions and picture layouts better align with the intention of reflecting the real-world variability in Bongard-RWR instances. \n\nAt the same time, we fully agree that a rigorous, in terms of image formatting, comparison of synthetic and real-world instances could also lead to meaningful conclusions from a slightly different perspective. Toward this end, following the Reviewer’s suggestions, we constructed 3 new dataset variants featuring square images (RWR-S), greyscale images (RWR-G), and square greyscale images (RWR-SG). In square image variants, we manually cropped all images to square format and rescaled to 512x512 resolution. The full BP matrix was resized to 1024px width, maintaining aspect ratio, leading to 763px height.\n\nWe evaluated the 4 open-access models from the main paper and 3 additional large open-access models introduced during the rebuttal (InternVL-2.5 78B, Qwen-VL 72B, and LLaVA-NeXT 110B) in 3 binary classification setups from Section 3.3. In Ground-truth and Incorrect label settings, all models performed similarly on RWR and the 3 new variants. In Images to Sides, however, model performance differed:\n* RWR-G: Most models solved slightly more problems compared to RWR, e.g. for Phi 3.5V (+15), Qwen2-VL 72B (+5), and LLaVA-NeXT 110B (+5). We hypothesize that since synthetic BPs are black-and-white, removing color in their real-world counterparts in RWR reduces unnecessary variation, making the task easier.\n* RWR-S: The difference was even more pronounced, e.g. for InternVL2 8B (+13), InternVL2.5 78B (+9), Pixtral 12B (+9), LLaVA-NeXT 110B (+9), and Phi 3.5V (+7). We think that cropping increased the signal-to-noise ratio, by eliminating empty space and irrelevant background details.\n* RWR-SG: Results aligned with the above trends, e.g. for InternVL2 8B (+11), InternVL2.5 78B (+8), Pixtral 12B (+7), LLaVA-NeXT 110B (+8), and Phi 3.5V (+17).\n\nWe will include the above results in the paper’s appendix and will release these new dataset variants alongside the main dataset. Once again, thank you for this excellent comment.\n\n> The discrepancy between the dataset sizes …\n\nThank you for raising this point. We reported the results in raw counts to simplify their interpretation for Bongard-RWR, but we acknowledge that percentages may simplify comparisons across datasets. We will switch to percentages in the revised version. \n\n> … confuse the reader in Figure 4 …\n\nWe agree that differences in dataset sizes may affect the clarity of Figure 4. Using percentages would allow the random baseline to be consistently placed at 50%. We will revise the paper accordingly.\n\n> … how the 100 problems of the larger datasets have been selected …\n\nThe selection of 100 problems from the larger datasets was driven by the need to balance diversity and computational feasibility. To ensure high instance diversity, we performed stratified sampling based on problem concept labels, selecting instances from the test splits on Bongard HOI and Bongard-OpenWorld. We will add the above clarification to the main paper in Section 5. Furthermore, as detailed in Section 3.2, our evaluation relies on an MLLM voting committee with 4 proprietary models, making inference costs scale with the number of datasets, problem instances, models, and generation strategies. Given these constraints, running multiple experiments on different random subsets was not feasible within our research budget.", "claims_and_evidence": "The paper claims to examine MLLM AVR limitations by introducing a focused real-world dataset Bongard-RWR. The intention is to facilitate direct comparisons between the synthetic and real-world domains, but, with respect to the comments below, I believe this claim is not supported sufficiently.", "methods_and_evaluation_criteria": "The presented Bongard-RWR dataset takes an interesting step in exploring the gap between reasoning on synthetic and real-world BPs. The generation of the Bongard-RWR seems plausible and justified, given the goals of the work. Nevertheless, considering the presentation in Figure 1 and the referenced github, it seems to me that the similarity between Bongard-RWR and synthetic examples could be substantially improved. For once, the real-world images do not follow the same formatting as the synthetic ones, i.e., they (i) have arbitrary aspect ratios, (ii) are not placed in the same grid, (iii) are not separated as clearly by a divider (left to right) as synthetic equivalents and (iv) do not have the same resolution. Additionally, including grayscale variants of Bongard-RWR problems in the dataset would have been appropriate to explore the gap between the synthetic and real-world problems.", "theoretical_claims": "There are no proofs or theoretical claims.", "experimental_designs_or_analyses": "The experimental design aims to show how the performance of proprietary and open MLLMs differs between synthetic BPs, state-of-the-art real-world BPs, and the proposed new dataset Bongard-RWR. The authors describe the experimental design clearly, with a granular differentiation between five modes of answer generation. With regard to the presented results, the analyses could be improved. For example, Table 1 shows the numbers of correct answers given by the MLLMs depending on dataset and answer generation strategy. However, the number of samples taken from each dataset differs (100 for prior art datasets, 60 for Bongard-RWR). This makes it less obvious to the reader to assess the relative performance difference. For example, the table shows GPT-4 Turbo is correct in 5 Bongard-HOI problems with Contrastive answer generation and in 5 Bongard-RWR problems with Descriptive answer generation. However, given the different number of overall samples, the former is correct in 5% of the cases and the latter in 0.083%. It did furthermore not become clear to me how the 100 problems of the larger datasets have been selected and why this experiment was then not repeated on different random subsets of these datasets.", "relation_to_broader_scientific_literature": "The paper is related to prior art in two ways. First, BPs have been discussed throughout the literature as benchmark problems for visual reasoning, and this work presents an evaluation on a novel BP dataset. Second, the gap between visual reasoning capabilities on synthetic and real-world datasets is an important topic throughout the computer vision literature, including real-world variants of BPs such as the referenced Bongard-HOI and Bongard-OpenWorld datasets. In a broader context, the paper relates to the introduction of MLLMs and adds to our understanding of their visual reasoning capabilities.", "essential_references_not_discussed": "Essential references have been discussed.", "other_strengths_and_weaknesses": "Strengths:\n- I very much like the idea and direction this work takes, aiming to narrow down the differences between benchmarks to allow for improving our understanding of the points at which the visual reasoning capabilities of state-of-the-art models fail.\n- The paper is written clearly and very easy to follow. \n- Figures 1, 2, and 3 communicate the dataset and evaluation strategies well to the reader.\n- The Appendix has many interesting insights and discussions, some of which may be worth considering placing in the main paper instead to make them more visible.\n\nWeaknesses:\n- As discussed above, some easy adjustments to the dataset can further close the gap between Bongard-RWR and the synthetic dataset. Since this is the paper's primary goal, I think it is critical to address at least some of these points before publication.\n- The discrepancy between the dataset sizes is a bit problematic concerning clarity. For example, while the sizes are mentioned in the caption, its not clear to the reader what the actual relative performance differences are. As a reader, with the aim of this paper in mind, I would like to be able to look at Table 1 and directly compare entries between Synthetic and RWR results.\n- Similar to the point above, the discrepancies in dataset sizes may confuse the reader in Figure 4 regarding the dashed line reporting a random baseline.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "I have no questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BK7QiuZ0Qh", "reviewer": "Reviewer_2kYM", "summary": "This work evaluates vlms’ reasoning capabilities using Bongard problems. In addition to existing datasets, the authors constructed a new dataset that represents synthetic BP concepts using real-world images. Results show that some vlms can perform well if the task is simplified to binary classification, but the experiments under classical BP settings reveal vlms generally lack AVR capabilities.\n\n## update after rebuttal\nThe rebuttal partly solves my concerns. I will maintain my evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> … feed multiple (sub-)images from group A or B all at once …\n\nThank you for this valuable suggestion. We explored this approach in preliminary experiments but the results were not promising. Feeding in all images separately significantly increases the input context length, often exceeding the limit supported by the employed MLLMs. Even when models could process longer image sequences, their performance remained weak in this setup. We suspect this is due to the scarcity of multi-image training of MLLMs. \n\n> The sample sizes are relatively small …\n\nBongard-RWR was designed with significant manual effort, as detailed in Section 4.1. Of 100 synthetic BPs considered, only 12 were automatically represented in the real-world domain using Algorithm 1 (Appendix F). The remaining problems required manual intervention, including translating abstract concepts to the real-world domain, selecting appropriate images, or photographing manually constructed scenes. This ensures that the dataset captures fine-grained abstract concepts, such as “Ends of the curve are parallel vs. Ends of the curve are perpendicular” or “Extensions of segments cross at one point vs. Extensions of segments do not cross at one point”, which are challenging to represent with real-world images. The fine-grained nature of the synthetic BPs made even a manual translation problematic in certain cases. For instance, in problems #77 (Angle divided in half) or #99 (Outlines made by triangles and circles intersect), we could not come up with a proper implementation of these specific concepts in the real-world domain. For such instances we abandoned their “imperfect’” representation, to preserve the indisputability of translated concepts in the Bongard-RWR dataset.\n\nIn contrast, Bongard HOI and Bongard-OpenWorld datasets, which were generated automatically using existing data sources, focus on coarse-grained concepts such as “A person jumping on a surfboard. vs. Not a person jumping on a surfboard”. While this automated process enables larger sample sizes, it limits the representation of fine-grained abstract concepts. \n\n\n> … other strong open models …\n\nIn Appendix D, we analyse the impact of scaling model size on the model’s performance, evaluating larger InternVL2 variants (up to 76B params) and LLaVa-NeXT (up to 110B params). Figure 9 shows that proprietary models still consistently outperform open-access ones. \n\nPrompted by this feedback, during the rebuttal phase, we evaluated 3 recent open-access MLLMs: InternVL-2.5 78B (AA), Qwen2-VL 72B (BB), and LLaVA-NeXT 110B (CC), on all 4 considered datasets using 3 binary classification strategies described in Section 3.3. The models were selected based on the integration readiness within our evaluation framework (LLaMa 3.2 vision is unavailable in our geographic location). \n* Ground-truth and Incorrect Label\n  * AA improved significantly over InternVL2 8B, which was biased towards rejecting the tested concept (Figure 4, top row). The newer and larger variant achieves competitive performance to proprietary models (69/82, 83/99, 78/100, 17/55 with GT/IL for Synthetic, HOI, OpenWorld, and RWR, resp.).\n  * BB slightly underperformed AA on the first 3 datasets but excelled on RWR, scoring 36/50, and was the leader in terms of the total score (86 vs. 81 for GPT-4o, the 2nd best model).\n  * CC showed decent but lower performance compared to AA and BB.\n* Images to Sides\n  * BB performed best across the first 3 datasets, being on-par with proprietary models on HOI and OpenWorld, solving 96 problems in both cases.\n  * On RWR, CC performed best, solving 24 problems, on par with Claude 3.5 Sonnet, the best-performing proprietary model on this dataset.\n\nWe shall add the above scaling results to Appendix D.\n\n> What does Bongard Problems have to offer more than existing AVR benchmarks?\n\nThe motivation for using Bongard Problems is outlined in Section 1, where we highlight their unique challenge: solving BPs requires identifying an underlying concept expressed through images and articulate it in natural language. The requirement of expressing the answer in natural language forms a fundamental distinction from other AVR tasks. Moreover, the limited number of samples in BPs forces a few-shot learning setup, as opposed to typically large training datasets used for other AVR tasks (e.g., the most commonly considered Raven’s Progressive Matrices). Additionally, in the “AVR tasks” paragraph in Section 2, we discuss how BPs compare to existing AVR benchmarks, emphasizing that BP datasets considered in our work besides presenting a few-shot learning challenge, also cover both synthetic and real-world images, and enable both binary classification and open-ended answer generation. Appendix A extends this discussion, outlining future research directions for extending our research to other AVR tasks.", "claims_and_evidence": "Most claims made are well-supported by experimental results, while it can be improved by adding more results of strong open models and using larger sample sizes.", "methods_and_evaluation_criteria": "The evaluation procedures and metrics are well designed, especially the various prompting strategies are insightful.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- It’s worth trying to feed multiple (sub-)images from group A or B all at once to vlms, which may help the models have a global understanding.\n- This work uses 100 samples each from existing datasets and 60 from the constructed Bongard-RWR dataset. The sample sizes are relatively small, and thus the results may not be very stable.", "relation_to_broader_scientific_literature": "This work can augment existing efforts to benchmark the reasoning capabilities of vlms", "essential_references_not_discussed": "This work can benefit from adding more discussion (and results) of other strong open models, including but not limited to QVQ, llama-3.2-vision, llava-onevision, etc.", "other_strengths_and_weaknesses": "- This paper is well-written and easy to follow\n- The human study is interesting, setting a valuable baseline for vlms’ reasoning capabilities evaluated by this task", "other_comments_or_suggestions": "Without comparing with the more capable open models, it’s hard to compare the difference between proprietary and open models. Some large open models can be used by api calling and should not be as expensive as proprietary models. It would be interesting to see how those strong open models perform on this task.", "questions_for_authors": "What does Bongard Problems have to offer more than existing AVR benchmarks? This is also my main concern of this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yrrqYxmna", "reviewer": "Reviewer_EWdU", "summary": "This paper proposes a novel benchmark for multi-modal large language models based on Bongard problems.  In addition to standard Bongard problems the paper proposes \"real-world\" variants of such problems, where abstract images are replaced with real-world images. Furthermore, the paper proposes simpler versions of the Bongard problems where the model needs to classify correct/wrong answers instead of actually providing the answer. The paper evaluates 4 proprietary and 4 openaccess models on the proposed benchmark.\n\n## update after rebuttal\n\nWhile I am not against acceptance, I am still of the opinion that the size of the benchmark is way too small and prone to noise.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> … the novelty … is not clear.\n\nIn preliminary experiments, we observed that while MLLMs perform well on HOI and OpenWorld, they struggle with Synthetic BPs. We identified two key factors distinguishing Synthetic from HOI and OpenWorld. Firstly, Synthetic feature simple 2D geometric shapes, while HOI and OpenWorld consist of real-world images. A potential explanation might be that the lack of training data on synthetic images could contribute to the lower performance on Synthetic BPs. Secondly, Synthetic focus on abstract concepts (e.g., “One line vs. Two lines”), while HOI and OpenWorld involve concepts grounded in the real world (e.g., “A person jumping on a surfboard vs. Not a person jumping on a surfboard”). This led us to hypothesize that MLLMs may find abstract concepts harder to recognize.\n\nThis motivated us to create Bongard-RWR, which bridges the gap between these two domains by representing abstract concepts with real-world images. Apparently, RWR presents a major challenge to contemporary MLLMs, despite relying on real-world images. This indicates that abstract concepts are inherently difficult for models to recognize regardless of their representation. We hope this summary clarifies the novelty of Bongard-RWR over the existing benchmarks.\n\n> … relation to prior work …\n\nWe discuss the motivation for Bongard-RWR in Section 1 and in Section 2 position our work within AVR tasks and approaches to solve BPs. We further elaborate on recent trends and future directions in Appendix A. If the Reviewer finds any aspect of this discussion missing, we would greatly appreciate specific pointers to improve the paper accordingly.\n\n> … the limited size of the benchmark.\n\nPlease, see our response to Reviewer 2kYM (*“The sample sizes are relatively small …”*).\n\n> Are there any structural differences …\n\nFigure 22 summarizes per model Bongard-RWR results. Notably, 38 problems remain unsolved by any model, while 8 problems were solved by at least 3 models: #2, #7, #10, #23, #32, #57, #88, and #100. These BPs encompass the following concepts: Large vs. Small figures, Figures elongated vertically vs. horizontally, Triangles vs. Quadrangles, One vs. Two figures, A sharp vs. No sharp projection, Identical vs. Not identical figures, Three vs. Five parts, and The letter A vs. B. Analysis of specific models reveals that Claude 3.5 Sonnet correctly solved 3 tasks concerning the distinction between triangles and other shapes (#10, #97, #98), suggesting that concepts involving object shapes are better recognized by the model. Other models do not present clear result patterns, and generally struggle with concept abstraction in BPs.\n\n> In Figure 4, …\n\nIn the human study, participants were asked to express the rule in natural language. Their responses were then evaluated by expert annotators, who compared them to ground-truth labels and classified as correct / incorrect. This final binary assessment was used to compute accuracy. Since the human evaluation setup differs from the classification task in Figure 4, a direct comparison may be misleading. Instead, we will add human performance in the right-most section of Table 1, which concerns Bongard-RWR scores.\n\n> … clarify the input resolution …\n\nSingle images were provided to the models at their original resolution. Whole problem matrices were constructed using the original images and then resized such that the longer edge (height or width) did not exceed 1024px, while keeping the aspect ratio fixed.\n\n> … some SOTA models are missing …\n\nPlease see our response to Reviewer 2kYM (*“other strong open models”*).\n\n> … analyze the effect of the prompt …\n\nWe manually optimized prompts based on preliminary results. The final prompts, shown in Appendix I, reflect these optimizations. For instance, this process led us to add to Prompt 6 “The description of the rule should be simple and concise.” along with two examples, adjusting the order of JSON fields in Prompt 7 (generate an explanation and concept before the binary answer), and adding guiding sentences in Prompt 10 (Descriptive strategy).\n\nIn addition, we manually reviewed selected model responses and found that models generally followed the task correctly, except for rare cases where they explicitly stated an inability to detect a concept or classified test images as belonging to neither side in the Images to Sides strategy. Such refusals indicate that models correctly interpreted the instructions but lacked the necessary visual reasoning capabilities.\n\n> …  qualitative examples in the main paper, …\n\nDue to space constraints, we could not fit qualitative examples in the main paper. However, Appendix H and Figures 23 – 26 provide qualitative examples that illustrate the challenges of Bongard-RWR, contrast it with synthetic BPs, and showcase specific problem instances alongside model responses.", "claims_and_evidence": "*The paper adds to the growing number of visual reasoning benchmarks on Bongard problems, e.g. \"Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World, ICLR 2024\", \"Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022\", the novelty over these benchmarks is not clear.\n* The size of the benchmark is limited, there are only 100 different problems. The small size of the benchmark would introduce noise to the results.\n* The paper shows that state of the art proprietary and openaccess models do not perform well on this benchmark. Although note that these problems are inherently very challenging, with humans showing high variability in ability to solve these problems.\n* The paper should include an analysis of Bongard problems that can be solved by the state of the art multi-modal LLMs. Are there any structural differences between the problems that can solved and problems that remain unsolved by all models?", "methods_and_evaluation_criteria": "* In Figure 4, it would be interesting to include human performance, especially in the real-world image case, to quantify the difficulty level of the dataset.\n* Evaluation strategy not clear: The paper should clarify the input resolution used both in case of single input images and the whole problem matrix.\n* Some SOTA models are missing in the evaluation, e.g., Qwen-VL (https://github.com/QwenLM/Qwen-VL).", "theoretical_claims": "None made.", "experimental_designs_or_analyses": "* While the paper includes extensive experiments, some SOTA models are missing in the evaluation, e.g., Qwen-VL (https://github.com/QwenLM/Qwen-VL).\n* Due to the challenging nature of the benchmark, it would be helpful to analyze the effect of the prompt in more detail. Otherwise, the results  would conflate the effect of instruction following performance and visual reasoning performance.", "relation_to_broader_scientific_literature": "* The paper adds to the growing number of visual reasoning benchmarks on Bongard problems, e.g. \"Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World, ICLR 2024\", \"Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022\", the novelty over these benchmarks is not clear.", "essential_references_not_discussed": "While \"Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, ICLR 2024\", \"Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022\" are cited, the novelty over these benchmarks should be discussed in more detail.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "* The paper should discuss the relation to prior work on Bongard problems in more detail.\n* The paper should discuss the effect of the limited size of the benchmark.\n* The paper should discuss qualitative examples in the main paper, that better highlight the difficultly level of the proposed problems.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVkOe7NuCu", "reviewer": "Reviewer_Gh92", "summary": "- The paper proposes an evaluation benchmark for assessing the visual abstraction capabilities of large multimodal models by using real-world images to mimic Bongard problems (originally formulated with synthetic images), to close the real-synth gap that can hinder evaluation. \n- The analysis uses multiple evaluation setups (e.g. where the model sees directly the Bongard image matrix and needs to output the hidden concept, or the model first captions each image independently and then provides the answer, or evaluates contrastively pairs from left-right sides etc). \n- The task is formulated either as binary classification (where the model is given a concept and needs to predict if it is correct or not), or open-ended language answer (where a majority-vote by multiple LLMs is used to score the results). \n- 4 proprietary and 4 open-sourced models are evaluated. \n- The analysis includes examples from existing Bongard sets (synthetic images, Bongard-OpenWorld, Bongard-HOI) and the newly proposed set (60 tasks created some automatically, some manually, even by photographing object configurations). \n- A human baseline is also collected for the proposed set.\n- The results show that there is still a significant gap between humans and models.\n\n## Update after rebuttal ##\nI thank the authors for the detailed rebuttal.\nI maintain my concerns related to the fairly low performance of the human baseline which might point to unwanted ambiguity, which, coupled with the reduced scale of the dataset, might make it hard to pinpoint causes of model failures. \nThe multi-class setup is a welcome addition, to not rely exclusively on LLMs for evaluation (LLMs evolve fast, which might make it difficult to compare models over time) -- but this needs to be properly introduced and reviewed. \nI am inclined to maintain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we are grateful for your insightful comments and suggestions, which helped us to improve the paper. \n\n> The first claimed contribution is confusing: …\n\nThank you for this comment. We will rephrase this statement. Prior work on BP benchmarks has primarily focused on pre-trained image encoders with few-shot classifiers, rather than comprehensively evaluating MLLMs. While Wu et al. (2024) did assess GPT-4V on Bongard-OpenWorld, they did not compare it to other MLLMs. Furthermore, prior work has primarily framed BPs as a binary classification task. Wu et al. (2024) consider an explicit concept generation setup in their appendix, but use standard NLP metrics for evaluation, making it difficult to assess model accuracy in solving BPs. In contrast, we systematically evaluate a set of proprietary and open-access MLLMs in both binary classification and open-ended generation settings across 4 datasets comprising both synthetic and real-world images.\n\n> … negative pairings …\n\nFor the negative pairings in the binary classification setup, we assigned each BP instance a negative concept from a different BP. Specifically, for a given BP instance $i$, we selected the concept from BP $(i + 20) mod 100$. We manually reviewed the resulting pairings to confirm that the selected negative concept was distinct from the original concept. Thank you for bringing this point to our attention. We shall include the above description in the revised version of the paper.\n\n> For the open-language setup, …\n\nGiven the scale of our experiments (the number of datasets, problem instances, models, and generation strategies), manual assessment of every response is infeasible. Hence, in Section 3.2 we proposed an automatic assessment strategy. To improve its robustness, we carefully selected the best-performing evaluation models based on a manual review of selected evaluations and tuned the evaluation prompt to align results with human assessments (Appendix E). While open-ended answer evaluation remains an open problem in NLP research, Tables 3 and 4 in Appendix E show high consistency between our final evaluation pipeline and scores of human experts.\n\nFurthermore, in the evaluation process we did not rely on a single model judgement, but instead evaluated the answers in the mixture-of-experts manner, with the voting scheme of 4 MLLMs manually optimized in preliminary experiments.\n\n> … formulated as multiple-choice …\n\nPlease, see our response to Reviewer Aobr (the last paragraph of *“… concurrent work, …”*).\n\n> … examples where the models and humans disagree …\n\nResponses from human participants were evaluated by expert annotators, who compared them to ground-truth labels and classified them as correct or incorrect. This final binary assessment was used to compute the average human accuracy. No manual corrections were made to their answers.\n\nIn Appendix E.2 (Tables 3  and 4) we compare the evaluations of the proposed MLLM ensemble with human experts. The voting approach achieves high consistency with human experts, scoring 99% and 90% for synthetic BPs and Bongard-OpenWorld, resp. Figures 11 and 12 highlight cases where the evaluations differed. High agreement rates suggest that an automated evaluation is sufficiently robust and well-suited for the scale of experiments conducted in our study.\n\n> The human baseline …\n\nPlease, see our response to Reviewer NTSJ (*“Another concerning factor is that human performance …”*).\n\n> … human participants were given all examples …\n\nParticipants were given all 60 examples in a document and asked to submit their responses via a form. They had full control over the order in which they solved the problems and could allocate as much time as needed for each question. Figure 6 (top-right) shows the amount of time spent by participants, as reported in the closing survey. On average, respondents spent between 1 and 2 hours to complete the challenge. Since the participants were free to choose the order of problem-solving, we do not have sufficient data to systematically analyze the influence of fatigue or boredom on performance. However, this flexibility likely helped mitigate such effects, as participants could distribute their effort based on individual preferences.\n\n> Other works relevant …\n\nThank you for pointing out these relevant works. KiVA focuses on visual analogical reasoning inspired by developmental psychology, evaluating how models generalize visual transformations. Perception Test assesses multimodal video understanding across various reasoning tasks. In contrast, our work investigates abstract visual reasoning through Bongard Problems, spanning both synthetic and real-world domains. Our setting differs from both KiVA and Perception Test by focusing on contextual visual analogies rather than perceptual transformations or video understanding. These works assess MLLM reasoning capabilities from different angles and we view them as complementary. We will reference both papers in Section 2.", "claims_and_evidence": "Strengths: \n- The exercise of \"translating\" synthetic Bongard problems into real world images is very interesting and creative.\n- The paper includes 4 proprietary and 4 open-sourced models. This is a good coverage of the model landscape.\n- The different evaluation strategies (description, contrastive etc) are valuable.\n\nWeakness:\n- The first claimed contribution is confusing: “for the first time in the literature, we consider BPs in the context of MLLMs and propose…” – for analysis, the paper compiles tasks from existing benchmarks (BP-OpenWorld, BP-HOI) and adds a new set, so not sure why the proposed work is the first work to consider this set of problems.", "methods_and_evaluation_criteria": "The authors propose binary classification and open-ended language evaluation using LLMs. \n\nOn the binary classification task, the models are much better at recognising the negative pairings between images and concepts, than the positive ones. This could point to failure in generating realistic negative pairings. I didn’t see details on how the negative pairings were generated (but I may have missed them too, the supplementary has some 25 pages). \n\nFor the open-language setup, I have strong concerns about using LLMs to evaluate correct answers (even by a majority voting). Given that in general LLMs struggle with nuanced descriptions of spatial relationships between objects or subtle differences, I do not consider this a robust evaluation strategy. The authors include in the appendix examples where the models and humans disagree on some answers. For the human baseline, the answers were corrected manually. Does this mean that the LLM evaluation is not robust enough? \nWouldn’t it be possible to frame the problem as multiple-choice question answering by (manually) creating negative options that are realistic and challenging?\n\nThe human baseline is not very robust in my opinion. 30 participants were involved, mainly with academic background (Master, PhD students) – this is not representative of the human population. Even more concerning to me, even these quite highly educated participants obtained only 65% on the benchmark, with the lowest scoring about 38%. Given that these are adult individuals capable of carrying out their daily tasks puts into question the validity of the examples included in the benchmark (I myself have been confused by some examples shown in the appendix, eg 14a), and the predictive power of this benchmark (i.e. if a model performs well on this benchmark, will it perform well in real life tasks and the other way around).\n\nIn the appendix, it is mentioned that human participants were given all examples in a document and asked to submit a form? How many examples did each participant have to answer? All 60 in one session? Is there a way to check if fatigue/boredom affected their performance, e.g. if answers towards the end were more likely to be incorrect?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See Methods and evaluation above.", "relation_to_broader_scientific_literature": "Other works relevant for evaluating capabilities at the intersection between perception and cognition: KiVA (ICLR2025) https://arxiv.org/abs/2407.17773 in the image domain, Perception Test in the video domain (NeurIPS2023).", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is clearly written, I didn’t see typos.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How were the negative pairings created for the binary classification setup?\n2. Could the language task be formulated as multiple-choice with carefully collected realistic and challenging negative options? this would significantly reduce the ambiguity inherent in complex real world images.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work deals with the challenging problem of evaluating the abstract visual reasoning capabilities of multimodal large language models (MLLMs). There has been a recent interest in the Bongard problems and how these can still stump the so-called advanced visual models, mostly due to a lack of reasoning. This work is another step towards the direction of showing that there exist inherent limitations in MLLMs that make them fail in abstract reasoning tasks.

Overall, the paper generated a good discussion among the authors and the reviewers, and most reviewers leaned towards (weakly) accepting the paper at the end. The AC wants the authors to closely look at the remarks of Reviewer Gh92 while preparing the camera-ready version of the paper, especially the following points:

1. The work demonstrates relatively poor human baseline performance, which might point to unwanted ambiguity in the dataset. Potential ambiguity coupled with reduced dataset scale might make it hard to draw useful conclusions when ranking different models or in trying to understand model failures to guide future improvement.

2. The real efficacy of the multi-class classification setup since not many details were provided in the rebuttal regarding the same, as well as no human baseline results were provided for this setup.

I recommend weak acceptance for this work.
Decision: Accept (poster)