Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Authors: Antonia Wüst
Tim Tobiasch
Lukas Helff
Inga Ibs
Wolfgang Stammer
Devendra Singh Dhami
Constantin A. Rothkopf
Kristian Kersting
Venue: ICML 2025 poster (2025)
Keywords: Visual Reasoning
Vision Language Models
Bongard problems
Primary area: deep_learning->foundation_models
TL;DR: Diagnostic evaluation of VLMs on Bongard Problems that reveals problems in perception and reasoning.

Abstract: Recently, newly developed Vision-Language Models (VLMs), such as OpenAI's o1, have emerged, seemingly demonstrating advanced reasoning capabilities across text and image modalities. However, the depth of these advances in language-guided perception and abstract reasoning remains underexplored, and it is unclear whether these models can truly live up to their ambitious promises. To assess the progress and identify shortcomings, we enter the wonderland of Bongard problems, a set of classic visual reasoning puzzles that require human-like abilities of pattern recognition and abstract reasoning. With our extensive evaluation setup, we show that while VLMs occasionally succeed in identifying discriminative concepts and solving some of the problems, they frequently falter. Surprisingly, even elementary concepts that may seem trivial to humans, such as simple spirals, pose significant challenges. Moreover, when explicitly asked to recognize ground truth concepts, they continue to falter, suggesting not only a lack of understanding of these elementary visual concepts but also an inability to generalize to unseen concepts. We compare the results of VLMs to human performance and observe that a significant gap remains between human visual reasoning capabilities and machine cognition.

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Antonia Wüst! Tim Woydt
Lukas Helff Inga lbs Wolfgang Stammer
Devendra S. Dhami
Constantin A. Rothkopf
Kristian Kersting 1246
Abstract
Recently, newly developed Vision-Language Models (VLMs), such as OpenAl's o1, have emerged, seemingly demonstrating advanced rea- soning capabilities across text and image modal- ities. However, the depth of these advances in singuage-guidee perception and abstract rea- soning remains underexplored, and it is unclear whether these models can truly live up to their ambitious promises. To assess the progress and identify shortcomings, we enter the wonderland of Bongard problems, a set of classic visual rea- soning puzzles that require human-like abilities of pattern recognition and abstract reasoning. With extensive evaluation setup, show that while VL.Ms occasionally succeed in identifying dis- criminative concepts and solving some of the problems, they frequently falter. Surprisingly, even elementary concepts that may seem trivial to humans, such as simple spirals, pose significant challenges. Moreover, when explicitly asked recognize ground truth concepts, they continue to falter, suggesting not only a lack of understanding of these elementary visual concepts but also an inability to generalize to unseen concepts. We compare the results VLMs to human perfor- mance and observe that significant gap remains between human visual reasoning capabilities and machine cognition."
Figure 1: The wonderland of Bongard problems. The challenging puzzles circle around simple black-and-whit diagrams that picture geometric shapes, patterns, or lines arranged in various ways. Despite their simple appearance, the underlying concepts be abstract and complex.
1. Introduction
Visual reasoning, the ability to understand, interpret, and reason about the visual world, is fundamental aspect of human intelligence (Marr, 2010). It allows us to navigate our environment, interact with objects, and make sense of complex visual scenes. In recent years, the field of artificial intelligence (AI) has advanced rapidly toward replicating as- pects of this visual reasoning, with significant focus placed on Vision-L Language Models (VLMs) (Bordes et al., 2024; Liu et al., 2023; 2024a). These models integrate visual and textual information to generate descriptive content, aiming to mimic how humans comprehend and reason about the physical world. Because of their human-like responses, VLMs often create the illusion of possessing human-like perception and intelligence. However, as recent work shows, VL.MS and the Large Language Models (LLM) on which they are based have dramatic shortcomings in the case of reasoning (Nezhurina et al., 2024) and visual perception (Kamath et al., 2023: Rahmanzadehgervi al., 2024; Gei- gle al., 2024; Gou et al., 2024) or their combination (Zhou et al., 2023; Zhang et al., 2024; Wang et al., 2024b).
"IML Lab, TU Darmstadt "Hessian Center for Al (hessian.ai 'Iustitute of Psychology. TU Darmstadt *Centre for Cognitive Science, TU Darmstadt Uncertainty Al Group, TU Eindhoven 'German Center for AI (DFKI). Correspondence to: Antonia Wüist <antonia wuest@es. weteees.ttudeee co-darmstddr.de>>
Proceedings of the 4rnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Bongard problems (BPs), a class of visual puzzles that re- quire identifying underlying rules based on a limited set of images, provide unique and challenging benchmark for as- sessing visual reasoning abilities Al (Bongard & Hawkins,
Our code is available sttpa://githin com/ ml www.reeeerehbonggee.eeeee ieeeaard--i---eeeeeeeee -n-d--derwoldddd

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
1970). Conceived by Mikhail Bongard in 1970, these visual puzzles test cognitive abilities in pattern recognition and abstract reasoning, posing formidable challenge even to advanced AI systems Gremmnndez-Orallo et al., 2016).
the solutions formulated by Bongard himself. Raghuraman et al. (2023) explore BPs in both classical and real-world formats but shift from open-ended folmllarmul task to a classification approach BPs with a reinforcement learning setting for ex- tracting meaningful representations and counterfactual ex- planations. However, despite these efforts, BPs remain an unsolved challenge.
approach. Youssef et al. (2022)
A BP consists of twelve diagrams, divided into two sides. For each side, a distinct and specific conceptual theme must be identified, which clearly differentiates it from the other side. Although the diagrams themselves are visually simple (see Figure 1), the underlying concepts that connect the im- ages within each group can be abstract, such as more filled objects than outlined objects or turning direction of spiral shape. Thus, unlike pattern recognition in classification tasks, BPs are not about finding visual patterns in a sin- gle diagram that match a certain concept but about finding concepts that allow for the description of a set of diagrams.
Benchmarks for VLMs. Benchmarks specifically designed for VLMS usually involve complex tasks such as image captioning, scene or diagram understanding, visual ques- tion answering (VQA), or visual-commonsense reasoning (Masry et al., 2022; Yue et al., 2024; Rabmanzadehgervi et al., 2024; Kamath et al., 2023; Antol et al., 2015; Johnson et al., 2017: Hong et al., 2021; Zellers et al., 2019; Hudson & Manning, 2019; Qiao et al., 2024; Liu et al., 2024b). Yet, most of these only require simple reasoning abilities. More recent benchmarks have been introduced to probe advanced reasoning skills, e.g., logical learning (Helff et al., 2025; Vedantam et al., 2021; Xiao et al., 2024), mathematical reasoning (Lu et al., 2024) or analogy-based visual reason- ing (Chollet, 2019; Moskvichev et al., 2023; Zhang et al., 2019). Malkiński et al., 2025 considered BPs for evaluat- ing VLMs, concentrating on open-ended and classification- based settings. Our work shares their open-ended focus but goes further by adding two additional tasks to examine spe- cific abilities we hypothesize are vital for solving Bongard problems. While Malkiáski et al., 2025 compare synthetic Bongard problems to a newly proposed real-workd variant, we target the original Bongard set, pinpointing especially challenging cases and identifying concept-detection incon- sistencies. Generally, the shift towards more cognitively demanding tasks is promising, comprehensive diagnostic evaluations of VLMs' reasoning capabilities that pinpoint sources of error and model limitations remain scarce. Fur- thermore, the degree to which these models genuinely com- prehend complex, abstract visual concepts is yet to be fully investigated.
While traditional machine learning approaches have made some early progress on BPs (Raghuraman et al., 2023; De- peweg et al., 2024), the potential of VLMs remains largely unexplored. Given that VLMs already struggle with recog- nizing relatively simple visual patterns (Rahmanzadehgervi et al., 2024; Zhang et al., 2024), BPs present a particularly challenging task, offering valuable framework for investi- gating which patterns are easier or harder for ofte-offthheart models to identify. Recent work has explored BPs in the context of VLM evaluation (Malkiński et al., 2025). While their study provides meaningful insights, it does not analyze model behavior and failures in greater depth. This high- lights the need for a more comprehensive investigation into how VLMs process and reason about BPs.
In this work, we assess the performance of VLMs in solving Bongard problems (BPs) across multiple task settings. (1) We evaluate how effectively different VLMs can identify the underlying rules in BPs through both an open-ended problem-solving approach and a multiple-choice format. (2) We then compare their performance to the results of a newly conducted human study, offering insights into how VLMs measure up against human reasoning. (3) Next, we explore the models' pattern recognition abilities in greater detail by testing their capacity to classify images based on BP rules. (4) Finally, we examine their ability to generate correct hypotheses for the given problems. Our findings provide valuable insights into the perceptual madness of VLMs and suggest opportunities for improvement.
Open-Ender Visual Reasoning, Most existing benchmarks rely on classification tasks (Chollet, 2019; Helff et al., 2025; Vedantam et al., 2021). multiple-choice formats (Zhang et al., 2019; Lu et al., 2024), or synthetic environments with a limited set of operations (Moskvichev et al., 2023).
While these setups tackle pattern recognition and visual reasoning, they do not fully capture real-world complexity. Open-ended visual puzzles, such as BPs, present a greater challenge by requiring models to map visual input to text without predefined rules or language biases. This lack of structured guidance makes the task more demanding, better reflecting real-world reasoning.
2. Related Work
Bongard and ML. Depeweg et al. (2024) define a formal language to represent compositional visual concepts. Us- ing this language and Bayesian inference, concepts can be induced from the examples provided in each problem. For subset of 35 problems, there is reasonable agreement between the concepts with high posterior probability and
2

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Left side:
Right side:
Figures positioned right
Figures positioned left
A
O
I
BP48
What is the rule for the left side?
What is the rule for the right side?
01:
basic geometric/icons with no inversion or distortion x
inverted or morphed version of of the basic shapes
Claude 3.5. shapes either point upward or have no directional orientation x shapes are composed of only straight lines x
shapes either point downward or have horizontal symmetry X
Gemini 2.0:
shapes have at least one curved line x
Figure 2: VLMs struggle to solve BPs out of the box. The images of BP#8 are depicted, where the left side has images with shapes positioned right and the right side has images of shapes positioned on the left. Although the concepts left and right may seem trivial to human, the VLMs struggle to generate discriminative rules.
3. Bongard Problems and VLMs
setup. A model is given a text prompt alongside an image containing twelve black-and- tlnck-and-white diagrams arranged into left and right sides. The prompt first describes the image structure, including the number and arrangement of dia- grams. It then defines the task, explaining that each side follows a distinct rule that does not apply to the other. The model is instructed to analyze the diagrams step by step to infer these underlying rules. Finally, the required output format is specified as a dictionary with two entries, one for each rule. The expected response are two rules in natural language, one for the left side and one for the right side. The complete prompt can be found in Listing 1.
Bongard problems (BP), introduced in 1970 by Mikhail Bongard (Bongard& & Hawkins, 1970), are visual puzzles that test for capabilities of pattern recognition, concept for- mation, and abstraction. Each BP consists of twelve simple black-and-white diagrams divided into a left and a right group. While both sides may share similarities, each side has one distinet property or rule that is shared by all six images on that side but is not present in any image the opposite side. An example BP is shown in Figure 2 where the distinguishing properties are right and left position.
The task is the linguistic expression of the underlying rule that distinguishes the two groups. These rules vary in com- plexity, ranging from simple geometric properties like the presence of circle to more abstract relational concepts like symmetry or the presence of right angle. In some cases, the rule of the right side is just the negative of the left rule, like BP#24 (a circle present vs. no circle presenf). Still, for the majority of BPs the second rule is a more specific opposite of the first e.g., BP#6 (triangles vs. quadrangles).
Task 1.1: Multiple Choice Setting. In this setting, rather than having the model generate the rules itself, we provide it with a set of predefined rule pairs from the BP domain. The model is then tasked with selecting the correct rule pair from these options. The prompt follows the same struc- ture as in the previous setting but includes an additional list of available rules for the model to choose from. The expected answer is the ID corresponding to the correct rule- pair (cf. Listing 2).
In contrast to mainstream classification tasks, BPs differ in their complexity and reliance abstract reasoning rather than direct pattern recognition. Specifically, BPs test the ability to express distinctive and common features of images, including the pattern recognition necessary to correctly as- sociate the features with images, as well as the ability to come up with textual rules that can characterize the meta- pattern (not within each but) across all twelve diagrams that constitute BP.
Task 2:
Detect Specific Concepts. To specifically inves- tigate the limitations of visual descriptions we create an additional perception task. Here, the relevant concepts for the BP are provided as context (e.g., horizontal and vertical orientation). Based this, the task is to predict for every single image of the BP whether the left side or the right side concept is true.
To achieve this, we designed a tailored prompt for each BP that directly targets its ground truth concepts. For each of the 12 images in a BP, the prompt asks whether the concept from the left or right side of the problem applies to that specific image. To generate these prompts, we provide an LLM with example prompts for BP# 16 and BP455 (cf. Listing 4 and Listing 5) along with the corresponding ground truth rule?. Based on this input, the LLM automatically generates a custom prompt for each BP (cf. Listing 3).
This multi-modality of BPs makes them an interesting chal- lenge for multimodal Al such as VLMs. However, the specific nature of these puzzles raises questions concerning the best experimental setup for VLM evaluation. the following we therefore introduce several, different prompt strategies for VLMs solve BPs.
Task 1: Open-Ended Solving of Bongard Problems. In this setting, for each BP, model is prompted individually by providing a text prompt together with an image show- ing all twelve diagrams. The prompt follows the following
Ptttps: htapp:://www... foundal is cww.co/////ppppeeeeeeeeeeee bongard_probe www.rolmmmmm....m ems solut ut oons...... hhtm
3

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Open Question
Detect Concept
Ideally, one would expect, that if a model is able solve Task 1, it would also succeed in the other tasks, leading to only Type IV subsets. If there occur subsets of the other types, this raises questions regarding the robustness and reasoning capabilities of the models. This will be evaluated in the following within the course of our evaluation.
Task
Type
Task 2
Type
Type
Type
Task
Formulate Hypotheses
4. Experimental Evaluation
Our experimental evaluations investigate to what extent state-of-the-an VLMS can solve Bongard problems. For this, we first evaluate the models quantitatively on all 100 puzzles based on the settings of Task 1 and Task 1.1. We then compare their performance against humans and investigate them qualitatively in more detail based on the settings of Task 2 and Task 3. We hereby address the following research questions:
Figure 3: Overview our three different tasks and how model behaviour across them can intersect. The subsets define different types of model behaviours.
Task 3: Formulate Hypotheses. To solve BP, identifying the correct rule is essential. Even if model can accurately evaluate whether a rule applies to given BP, it cannot solve the BP unless it can also generate the rule itself. Thus, we propose a fourth evaluation setting in which the task is to hypothesize potential discriminative rules given a BP. Unlike the Open- setting, these rules do not need to be true. Instead, this setting is more akin to a "brainstorming" exercise. The task is considered successful if the ground truth rule is included among the proposed hypotheses. A hypotheses should include a rule for both the left and right side of the BP and will only be valid, if both rules are correct. The prompt for this setting can found in Listing 6.
(Q1) How well can state-of-the-ar VLMs solve Bongard problems? (Task + Task 1.1)
(Q2) How do VLMs perform in comparison to humans?
(Q3) How accurately can VLMs classify images of the BPs given the ground truth rule? (Task 2)
(Q4) How accurately can VLMs generate an underlying hy- pothesis? (Task 3)
Investigating Different Model Behaviours. With our dif- ferent evaluation tasks we want investigate the different behaviours of the models and how the successful solving of one tasks relates to the model being able to solve the other tasks. For this we differentiate between different subsets of the introduced tasks, Type I, Type I1, Type III and Type IV in Figure 3, that result from the intersections of the solved BPs in the different tasks. These are defined in the following:
Data. For our evaluations, we considered the 100 original Bongard problems of (Bongard Hawkins, 1970). We used the dataset variation of (Depeweg et al., 2024), which contains high-resolution images of the original diagrams.
Models. We evaluated the proprietary models ol (Ope- nAI, 2024b), GPT-4o (OpenAI, 2024a), Claude 3.5 Sonnet (Antropic, 2024), Gemini 2.0 Flash Experimental (Deep- mind, 2025), Gemini 1.5 Pro (Team, 2024), and the open- source models Owen2VL-72BBInsteee ILLVVA-One Vision 72B (Li et al., 2025) and Intern 2.5 78B (Chen al., 2024). For simplicity. in the following the models are referred to as ol, GPT-4o, Claude, Gemini 2.0 and 1.5, Qwen2VL, LLaVA-One Vision, and Intern 2.5 respectively. The specific models and their configurations are given in Suppl. A.2.
(Wang et al., 2024a),
Type I: A model is able to solve the BP and can clas- sify images according to the underlying ground truth concepts correctly but fails to come up with the correct hypotheses in Task 3.
Type II: model can hypothesize the correct rule and is able to apply the rule correctly to all images. However, in the open-ended setting its not able connect the dots to solve the problem out-of-the-box.
LLM-as-a-Judge To evaluate the open-ended responses of Task 1 and Task 3, follow common practice and employ an LLM-as-a-Judge (Zheng et al., 2023) (from now on referred to as L.LM-Judge) to automatically judge the responses. In our evaluations, we use the model GPT-4o for it. (cf. Suppl. A.3 for more details).
Type III: A model is able solve the BP and can formulate a correct hypothesis. However, it cannot classify the images of the BP correctly based the ground truth rule.
Model Setup. For all evaluations we tasked our selection of VLMs with solving each BP three times3. The answers in
Type IV: A model is able to answer a BP correctly over all three task settings.
'Eceeption: In Task al was prompted once.
4

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Table 1: Performance of VLMs on 100 BPs (top) as well as multiple-choice BPs (bottom). Results depict the rounded average of solved BPs over runs, All models struggled with the classical BP setup, with ol achieving the highest score, solving only 43 out of 100 BPs. Even on the multiple-choice BPs, difficulties persist. Only when the number of choices considerably limited does the performance increase.
ol
GPT-4o
Claude 3.5 Sonnet
Gemini 2.0 Flash
Gemini 1.5 Pro
LLaVA-OneVision
Qwen2VL
InterVLL 2.5
Solved BPs (of 100)
43
25
31
25
7
Multiple Choice (100)
57
23
37
27
6
Multiple Choice (10)
91
68
80
59
37
66
the context of Task and Task 3 were evaluated by the LLM- judge, which determined whether each response correctly solved the BP. We provide the quantitative results of (Q1) as rounded averages and for the rest of the evaluations consider a BP solved if it is solved in two out of three attempts.
(cf. Figure 2) reveal that the models fail to identify the correct rule and instead focus on irrelevant features, such as shape distortion, horizontal symmerry, and curved lines.
In an extended setup (Task 1.1), we evaluate how perfor- mance changes when the models are explicitly provided with all existing rule pairs of the BPs and asked to select the correct one (cf. Table Multiple Choice (100)). In- terestingly, this does not substantially change the results of GPT-4o, Gemini 2.0, LLaVA- Oisionn or Qwen2VL. However, ol, Claude, Gemini 1.5, and Intern 2.5 do benefit from this setup, e.g., Claude solves now 37 BPs on average, while ol solves even 57.
Human Evaluation Setup. For evaluating human perfor- mances on BPs, 30 participants were recruited among Pro- lific users (Prolific, 2014) to participate in an online experi- ment run via the online platform (LimeSurvey, 2012). The only selection criterion was English as first language. The experiment was divided into three sessions, each taking ap- proximately 30 minutes. In each session, participants were asked to solve 33 BPs. The BPs were shown in the original order (#2 to #100). BP#1 was used in the instructions. In each problem, participants had to provide rule for each side of the BP or click "I don't know" and move on to the next BP. Participants were only invited to participate in the following session if they tried to answer more than 50 % of the BPs presented to them in the session. All participants were reimbursed for their time and could receive bonus pay- ments for correct solutions to problems and completing all three study parts. 23 participants completed all the sessions. Out of these, we excluded three datasets: two because they did not try to answer more than 50 % of the BPs, and one for submitting nonsensical answers in the last session. The resulting dataset consists of the solution of 20 participants for 99 BPs each.
To simplify the task, we reduce the number of answer choices to 10 possible rule pairs, ensuring that the true underlying rule is always included, and repeat the evalua- tion procedure using Task 1.1. In this simplified setting, the selected models show significant improvement, solving up to 91 BPs (cf. Table 1, Multiple Choice (10)). Notably, the correct solution was already present among the choices in the previous setting (Multiple Choice (100)), where the mod- els previously selected incorrectly. This raises an important question: Do the models genuinely grasp the underlying concepts, or is the improvement merely a byproduct of a less complex elimination process? The results suggest that simplification helps models better leverage contextual cues, but does not conclusively demonstrate conceptual under- standing. This leads to a broader inquiry: Can specialized contextual cues enable a model to solve a BP that it previ- ously failed to solve? We explore this in our (Q3) evalua- tions. Before that, however, let us first compare the results of the VLMs human reasoning abilities.
Can VLMs solve Bongard problems? (Q1) As a first step, we aim to assess how well current statc-of-the-arr VLMs can solve BPs according to Task I, i.e., the open-ended query setting. The evaluation results are shown in the top row of Table 1. Notably, across the set of VLMs, ol emerged as the best-performing model, with an average of 43 solved BPs. This is followed by Claude with 31, and GPT-4o and Gemini 2.0 with 25 solved problems. This performance is still surprisingly low, especially when compared to human abilities (cf. (Q2) and (Linhares, 2000; Nie et al., 2020)). A more detailed breakdown of which BPs were solved is provided in Table 4 (Suppl.). Even comparatively simple BPs, such as positioned right vs. left (BP#8) and thin and elongated vs. compact (BP#111 1)), remained unsolved in all attempts. Example responses from the VLMs for BP48
How do VLMs perform in comparison to humans? (Q2) We next evaluate how well the individual VLMs compare to human performance on the BPs. Hereby. for a more detailed comparison, we group the BPs into five categories based on the nature of their ground truth rules (cf., Table 4 (Suppl.) for the categorization). The results of the human evaluations over all BPs are reported in Figure 4 (left), where we have added the corresponding VLM performances (over all models and runs) for comparison. We observe that the best human performance surpasses VLM performance by
5

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
70
100
Humars Models
80
50
Solved BPs (%)
*Solved
40-
60
72
30
40
20
20
Humans Models
90
0-
0
Humans
Models
Size
Concept
Number
Spabial
Same
Solved
50 100
Unsolved
Figure 4: Human results compared to results of VLMs. (Left): Comparing the total number of solved BPs between human participants and VLMs. (Middle): Breakdown of solved BPs by category. (Right): Heatmap illustrating which BPs were solved by the set of human participants and VLMs overall. Humans and models tend to solve different problems.
far; ie.., 68 BPs were solved. Interestingly, we see that the performance varies lot between the subjects. However, there is a substantial number of them able to solve 50 or more of the problems.
least once by any model. A more detailed breakdown of the model's results can be found in Figure 8 (bottom). The results show that there exist group of BPs that remains particularly challenging for current state tute-of-theeart VLMs.
When focusing on the different BP categories (cf. Figure 4 middle), we observe that humans are substantially better at solving BPs that have spatial reasoning components. L.e., spatial one of the trickiest categories for current VI.M models, with all investigated models solving under 25%. For comparison, the top-performing humans solve more than 60%. This is in line with recent works suggesting that VLMs struggle with spatial relations (Rahmanzadehgervvi et al., 2024; Kamath et al., 2023; Zhang et al., 2024).
In summary, we must conclude in response to (Q2) that even though VLMs exceed humans in some of the prob- lems, they perform significantly worse in the overall picture, falling short of matching the visual pattern recognition and reasoning capabilities of humans. To better understand the shortcomings, we now conduct detailed analysis deter- mine whether they recognize the core concepts of the BPs when explicitly prompted for them in (Q3).
Can VLMs detect concepts of BPs? (Q3) In the previ- ous evaluations, we observed that the investigated VLMs performed poorly on the BP dataset. This could be due to difficulties in accurately perceiving the diagrams, well as reasoning failures, such incorrectly formulating rules that apply differently to each side. To investigate this in more detail, we next evaluate the models according to Task 2. Recall that in this setting, a model is asked to classify the single images of a BP based on the ground truth concepts for the left and the right side.
In the concept category, ol scored the highest accuracy with 77% solved of these BPs (ef. Figure 8 for the detailed results). This is category where humans appeared to strug- gle more, ie., solved on average 54% of the BPs. This could be due to that VLMs hereby benefit from their extensive world knowledge while not all humans might be familiar with geometrical visual concepts, such as convex vs. con- cave (BP44) and thin elongated convex hull vs. compact convex hull (BP#12).
As indicated by the distinct comparison across the BP cate- gories, the BPs solved by the models differ from the ones solved by the humans. If we aggregate the values of the study participants and compare which BPs they solved (more than half of them found a solution) against which ol solved (in at least 2 attempts) we see that there are 16 BPs that ol solved but not the humans. On the other hand, the humans solved 31 BPs that ol was not able to solve.
Intuitively, one might expect that if model can solve a BP in Task 1, it should also be able solve Task 2 for this BP. To examine this, we compare the set of BPs solved in Task I to those successfully completed in Task 2. In Task 1, we define a BP as solved it is correctly identified in at least two out of three attempts. Similarly, in Task 2, a BP is considered solved all its images are correctly classified in at least two out of three attempts. We visualize these two sets and their intersection in Figure 5. Surprisingly. we find that the overlap between these two sets is quite small. This suggests that for substantial subset of BPs, the models are able to solve them in Task I despite failing to classify every individual image correctly according to the ground
When comparing BPs that have been solved at least once by any VIM to those solved at least once by human, an interesting pattern emerges, as visualized in Figure 4 (right). Out of the 100 BPs, nearly all are solved by at least one human participant. In contrast, only 66 BPs are solved at
6

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
01
GPT4o
Claude 3.5
Gemini 2.0
Which direction is the spiral turning?
#16
28
16
13
11
20
11
11
13
11
19
ol:
cw
CCW
X
CCW
Gemini 1.5
LLaVA-OneVision
Qwen2VL
InternVL2.5
Claude 3.5:
CW
V
CW
Gemini 2.0:
InternVL255:
4
18
2 3 12
9 14
11
Is the circle left or right of the cavity?
Detect Concepts (Task 2) Figure 5: Comparison of VLM performance on Task and Task 2. Blue circles represent the set of BPs solved in Task 1 (open-ended solving); orange circles those correctly classified in Task 2 (concept recognition). The overlapping region indicates BPs where a model succeeded in both.
Open Question (Task 1)
#55
10
ol:
left
left
left
X
left
X
Claude 3.5:
right X
right
right
Gemini 2.0:
right X
left
left
X
left
InternVL2.5: right X
right
right
left
Figure 6: VLMs fail to identify simple visual concepts
(Task 2). VLMs are challenged with identifying visual concepts in the individual images of the BPs. Abbreviations used for clockwise (cw) and counter-clockwise (cow).
truth rule. Conversely, there are also many BPs where the models perfectly classify images but still fail to solve the open-ended task. This large discrepancy highlights a sur- prising gap between recognizing correct classifications and effectively applying that knowledge in problem-solye solving. We report the number of correctly identified concepts for Task in Table 9 (Suppl.) and further investigate Suppl. B.4 whether the observed gap is affected by presenting full BP images versus individual images to the VLM.
while, surprisingly, the stronger models perform relatively worse compared to their performance in Task 1. E.g., Gem- ini 2.0 solves the most problems in this setting, with only 30 correct concepts. This unexpected trend suggests that higher-performinn models in the open-ended task do not necessarily excel in applying ground truth concepts to in- dividual images. Overall, the number of puzzles in which the models correctly identify the presence of the provided ground truth concept is substantially low, highlighting per- sistent challenge in concept recognition and classification.
For a better assessment of these results, we give a quali- tative impression of the models' mistakes when prompted to identify specific, rather simple, concepts of BP#16 and BP#55 Figure 6 (ef. Table 10, 11 for all responses and Figure 9 for additional BP#29 and #37). We find that for BP#16 (ef. Figure 6 top), even though some images are clas- sified correctly, none of the models can classify all images correctly. Instead, we can see a tendency to classify one of the directions rather than the other, egg., Claude and GPT-4o almost always predict clockwise as the turning direction for all of the diagrams.
Conclusively, the observed behavior is remarkable and seems to indicate that perception is a key issue for not iden- tifying the correct rules of BPs. With regard to (Q3) we have to conclude that the current perception capabilities of the evaluated VLMs are still insufficient to visually capture the concepts required for solving BPs.
For BP#55 (ef. Figure 6 bottom) also see that none of the models is able to classify the concepts of all images correctly. Here, we can see very similar behaviors across the different models, e.g., almost all models categorize the first diagram of the BP falsely, but in most of the attempts, they are able to categorize the third diagram correctly. Interestingly, we found a strong correlation between the absolute position of the circle in the diagram (left vs. right) and the models responses. We take this as indicator, that the models did not manage to reason spatially about the position of the circle in relation to the cavity.
Can VLMS generate relevant hypotheses? (Q4) In next task evaluation, we investigate the potential of the mod- els for generating correct rules for the BPs. In (Q3), we observed that the investigated models failed to classify the BP images accurately. This raises the question of whether the models might be finding the correct rule for most prob- lems but mainly failing to apply it properly to individual BP images, ultimately overlooking it. To explore this, we evalu- ate the models on Task 3 and report the results in Figure 7 and Table 15.
Surprisingly, we observe that the set of correctly hypothe- sized rules Task 3 is significantly larger than the number of solved BPs in Task 1. For instance, ol proposes correct hypotheses in out of 100 BPs while only solving 44 open
When we consider the overall number of solved concepts across the models (ef. Table 9), we see that weaker models from Task 1 achieve slightly better results in this setting,

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
01
GPT40
Claude 3.5
Gemini 2.0
the presence of significant number of Type II subsets is surprising. This implies that while the models can formu- late correct hypothesis and correctly classify BP images according to the ground truth rule, they still fail to solve the BP in the open-ended task. This suggests that, although the models theoretically possess the necessary knowledge, they struggle to apply it effectively.
21
32
28
32
Gemini 1.5
LLaVA-OneVision
QwenZVL
InterVLL.55
0
When examining Type III subsets, we find that they make up a substantial portion of the diagrams, particularly for the stronger models. This is intriguing because it suggests that the models can determine the correct rules even in the open-ended task, yet they fail to apply it correctly to individual BP images (further undermining the findings of (Q2)). This raises the question of why this discrepancy occurs. One possible explanation is that the proposed rules are not derived through explicit reasoning. If the models were truly reasoning based on the rules, we would expect the individual images to be classified correctly. Instead, it seems that the models may infer the most likely rules based on the overall set of images without actually executing these rules for each image, leading to occasional success despite the underlying inconsistency.
12
35
27
30
37
Open Question (Task 1)
Formulate Hypotheses (Task 3)
Figure 7: Comparison of VLM performance on Task and Task 3. Blue circles represent the set of BPs solved in Task 1 (open-ended solving); green circles those correctly classified in Task (formulate hypotheses). The overlapping region indicates BPs where a model succeeded in both.
question BPs. Such a disparity is particularly pronounced for VLMS that initially performed poorly in Task 1, such as LLaVA-One Vision which manages to generate 32 correct hypotheses while solving open question BPs.
Interestingly, we find that the proportion of Type IV BPs (i.e., solving BP across all three tasks) across the models is relatively low compared to the number of solved BPs in Task 1. For example, ol has only 14 Type IV BPs out of 44 solved BPs, representing just around 32%. While this ratio is slightly better for the weaker models in Task 1, the overall number of solved BPs in these models remains low, making the comparison less meaningful (cf. Table 15).
Furthermore, we observe that for most models, the set of BPs solved in Task is not subset of those with correctly identified rules in Task 3. This is somewhat unexpected, as one might assume that successfully solving a BP would imply the correct rule is among the hypotheses generated in Task 3. This finding may point to a lack of robustness in generalizing across the different task setups.
Most challenging BPs. Throughout the experiments, we saw that the investigated VLMs can solve some BPs better than others, where the set of solved puzzles appears to differ across task setting. Interestingly, we find that there are 10 BPs that are especially hard, i.e., they were not solved in any of the tasks (cf. Suppl. B.7). We think that these BPs can serve as good foundation for further research and bench-marking.
In conclusion, the results suggest that while VLMs may struggle to solve BPs directly, they more often succeed in formulating correct hypotheses about the underlying rules. However, this process might be brittle, as for some models solved BPs from Task 1 are not solved in Task 3. With regard to (Q4), we conclude that while hypothesizing correct rules seems to be easier for the models than solving the BP straight ahead, they are still not able come up with correct hypotheses for all the problems.
Limitations. While BPs are valuable for assessing abstract reasoning, they also represent a narrow and highly spe- cialized set of challenges that may not comprehensively reflect the broad range of problems VLMs encounter in real- world applications. Even though we take the less distributed dataset of the BPs (high resolution), it is still possible that the problems might be part of the training data. Future work should expand these evaluations to new and more diverse tasks. Additionally, the reliance on our MJdge intro- duces some uncertainty in the evaluation process, which we discuss in Suppl. A.3.
5. Discussion: Remaining Challenges for AI
Based on the results of the previous evaluations which are summarized jointly in Figure 10, there are several additional important aspects to address. Specifically, we want to inves- tigate the relations between the solved BPs in our proposed task settings (cf. Figure 3).
The first notable observation is that there are almost no Type subsets, suggesting that if a model can solve the BP and correctly classify the concept, it is also likely to generate the correct ground truth rule in Task 3. However,
8

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
6. Conclusion and Future Work
Impact Statement
This work presented a diagnostic evaluation of VLMs using the classical Bongard problems (BPs), providing valuable insights into their current capabilities of pattern recognition and abstract reasoning. Our experimental results highlight significant gap between human-like visual reasoning and machine cognition. Specifically, we found that VLMs are still largely unable to solve the majority of BPs, with the best-performing model, o1, solving only 43 out of the 100 BPs. Moreover, our analysis suggests that the limitations of current VLMs extend beyond just visual reasoning; they also struggle perceive and comprehend elementary visual concepts, such simple spirals.
This study critically examines the reasoning capabilities of VLMs in the context of Bongard problems, benchmark for human-like pattern recognition and abstract reasoning. By highlighting the limitations of VLMs in detecting and under- standing fundamental visual concepts, our work provides insights into the gaps that remain between human cogni- tion and machine perception and reasoning. These findings contribute to the broader discussion on the reliability and interpretability of Al models, particularly in tasks requiring abstraction and generalization.
From ethical standpoint, our research underscores the risks associated with overestimating the reasoning abilities of VLMs, which could lead to premature deployment in critical applications such as medical diagnosis, autonomous systems, and decision- decision-making tasks. As VLMs continue to be integrated into real-world applications, it is essential to remain cautious about their limitations and to develop methods for improving their interpretability and robustness.
Our findings raise several critical questions: Why do VLMs encounter difficulties with seemingly simple Bongard Prob- lems, despite performing impressively across various es- tablished VLM benchmarks (Li et al., 2024; Duan et al., 2024)? How meaningful are these benchmarks in assess- ing true reasoning capabilities? Future work could use our results as base to develop new benchmarks. Further, the dis- covered Type II subsets suggest opportunities for improving the performance, egg., via a multi-stage reasoning approach.
While our work does not present immediate societal risks, it highlights the importance of ongoing evaluation and re- sponsible deployment of AI systems, ensuring that their capabilities are well understood before being applied in high-stakes environments.
Acknowledgements
The authors would like to thank Frank Jäkel for useful discussions. This work was supported by the Priority Program (SPP) 2422 in the subproject "Optimization of active surface design of high-speed progressive tools us- ing machine and deep learning algorithms" funded by the German Research Foundation (DFG). We acknowl- edge support of the hessian.Al Service Center (funded by the Federal Ministry of Education and Research, BMBF, grant No 011S22091), the "Third Wave of AI", and "The Adaptive Mind". The work was also supported by the LOEWE research priority program "WhiteBox" [grant num- (funded by the Hessian Ministry of Higher Education, Research, Science and the Arts). Further, this work was funded by the Euro- pean Union (Grant Agreement no. 101120763- TANGO). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Exec- utive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. The authors of the Eindhoven University of Technology re- ceived support from their Department of Mathematics and Computer Science and the Eindhoven Artificial Intelligence Systems Institute.
References
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.,, and Parikh, D. VQA: visual question answering. In International Conference on Computer Vision (ICCV),
Antropic.
Claude 3.5 sonnet. https:/ hetps:///ww.. anthropic. news/ claude-3-S-sonne 2024. Accessed: 2024-09-25.
ber LOEWE/ 2/1 3/51 10//3//9//333
Bongard, M. and Hawkins, J. Pattern Recognition. Spartan Books, 1970.
Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., Jayaraman, B., et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405. 17247, 2024.
Chen, Z., Wang. W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J.. Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Chollet, F, On the measure of intelligence. arXiv preprint 1rXiv:/9111 1915514477 2019.
Deepmind, G. Gemini 2.0 flash experimental, 2025. URL
9

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
https://deepminde googll gemini/flashi Accessed: 2025-01-28.
google htt.ettehhcoooceeecccccccc
Li, B., Zhang, P., Zhang, K., Pu, F., Du, X., Dong, Y., Liu, H., Zhang. Y., Zhang, G., Li, C., and Liu, Z. Lmms- eval: Accelerating the development of large multimoal models, March 2024. URL https: http:://ggttbb...oo/ com/ Evol v -ML.MM---MMM laaaaa/mmmm Imms-evall
Depeweg, S., Rothkopf, C. A., and Jäkel, F. Solving bon- gard problems with a visual language and pragmatic con- straints. Cognitive Science, 2024.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., and Li, LLaVA- onevision: Easy visual task transfer. Transuctions on Machine Learning Research, 2025.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y. Dong, X., 7ang, Y., Zhang, P., Wang, J., et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, 2024.
LimeSurvey. Limesurvey: An open source survey tool, 2012. URL https:///veee /wwp:://wwwwww./ www.:lumeeuue...omm org..
Geigle, G., Timofte, R., and Glavaš, G. African or european swallow? benchmarking large vision- -aiinnl models for fine-grained object classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Linhares, A. A glimpse at the metaphysics of bongard
problems. Artificial Intelligence, 2000.
Liu, H., Li, C.,, Wu, Q., and Lee, Y. Visual instruction ing. Advances in neural information processing systems, 36. 2023.
Gou, C., Felemban, A., Khan, F. F., Zhu, D., Cai, J., Rezatofighi, H., and Elhoseiny, M. How well can vi- sion language models see image details? arXiv preprint arXiv:2408. 03940, 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S.. and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-ve github..h...... .ioubb/.o//lo///// 2024 204-01-30011ava-neee
Helff, L., Stammer, W., Shindo, H., Dhami, D. S., and Kersting, K. V-lol: diagnostic dataset for visual logical learning. Journal of Data-centric Machine Learning Research, 2025.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, 2024b.
Hernández Oralloo J., Martínez-Plumed, F., Schmid, U., Siebers, M., and Dowe, L. Computer models solving intelligence test problems: Progress and implications. Artificial Intelligence, 230, 2016.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W,, Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learming Representations (ICLR), 2024.
Hong, Y., Yi, L., Tenenbaum, J., Torralba, A., and Gan, C. PTR: A benchmark for part-based conceptual, rela- tional, and physical reasoning. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Matkiáski, M., Pawlonka, S., and Mandziuk, Reasoning limitations of multimodal large language models. case study of bongard problems. International Conference on Machine Learning (ICMI.), 2025.
Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoming and compositional ques- tion answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Marr, D. Vision: A computational investigation into the hu- man representation and processing of visual information. MIT press, 2010.
Johnson, J., Hariharan, B., van der Maaten, L.,, Fei-Fei, L., Zitnick, C. L., and Girshick, R. B. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Masry, A., Do, X. L., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2022.
Kamath, A., Hessel, J., and Chang, K.-W. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Moskvichev, A. K., Odouard, V. V., and Mitchell, M. The conceptare benchmark: Evaluating understanding and generalization in the are domain. Transactions on ma- chine learning research, 2023.
10

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Nezhurina, M., Cipolina-Kun, L., Cherti, M., and Jitsev, Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406. 02061, 2024.
Xiao, Y., Sun, E.,, Liu, T., and Wang, W. Logievista: Multi- modal IIm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407..4997, 2024.
Youssef, Y., Zečević, M., Dhami, D. S., and Kersting, K. Towards solution bongard problems: A causal approach. arXiv preprint arXiv:2206.077 2200:27996.6 2022.
Nie, W., Yu, Z., Mao, L.,, Patel, A. B., Zhu, Y., and Anand- kumar, A. Bongard-logo: A new benchmark for human- level concept learning and reasoning. Advances in Neural Information Processing Systems, 2020.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang. W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-disciplle multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
OpenAL. Gpt-4o. htttps://ppent www.coooo///./oo index/
hello-gpt-4o//
Accessed: 2024-09-25.
OpenAl, ol. hwtpt:tt/////. woweaapena...mm.ooooo com/ol /// 2024b. Ac- cessed: 2025-01-23.
Zellers, R., Bisk, Y., Farhadi, A., and Choi, From recog- nition to cognition: Visual commonsense reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Paolacci, G. and Chandler, J. Inside the turk: Understanding mechanical turk as a participant pool. Current directions in psychological science, 2014.
Prolific. https: www http://wwwowl...ccooo 2014. Ac- cessed: 2025-01.
Zhang, C.,, Gao, F., Jia, B., Zhu, Y. and Zhu, RAVEN: dataset for relational and analogical visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Qiao, Y., Duan, H., Fang, X., Yang, J., Chen, I.., Zhang, S., Wang, J., Lin, D., and Chen, K. Prism: A framework for decoupling and assessing the capabilities of vlms. In The eiirty- Amnual Conference on Neural Information Processing Systems, 2024.
Zhang, Y., Bai, H., Zhang. R., Gu, J., Zhai, S., Susskind, J., and Jaitly, N. How far from intelligent visual deductive reasoning? ICLR 2024 Workshop: How Far Are We From AGI, 2024.
Raghuraman, N., Harley, A. W., and Guibas, L. Cross-image context matters for bongard problems. arXiv preprint 1rX2v:2300.033668, 2023.
Zheng, L., Chiang, W.-L., Sheng, Y.. Zhuang, S., Wu. Z., Zhuang, Y., Lin, Z.,, Li, 7., Li, D., Xing, E., al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 2023.
Rahmanzadehgervv. P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. arXiv preprint arXiv:2407.06581 2024.
Zhou, K., Lai, E., Yeong, W. B. A., Mouratidis, K., and Jiang, J. Rome: Evaluating pre-trained vision- language models on reasoning beyond visual common sense. EMNLP, 2023.
Team, G. Gemini: family of highly capable multimodal models, 2024.
Vedantam, R., Szlam, A., Nickel, M., Morcos, A., and Lake, B. M. CURI: benchmark for productive concept learning under uncertainty. In International Conference Machine Learning (ICML), 2021.
Wang, P., Bai, S., Tan, S., Wang, Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W. al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409, 12/91, 2024a.
Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., and Yang, H. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.0680) 2024b.

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Supplementary Materials
In the following, we provide details on the experimental evaluations as well as additional results.
A. Experimental Details
We provide here the prompts for our evaluations (Suppl. A.1), model details (Suppl. A.2), details about the LLM-Judge (Suppl. A.3) and details about our human study (Suppl. A.4).
A.1. Prompts for the experiments
In the following, the prompts used during the experiments are provided. The prompt for open-ended solving (Task 1) is shown in Listing 1. The prompt for the multiple choice setting (Task 1.1) is in Listing 2. The prompts for Task 2 and Task 3 are provided in Listing and Listing 6.
Listing Prompt for Task 1. The model is asked to provide rules for the left side and the right side images of the Bongard problem.
You provided with black and-whadd white image consisting of 12 simple diagzams, Bach diagram represents shapes with specific reatures, such as geonetric propectiea or nigher-ievel concepts.
The 6 diagrams on the left side belong to set A. 5.
The diagrams on right belang
44 Task:
Your task is to determine two distinct rules:
10 1. Set A sule: identify rele that applies to all diagrans in Set A. Rule: Identify separate that applies all diagroma Set
11
12
13 Important: The zule for Set A must not apply to any diagram in Set B, and the rule For Set B must not apply to any
diagran in Set A.
14
15 step-by-Step Process:
16
17 1. Diagram Analysis: Carefully describe each diagram detail, roting any geometric properties, patterna, or
conceptual reatures,
18 2. Rule Desivation: Based on your analysis, deduce the rule for Set & and the rule for Set B, ensuring that each sule unique to its
19
20
Answer Format:
21
22 Provide the final answer vaing the following format:
23
24
"pyytoon
26 'set A rule': ([BPT RULE)
rule': [RIGHT RULE]
78
29
30
31
Ensure that the rules are clearly defined, concise, and do not everlap between the sets.
A.2. Model details
We provide details on the models used in our experiments in Table 2.
A.3. I.MM-Judee
For the systematic and automated evaluation of Task and Task we made use of LLdeas-a-ludee (Zheng et al., 2023). For this, we designed a prompt that explains the task in detail and gives five example judgements as directive to improve the performance via few-shot (ef. Listing 7 and Table 3 for examples). The answers the reasoning task are compared to
12

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Listing 2 Prompt used in multiple choice experiment for solving BPs with solution options provided. The model is asked select the rules for the left side and the ride side images of the BP that fits best. <SOLUTIONS>> is replaced by dictionary of the possible solutions the model can select from (either all 100 or subset of 10).
are provided with a black-and-whita image consisting of 12 simple diagrams. Each diagram representa shapes with specitic reatures, such as geonetrie propertiea or nigher-level. concepts.
The diagrams the left side belong Set The 6 diagrams on the right side belung to Set B.
addicionally, are given a list of possible sule pairs, one of which is true for this image. Your goal is to idensify the correct rile patr based on the features the diagrams in Set A and Set 5.
4 Task:
10 Your Lask is to ideatify and select the correct rule pair that is true for the sets. The zule pair is structured follows:
11
12 1. Rule part 1: This rule ahould apply to all diagrama in set A
13 2. Rule pact 2: This rule should apply all diagrama in Set B
14
15 Important: the rule for Sat A not apply any dingram in set E, and the rule for set must not apply any diagren in Set A.
17 # Scop-by-Stop Process:
18
19 1. Diagram Anslysis: Carefully describe each diagram detail, noting any geometric properties, patterns,
conceptusl festures.
20 2. Rule Derivation: Based your analysia of the diagrams, select rule from the provided list for Set and different zule for Set B.
22 #4 Aval lable Rules
23 You can choose Irom the following zule pai:s:
24
25 <SOLUTIONS>
26
27 14 Pinal Answer Format:
28
29 Provide the final answer using the following Tormat:
30
31 "pyython
32
34
'anawe=": <Bolutios ID>,
16
36
37
Where <solution IDS is the number corsesponding to the correct rule pair that fits the criteris.
the ground truth BP r To validate the automated results, we conduct manual annotations on large subset, i.e., 696 BP responses, both from the human study and the VLM answers. The human annotations were obtained from two experts that were recruited in-house. The two experts annotated the rules independently of the judge. We found that the judge has a 91% agreement score with the human judgement. One caveat in this method is that, in principle, many rules can be found for one Bongard problem (Depeweg al., 2024), and each rule can be expressed in many ways that differ in context and specificity. We acknowledge that the LLLM- evaluates the answers only within the context of the ground truth rules and might misjudge technically correct rules. However, constraining the rule evaluation to the ground truth rules captures the pragmatics of Bongard's 's original rules.
A.4. Human Study on BPs
Depeweg et al., 2024 evaluated in their work already humans solving BPs in the open-ended task. Their results showed a high human performance; the five subjects that worked on all BPs solved 42 of the 49 (the first problem excluded for comparison) on average (Min: 40, Max: 45). In comparison, the twenty participants in this study only solved 20 BPs out of the first BPs correctly (Min: Max: 31). The difference in performance can be the result of multiple factors. While the dataset from Depeweg et al., 2024 was recorded in lab environment with university students, the participants in this study
*htpps: h/tps:///wwwww//
www.///wwww.aaaaaoa/o////aa founda wwm/ ream/res/tee httpee//bb////gaeeeeeeeeeeeeeeee cedddd_obbbb prroblemm__se ems_ i oos.. htm
13

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Listing 3 Prompt for classifying ground truth concepts in single images (Task 2).
You aze given a pair of contrary concepts. Your task is to oraft a single prompt that aaks which of those concepts appeacs image.
Exomples reference:
44 Example Contrary Concepts: I
"Spizal curls countecclockwise",
"Spiral curls elockwise"
Prompt: (ap' proonnt)))
11
12 # Exemple
13 Contrary Concepts: I
14
"a circle is at the left of the cavity if you look from inside the figuze",
circie
right cavicy you lock from inside figure"
16
17 Prompt: [cavity_prompt]
18
19 4 Your task:
20 Please formulate prompt for the controry concepts balow that asks which present the given image.
21 Contrezy Concepts: [value]
Listing 4 Prompt for concepts of BP#16.
Your task determine the direct fon which spiral deploted black and white diagram turning. The given dLagran shows spiral spiral-like shape. In which direction is the spiral turning, starting Tron the center?
Please decide carefully whether the spiral is surning in clockrise counterclockuise direction. Take deep breath and think step-oy-step, Give your answer in the fol lowing format: ...
answe: 1
rditection": syour answer>
where <yous
ba either "countarclookwwce"" "clockrise""
were sampled from the prolific participant pool, which offers a broad range of educational and experience backgrounds, which could explain the higher variability (Paolacci & Chandler, 2014). Additionally, participants in our study might have optimized their time over their performance. On average, participants spent 49 seconds with a problem before skipping it. The experimental design of Depeweg et al., 2024 allowed skipping only after 2 minutes, forcing the participants to spend more time on problems.
14

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Listing Prompt for concepts BP#55,
Your task is to determine the position circle in relstion to cavity in 20 black and white diagram.
given diagran shows shapa with cavity. From insida the figure, nead decide the circle the left or the right of the cavity, carefully analyse the diagram atep-by-step to Identifyy the correct alde.
Please decide careful ly. Take deep breath and think step- Give your answer in the fol lowing format:
"'python
answer =
"position": <your
10
12 where <youz answer> can be either "left" if the circle is to the lef: of the cavity "righe" if the circle is to the right of the cavity.
Listing 6 Prompt for hypotheses (Task 3).
You ace provided with a black-and whdtt Image consisting of 12 simple diagzams, Each diagzam repzesents shapes with specific features, such gecmetric propecties aigher-level concepts.
The diagroms
laft side balong The 6 diagrams on the right side belong to Set B.
#4 Task
Prepose multiple distinet hypotheses explaining how the diagrams in Set might share rule (or set of zules) that coes not apply to Set B, and vice versa. in other words, each hypothesis should outline:
10 1. A feature rele that unifies all dingrams in Set A all diagrams
11 different feature or rule that
12
13 These rules must be mutually exclusive whatever defines Set à must not be true for Set B, and vice versa. You may consider variety factors, like type, number style, or any other relevant factor but only in the context of tha images of each set.
of shapes, orientation, synmetry, color usege, line
15 # Answer Format
16
17 1. Return your hypotheses Python list of paics of strings.
18 2. Each list element should be a tuple (cule_for__et____ rule_for_Set_B1 example:
20 "'python
22
("A'l the diagrams contain at least one triangular shape", "There exists no triangular shapa in the diagrams"),
23
("Dach diagram festures single shape that is positioned in the bottom half of the disgram", "Each diagram features a single shape that is positioned in the upper half of the diagram"),
25
17 4. Ensure each rule 1a clear, specific and unique.
28 5. Avold stating which hypothesss seems most Likely or correct; simply propose à variety of possibilities.
29
30 # Additional Guidalines
32 Each hypotheais should stand on ita own and be mutually exclusive to the other set.
crestive, only your statemeats plausible observations 12 diagrams.
34 Provice 20 instinet, well-derined hypotheses.
15

---

Bongard in Wonderland:
Visual Puzzles that Still Make AT Go Mad?
Table 2: Details the models used in the evaluations.
Model
Parameters
Framowork
Devices
o1
o1-2024-12-17
Default max tokens=20048
openai (API)
3ut-fo-2024-08-05
Defaull, 1aa__ookom-22048
opensi
Claude
claude-3-
Default max tokens=20488
anthrooiic (AP1)
Gemini
gamini-2.0-llasbbee
google (API)
Gemini 1.5
gemini-1.5
Default max tokens=2048
google (API)
Defaul,, iok
fornsformrrs
A100-SXM4-8UU0
Qwen2VL
models- Saaels--------e2------- Qwwn---w-2222----- Instract
Defauk, max.iokens=20088
transformers
GPUs INVIDIA A100-SXM- 80CB)
dkploy
A100-SXM4-80UC
GPT-4o Judge)
2pt-4o-2024-8-400
Defaukt temperature-O. max mox.sokens-2048
openai (API)
Listing Prompt for LLM-judge used across the experiments. There are five example judgements provided (<EXAMPLES> placeholder). The judge needs to decide whether the answer correct based the provided ground truth (1) incorrect (0).
right pattern scenario. You will slso be given answer from model that actempis describe these rules. Your task is to evaluate whether the model's answer accurstely reflects the insent and assence of the correct answer.
given correct ansver states
left
visuel
Evaluation Criteria:
Semantic Accuracy:
the model's answer convey same underlying concept
the correct answer,
if the wording differs?
Logical consistency: the model's answer logically consistent direstly address the rules provided in the correct answer?
corsect answer's rules?
Relevance: Does the model's
Reaponse "nat ronsi
model's enswez is correct according criteria 11 Respond with "answer": 0 if the model's answer is incorrect.
Respond "answer":
the model's answer only partially correct, considar whather partial match sufficiently corveys tha intended rule. if it does, respond with "answer"; 1; etherwise, respond with "answer"; 0.
13
14 Examples:
15 -EXAMPIES>
17 Uoe the fornat shove to judge the correctness of the model's 's answer based on the given correct answar.
Task
20
Correct Anawет:
22
Right: BIGHT_RJLE_BOLUTT
Answer:
24
left: RULE_ ANSNER
RULE_ANSMER
Right: SIGHT_ROLE_AAT
Response:
27
Table 3: Examples used as few-shot examples for the LLMM-udge.
Correct Answer Left
Correct Answer Right
Model Answer Left
Model Answer Right
Score
Example
Round outlined shapes
Filled squares
White circles Circular shapes
Black squares
Example
Large shapes
Small shapes
Irregular shapes
Example
Square on top of circle Circle
Circle on top of square Triangle
square top Circle
square bottom
Example
Non-circle
Example
Smooth contour figures
Twisting contour figures
smooth
jagged zig-zag
16

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Table 4: Results of each VLM on the individual Bongard problems compared to the results of our human study. Each model was prompted three times and the number of correct responses is reported (of 3). The reported ratios are marked based on four intervals: less than 1/3 (white), (1/3-2/3) (light green), (2/3-3/3) (green), 3/3 (dark green).
Caopesio
CPT-cc
Geneai Caal:33
Genteil 1.
LLLNW--V Qrasdh. InaesVL 25
Hanas
S0M
Caspestor
GPF-4
Claak
Guecni 20
Co
QeadVL
EaanNV. 2.1
Hazan
conces
35
301
-22
112
022
colcegt
13
311
37
30
10
315
30
375
19:30
0g1
-35
113
073
130
1/3
1603)
concegt
0/
07
077
071
0973
spuial
02
11/
(87
-07
305
1/3
37
DG
10
15:30
03
077
620
1/3
30
30
DG3
05
00
1723
32
1/5
1/
30
873
93
07
120
1501
02
007
05
180
325
20
169
025
011
125
1/3
23
05
03
1530
185
023
002
04
023
61
18
123
183
95
100
1503
125
05
205
00
03
157
07
1501
C0
on
4/3
DG
00
02
1833
66
194631
43
on
403
163
05
90
07
00
230
CORCES
315
1/3
30
1921
08
02
1201
C33
077
05
220
03
133
103
03
4:3
CONIEE
35
109
1989
66
183
183
10
103
100
CONOCS
125
073
063
01
21
10
05
00
110
(520)
1/3
007
02
320
05
07
623
335
2/7
153
05
12:3
05
02
220
173 335
1/3
1/3
DS
05
42
125
1933
75
02
03
103
23
05
90
07
02
1801
13
02
607
222
KEKER
05
113
23
02
0/
1/3
6/3
0/
quiia
01
421
07
60
10
600
6/
0/
1/
29
quiial
-05
00
120
1/1
1125
177
1.00
32
20
DS
125
130
25
725
422
05
(83
1K3
055
93
02
42
022
LONEES
00
320
05
1021
12
615
03
07
05
03
025
183
133
822
225
61
10
05
1633
183
183
05
95
02
02
323
07
00
720
CUrUl
025
30
D6
05
40
125
1623
05 on
125
183
05
90
07
40
120
CurUl
105
07
07
00
07
470
0n
1N7
07
0.20
42
Xhlal
135
625
025
00
169
89
1823
055
120
033
05
03
1501
43
LEWNE
125
00
109
88
09
05
1330
00
(03
95
000
0.00
X3lal
00
315
1101
M
435
on
07
47
06
88
00
45
1520
16
30
35
103
130
055
90
01
007
16:1
47
cuta)
131
07
105
07
17:11
97
6EKER
30
33
30
1701
015
05
077
05
03
043
07
1631
5J
CORCE::
12
0/
33
07
03
05
02
45
1929
140
60KE8
35
53
103
0/5
10
07
1/
6:3
B. Additional Results
In the following the detailed results of the evaluations are presented.
B.I. Task and 1.1
In Table 4 the results of of each model on Task are reported. Further, it includes the results of our human study. The detailed results for Task 1.1 are given in Table 5 (100 Options) and Table 6 (10 Options).
B.2. Comparison VLMS vs. Humans
In Table 4 we compare the results of the VLMs on a subset of BPs against human results from our study. Each BP has been categorized based on the ground truth rules, the possible categories are size, concept, number, spatial and same. Size means that the rule is based on the size of one or multiple shapes in the BP. Concept means, that the BP tests for a specific, more abstract concept, e.g.. convex and concave. Under number BPs are grouped that require some form of counting, e.g., one vs two shapes. Spatial takes into account BPs that require spatial reasoning, e.g., some shape is on top of the other. And finally same considers BPs that test for same-different reasoning, e.g.. solid quadrangles are identical vs. different.
In Figure& 8, we report detailed results of our evaluations regarding (Q2). We provide the results of each model type individually over each three runs and also the top 5 human study participants individually. Corresponding numerical results are provided in Table 7.
B.3. Task 2
For Task 2 we report the classification accuracy of each model for the ground truth concepts of each BP in Table 8. We provide quantitative results for Task 2 visually in Table 9 and discuss additional qualitative results for the evaluations of (Q3), iee, also for BP#29 and #37. For BP#29, the models were asked determine whether there are more shapes inside or
17

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Table 5: Results of each VLM on the individual Bongard Problems when provided with all possible solutions (Multiple Choice (100)). Each model was prompted three times and the number of correct responses is reported (of 3). The reported ratios are marked based on four intervals: less than 1/3 (white), (1/3-2/3) (light green), (2/3-3/3) (green), 3/3 (dark green)
(iPl-oo
Genine
Gonim
IIaVA-WVV
Qwes7Vi.
InsoVVi
GPTT-o
Clasee 3.5
BLaVA-ON
QwantNi)
IntomVVi 2,5
3/5
3/3
0/3
0/3
0/3
5/3
0/3
0/3
0/3
123
(V1
3/
323
(33
0V3
(V3
33
IV3
003
(V3
01
03
0/3
0/3
0/
3/
V1
56
2/1
129
073
103
0/3
36
3/
3/3
3/
03
03
3/
1/3
57
28
033
30
2/3
113
03
03
0/
03
03
0/3
03
03
003
03
58
043
103
003
(K3
113
033
0/3
3B
3/3
3/5
2/3
3/5
1/3
03
13
99
2/9
003
1/3
12/
0/3
0/
03
10
1/
1/1
3/5
1/5
121
VI
MI
173
122
085
(25
143
LI
36
03
2/3
1/3
033
083
03
61
1/3
033
2/3
313
113
0/3
0/3
312
12
03
03
03
03
033
083
03
02
2/3
183
003
[K3
113
0/3
0V3
0/2
13
365
03
3/3
323
123
03
13
63
0/
013
0/3
033
0/3
0/3
0/3
0/3
14
151
005
0/3
0/3
0/3
L/3
33
0V3
0/3
033
2/3
0/3
113
IV3
03
03
2/3
3/3
0/3
0/3
20
30
1/
IV3
1/3
0V3
21
2/
1/
39
1/3
023
0/3
0/
0/
23
0XS
033
003
0/3
72
(K5
0/3
1/1
23
3/5
3/3
3/3
2/3
03
1/3
3/3
2/3
73
DX3
003
DV3
003
IV3
0/3
0/3
03
24
1/3
3/3
3/3
36
013
03
33
74
IV3
003
D13
IV3
0/3
0/3
0/3
35
3/5
3/3
08
1/3
(09
0/3
2/3
73
3/3
(09
0/3
2/9
(V3
0/3
0V3
1/3
26
1/3
OCS
1/3
0:5
323
005
03
M/3
76
103
045
045
IF3
085
U/1
27
2/3
1/3
L/3
2/3
03
00
03
77
3/
03
0/3
0/3
0/3
0/3
0/3
28
03
03
03
03
03
03
D/3
73
1/3
03
1/3
0/3
0/3
0/3
29
1/S
IX3
2//
3/3
3/9
(03
1/3
79
2/3
045
0/3
0V5
0/3
0/3
0V9
0/3
30
31
L/3
0/3
Drrk
103
(V9
33
(V9
303
IM3
0/3
35
343
28
0/3
0/3
0/3
0/3
16
7/X
V1
M
0/3
17
0/3
(V)
0/3
(V9
(2)
0/3
0/
38
03
Lis
03
03
03
W/3
IN3
(43
IV3
0/3
0/3
39
1/
2/3
3/5
2/3
3/3
003
03
00
89
0/3
333
3/
003
0/3
0/3
0/3
03
40
1/1
1/5
063
101
0%
1/3
901
005
00%
103
101
41
0/S
0/3
101
0/
91
1/9
109
1/3
00%
10/
1V/
101
42
1/5
03
1/3
033
103
03
03
92
2/3
043
IV3
0/3
W/3
0/3
43
2/
03
1/3
63
03
00
03
03
98
1/3
0/3
0/
043
0/3
0/3
0/3
0/
46
0/5
0/5
1/1
2/5
2/1
45
1/5
0%
0/
6/
679
(11
04
0/1
95
4/1
V/1
1/1
129
(0)
0/3
(09
2/1
46
0/3
03
343
0/3
47
08
3/3
0/3
0/9
50
28
03
373
03
373
outside the bigger shape (cf. Figure 9 bottom left). Even though the final decisions of the models were primarily correct, we saw in the answers that except for Claude, none of the models was able to count the shapes correctly.
For the last BP. BP#37 (cf. Figure 9 bottom right), the models can identify the concept better, with some even classifying all 12 images correctly (cf. Table 12). The ground truth concept is triangle vs. circle on top and for this the perception seems to work more reliably. This could be because the concept of typical shapes positioned above or below one another is generally more familiar compared to other BPs. In this case, the challenge of solving the BP from scratch likely lies more pattern discovery or reasoning rather than perception.
For completeness, report image- wise classification results of the qualitative examples of BP#16 (Table 10), BP#29 (Table 11, BP#37 (Table 12) and BP#55 (Table 13).
B.4. Full vs. Single Images
To examine whether the gap between solving the BP in Task I and correctly classifying the underlying concepts in Task 2 arises from how the problem is presented (i.e., as a full image), we conduct an additional analysis using two of the top-performing models from Task 1. This time, we provide the models with the twelve individual images from the BP instead of a single composite image. The prompt remains largely unchanged, with only minor adjustments to the image introduction. We report the results in Table 14. The findings show that this change in setup does not significantly affect the number of correctly solved problems per task, nor does it reduce the observed gap between Task I and Task 2.
18

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Table 6: Results of each VLM on the individual Bongard Problems when provided with selection of 10 possible solutions (Multiple Choice (10)). Each model was prompted three times and the number of correct responses is reported (of 3). The reported ratios are marked based on four intervals: less than 1/3 (white), (1/3-2/3) (light green), (2/3-3/3) (green), 3/3 (dark green).
BPP
Clande 4P
Genini
Gemini
LLaVA
Qwen2VL
InteraVL.
GPT
Cluude
Gemmii 200
Genini
LLAVA
QwenzVi
IueeVLL
303
3/3
313
D3
2/3
2/2
3/3
2/3
2/9
0.3
0/3
1/3
3/3
3/3
1/3
3/3
3/
303
3/3
312
3/3
M2
3/
3/3
3/
56
Q.3
Q23
33
353
301
003
201
2/3
57
321
3/5
301
03
2/1
2/1
8
13
D/3
1/3
0/
0/
03
03
58
0/3
0/3
3/3
1/3
03
1/3
1/3
1/3
9
V1
1/1
2/%
1/1
1/1
V1
1/5
50
1/1
2/%
1/5
W11
065
1/
1/1
1/9
323
303
573
3/3
303
303
33
3/3
50
312
1/3
2/5
16
073
1/3
2/
11
33
3/3
2/3
1/3
1/3
23
36
61
3/3
3/3
3/3
3/
365
2/3
3/3
3/
3/5
1/1
1/5
301
1/5
211
3/3
303
2/3
3/9
0/3
2/3
3/3
3/3
L/3
2/9
2/%
2/1
0/3
1/3
2/3
1/5
221
3/3
3/3
3/3
0:3
0/3
003
303
03
68
3/3
3/3
2/3
3/3
2/3
19
33
3/3
3/3
273
0/3
1/8
113
1/3
89
3/3
2/3
3/3
28
2/
3/3
1/3
20
13
3/3
3/3
2/3
1/3
303
1/3
70
3/3
3/3
3/3
36
3/3
2/3
323
21
3/1
V1
223
2/%
4/1
1/3
4/1
2/5
71
1/1
1/X
1/5
2/1
1/5
10/1
0/1
1/1
22
3:3
DV3
213
0:3
0/3
D/3
1/3
033
72
312
323
L/3
2/3
3/3
0/3
03
(V3
23
33
3/3
3/3
3/8
1/3
5/3
3/3
73
3/3
1/3
2/3
2/3
08
0/3
1/3
0/3
2/1
301
1/5
2/1
313
2/9
2/3
3/3
3/3
3/3
V/1
2/1
2/9
V11
1/3
221
30
3/3
303
3/3
2/3
303
DK3
2/3
2/3
50
303
1/3
3/3
2/3
26
2/3
1/3
3/3
31
33
1/3
2/3
1/3
1/3
1/3
2/
Q6
R1
3/3
2/3
3/3
3/
1/3
2/3
1:3
92
VI
S/3
V0
273
271
7/5
82
2/1
1/5
WV3
1/5
10/1
1/3
173
33
3/
2/3
2/3
033
23
1/3
3/3
1/3
53
3/3
3/3
3/5
323
3/3
2/3
3/3
2/
34
3/
1/3
2/3
3/3
1/3
5/1
2/3
84
3/3
3/3
2/3
2/1
3/5
3/3
3/9
2/3
313
273
303
3/3
305
2/1
2/1
273
301
3/5
301
2/3
2/3
L/3
2/9
(09
30
373
3/3
3/3
3/
1/3
33
0/3
D/1
1/
3/1
1//
(V3
3/3
1/3
2/3
2/3
0/3
03
1/3
2/5
92
1/3
003
3/3
W33
2/3
1/3
1/3
33
3/3
2/3
2/3
0/3
3/3
3/3
2/
93
1/3
1/3
3/3
1/
1/
1/3
09
2/9
46
V/1
2/3
2/5
V0
273
1/5
941
301
3/5
3/5
1/5
1/1
1/1
2/1
45
3/
2/3
2/3
1/3
23
0/
03
2/5
95
3/3
3/3
3/3
33
2/
1/3
3/3
3/
33
1/3
0/3
2/
2/8
1/3
06
96
3/3
3/3
3/
3/3
3/3
3/
333
303
273
301
345
0/3
3/3
013
03
30
03
30
3/3
30
B.5. Error Types
We provide full visualization of all results over all tasks and including the different behavior types in Figure 10.
B.6. Task 3
The detailed results for Task are reported in Table 15 and Table 16.
B.7. Especially Hard Bongard Problems
Throughout the experiments, we saw that the investigated VLMs can solve some BPs better than others, where the set of solved puzzles appears to differ across task setting. Interestingly, we find that there are 10 BPs that are especially hard, i.e. they were not solved in any of the tasks. These are mainly BPs categorized as targetting spatial relations, such as points inside figure are on a line or not (BPH42). But also BPs from other categories are part of it, such as BP#82 that asks for the convex hull crosses forming an equilateral triangle (concept) and BP#58 where the solid squares are same size/not same size (same). In Figure 11 each of the 10 BPs depicted.
19

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Human
Human (Top 5)
o1
Solved BPs %
0.6
0.6
Claude 3.5
Gemini 1.5
Gemini
0.4
0.4
LiaVA-OV
Qwen2VL
0.2
0.2
InternVL. 2.5
0.0
0.0
all
size
concept
number
spatial
same
da0e 15
05
Humun
100
BP#
Figure 8: Human results compared results of VLMs. Mean results across all BPs (top left) together with mean results in different categories (top right). Number of BPs solved least once for different models (bottom).
Table 7: Mean percentages of solved BPs for humans and VLMs across categories.
Human
Human (Top S)
o1
GPT-4o
Claude 3.5
Gemini 2.0
Gemini 1.5
LlaVA-OV
Qwen2VL
Intern 2.5
53.17
56.67
66.67
27.78
38.89
5.56
0.00
0.00
0.00
0.00
concept
53.81
68.67
76.67
50.00
51.11
44.44
18.89
16.67
27.78
34.44
number
35.87
57.33
42.22
11.11
24.44
17.78
6.67
2.22
17.78
6.67
spatial
45.88
60.49
17.07
8.94
13.82
13.82
0.81
0,81
5.69
7.32
46.94
74.29
28.57
28.57
42.86
33.33
0.00
0.00
0.00
4.76
all
47.28
63.23
42.76
24.24
30.30
24.58
7.07
5.72
13.47
14.81
Which direction spiral turning?
Is the circle left or right of the cavity?
P
#16
#55
/
X
CW
X
left <
left
left
left
Claude
CW
CW
CW
CCW
right X
left
right
right
Gemini 2.0:
CW
CW
CW
CW
right
X
left
Left
X
left
InternVL2.5.
CW
CW
CCW
CW
right
X
right
right
left
Are more shapes inside or outside?
Which shape is on top of the other?
#29
#37
O
12
inside /
inside
outside
equal
triangle v
triangle <
circle /
circle
Claude 3.5:
inside V
inside
outside
outside
triangle
triangle
Circle V
triangle
X
Gemini 2.0:
inside
Inside
outside
inside
triangle
triangle
circle
circle
InternL2.5.. inside
outside
outside
outside
triangle
triangle
triangle X
circle
Figure 9; Task 2: VLMs fail to identify simple visual concepts. VLMs are challenged with identifying visual concepts in
BPs. Although the VLM is able to recognize some of the concepts when specifically asked for (bottom), on the others, continues to falter (top). Abbreviations used for clockwise (cw) and counter-clockwise (cow).
20

---

Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Table 8: Accuracy of each VLM when classifying the BP images in Task 2.
10M4
C2PI-4o
Chock 34
Gamini 1.0
Gamini
LaVAA-VV
Owen/VI
Inser/Vi. 2,5
RPW
CirE-io
Claste 3-5
Gamini 7.0
Genini 15 LLVA--N
QuenVVi.
IntenVI. 2.5
06.67
100100
100100
57.21
100,00
100,10
100.00
97.23
51
353.35
63:33
72.21
12:23
W6.11
$3.33
58.33
80.56
75.00
100.03
50.56
100.00
100.50
96.11
15.00
61.11
522
58.33
47.22
55.56
38,89
61.11
$2.78
66.67
52.78
100:00
100.00
$3.33
100,00
100.00
150,00
100,00
103.00
53
93.35
17.78
91.67
100,50
88.89
72.22
66.67
83.35
100:00
10000
100000
94.44
91.67
09.44
100,00
88.89
54
33.35
50.03
50.00
52.76
41.67
47.22
41.67
38,89
100000
00.00
100.00
100.00
100.00
100,00
100.00
100.00
35
50.00
50.00
50.00
41.67
41.67
50.00
50.00
41.77
100000
10000
$6.11
$3.33
83.33
83.33
91.67
91.67
56
91.67
91.67
56.67
13.00
91.67
50.00
58.33
75.00
100000
100.00
94.64
100,00
100.00
100,00
100,00
100.00
57
05.57
100,00
97.22
09.44
58.23
91.67
100.00
83.35
10000
100.00
94.68
100.00
105.00
100.00
100,00
102.00
56
75.00
66.07
K8.80
94.04
61.31
$3.53
75.00
71.32
91
$6.11
77.78
$6.11
91667
85.59
$1.67
63,80
59
83.35
56
88.80
69.54
63.59
91.67
75.00
80.56
10
83.33
93.33
72.22
72.22
63.33
94.4-
91.67
83.33
91.67
75.00
97.22
91.67
91.67
$6,80
75.00
80.56
V1
41.67
100000
18.00
K1.31
90.00
41.67
04.44
50.00
41.07
10000
100.09
41.00 67
$7.20
80.96
75.00
90.00
75.00
73.33
55.15
K0.56
75.00
66.46
10000
$3.35
94.64
61
66.67
90.00
50.00
61.44
81.35
15.00
07.21
94.44
100100
58 80
100.00
61.11
50000
77.78
58.33
$5.56
58.33
38.89
66.67
91
$3.33
100,00
72.22
75.00
58.35
50000
5000
$0.00
44 64
52.74
M0.00
50.00
50,00
83.35
K3.51
50.76
MM
58.33
6667
$0.56
$0.56
58.33
59.33
41.67
50.00
$0.00
36.89
1119
50,00
55.55
58.33
72.22
75.00
33
72.23
$7.53
50000
75.00
66.67
58.33
75.00
35.50
50,00
61.11
50.00
66.67
41677
50.00
56.33
21
66.6
66
44.64
100.00
75000
77.78
100.00
75.00
9L67
100,00
100000
10000
100000
66.67
61.11
66.67
100.00
100.00
100,00
$7.22
44.64
100000
100.00
100.00
$3.33
$0.00
47.22
41.67
100000
100.00
100,00
100,00
75,00
75.00
50.00
91.67
18.89
97.22
91.67
$4.64
$1.67
75,00
10000
10500
9107
04.44
100.00
100.00
A7
66.07
61:11
77.74
58.31
52.44
52.77
37.24
66.0
K1-31
33
1611
77-76
09.4-
$3.33
96.11
75.00
75400
13-33
N3.25
72.22
75,00
6607
91.6T
75,00
66.67
72.23
53:33
N3.25
80-56
10
77.78
75.0
41:22
63,89
58.33
57.7%
61.11
1724
36
10000
10000
100000
100200
100,00
100.00
100,00
100,00
100.00
75.00
97.21
7728
2224
83.23
83-33
73.00
37
10000
100,00
97.21
109.00
91.67
91.67
58.89
75:00
$3.33
22.4
75.00
20-56
50,00
38
10000
97.22
100,00
109.00
97.22
100.00
36.11
50.56
16
91.67
86.11
$3.23
61.11
39
10000
91.77
36 11
66.07
83:33
100.00
73.23
25.00
75.00
10000
83.89
16
59.23
40
58.31
$5.56
56.33
66.97
55.36
50.00
66.67
25.00
52.79
55-56
55.56
56-53
59.33
30.0
41
50.00
50,00
66.97
59.33
58.33
50.00
50.00
75.00
66.07
3333
13-33
95.11
27-78
66.67
43
40000
50.00
61.11
11
50.00
63.89
05.67
66.07
41W7
43
63-31
94-44
100.00
83:33
83:33
41.67
61.11
63.9
75:00
53.78
50.00
44
58.33
50.00
75.00
58.33
66.67
63.80
150.00
100.00
10200
10500
97:22
100.00
10000
35
6667
58.33
29.24
33.35
50.00
18.33
52.76
195
100L1))
100000
10000
10000
100000
100000
100/00
10000
46
7500
66.57
$0.56
61.89
35.00
70.23
90.00
61 11
46.44
72.74
67.28
75
47 91
$3.33
15.33
$1.07
72.76
$8.35
35.00
73.76
100.00
07:23
90.21
00.23
100
100.00
100.00
100.00
48
91.67
94.44
100000
100.00
108.00
61.11
100.00
94.44
98
06.67
91.07
100000
91
91-67
94.44
49
10000
63.09
80.56
$5.56
50.35
35.56
41.67
55.56
59
50.00
50000
55.56
50.76
63.89
09.60 64
56.33
55.56
50
10000
77.78
94.64
38.89
91.07
81.35
91.67
80.56
100)
91.67
63:33
100.00
100.00
100000
94.64
100/00
94.45
Table 9: The number of visual patterns that the VLMs can correctly identify is low (Task 2). Number of concepts of the single BPs that the VLMS were able to classify correctly for the 12 images of the BP. BPs either counted when for all 12 images 3/3 times the model was correct when it was at least 2/3 times correct for each image.
Task 2
Task 2
[ T2
[ n T2 | |/TI (Both 2/3)
Model
All 3/3
All 2/3
(Both 2/3)
o1
24
24
17
31.48%
GPT-4o
19
24
15
46.88 %
Claude
16
22
15
34.88%
Gemini 2.0 Flash
19
30
45.00% 18
Gemini 1.5 Pro
20
22
36.36% 4
LLaVA-One Vision
11
15
33.33% 4
Qwen2VL
20
20
37.50% 6
Intern VL2.5
10
16
34.48% 10
21

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Table 10: BP#16. Classification results when providing the single images of BP#16 and asking for clockwise counter- clockwise.
Clockwise
Counter-clockwise
5
7
10
11
12
o1
1/1
0/1
0/1
0/1
1/1
1/1
1/1
0/1
1/1
0/1
1/1
0/1
GPT-4o
3/3
3/3
3/3
3/3
3/3
3/3
0/3
0/3
0/3
0/3
0/3
0/3
Claude 3.5
2/3
3/3
1/3
2/3
3/3
2/3
0/3
1/3
0/3
1/3
0/3
1/3
Gemini 2.0
2/3
3/3
2/3
3/3
3/3
3/3
0/3
2/3
0/3
0/3
1/3
0/3
Gemini 1.5
3/3
3/3
2/3
2/3
3/3
3/3
0/3
0/3
0/3
0/3
2/3
0/3
Llava Onevision
2/3
2/3
3/3
1/3
3/3
2/3
Qwen2VL
3/3
3/3
3/3
3/3
3/3
3/3
0/3
0/3
0/3
0/3
0/3
0/3
Intern VL2.5
2/3
2/3
1/3
1/3
2/3
1/3
3/3
1/3
2/3
1/3
1/3
1/3
Table 11: BP#29. Correctly classified concepts of BP#29, Models were asked wether there are more shapes inside outside the big figure.
Inside
Outside
10
11
12
ol
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
0/1
1/1
1/1
1/1
GPT-4o
3/3
3/3
3/3
3/3
3/3
3/3
3/3
0/3
0/3
1/3
2/3
3/3
Claude 3.5
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
Geminí 2.0
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
1/3
3/3
3/3
3/3
Gemini 1.5
Llava-One Ceva--Onevisoon
3/3
3/3
2/3
3/3
3/3
3/3
3/3
2/3
1/3
3/3
3/3
3/3
Qwen2VL
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
0/3
3/3
3/3
3/3
Intern VL2.5
3/3
3/3
1/3
0/3
2/3
2/3
2/3
2/3
3/3
3/3
3/3
3/3
Table 12: BP#37. Correctly classified concepts for BP#37. Models were asked to output whether triangle or circle is on top.
Triangle
Circle
5
7
10
11
12
o1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/1
GPT-4o
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
Claude 3.5
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
0/3
Gemini 2.0
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
2/3
Gemini 1.5
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
Ilava Oneai-On
Qwen2VL
3/3
3/3
3/3
3/3
3/3
3/3
3/3
3/3
0/3
3/3
3/3
3/3
Intern VL2.5
3/3
2/3
3/3
2/3
3/3
3/3
3/3
3/3
1/3
3/3
3/3
3/3
22

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Table 13: BP#55. Correctly classified for concepts of BP#55. Models were asked whether the circle shape is located left or right from the cavity in the big shape (viewed from inside the shape).
Left
Right
2
3
4
5
7
10
11
12
ol
1/1
1/1
1/1
0/1
0/1
1/1
0/1
0/1
0/1
0/1
GPT-4o
0/3
3/3
3/3
3/3
0/3
0/3
3/3
0/3
0/3
3/3
0/3
3/3
Claude 3.5
1/3
2/3
0/3
0/3
2/3
0/3
2/3
3/3
1/3
3/3
Gemini 2.0
0/3
3/3
3/3
3/3
0/3
0/3
3/3
0/3
0/3
0/3
0/3
3/3
Gemini 1.5
0/3
3/3
3/3
3/3
0/3
0/3
3/3
0/3
0/3
0/3
0/3
3/3
Llava- Onevisonn
0/3
3/3
3/3
3/3
0/3
0/3
3/3
0/3
0/3
3/3
0/3
3/3
QWen2VL
0/3
3/3
3/3
3/3
0/3
0/3
3/3
0/3
0/3
3/3
0/3
3/3
Intern VI.2.5
0/3
3/3
1/3
3/3
0/3
0/3
3/3
0/3
2/3
1/3
0/3
2/3
Table 14: Comparison of model performance on Task and with original setup (full image) and single images as input.
Tl w/o T2
T2
T2 w/o TI
GPT-4o (orig)
13
11
13
GPT-4o (single images)
12
12
12
Claude (orig)
20
11
Claude (single images)
20
12
10
01
GPT4o
Claude 3.5
Gemini 2.0
Gemini 1.5
14
10
3
12
12
9
3
4
6
7
29
26
21
19
26
LLaVA-OneVision
Qwen2VL
InternVL2.5
7
23
6
Type
Type iII
21
22
32
Type II
Type
Open Question (Task 1)
Detect Concepts (Task 2)
Formulate Hypotheses
(Task 3)
Figure 10: Solved BPs across all tasks, Task 1, Task and Task 3, further highlighting the intersections of these, Type I, Type II, Type III and Type IV.
23

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
Table 15: Task 3: Number of correct hypotheses together with number of BPs for specific types.
Model
Task 3
[T n /TI
Type I
Type
Type III
Type
Type IV/
ol
72
90.91%
2
6
26
14
31.82%
GPT-4o
53
87.50%
2
6
12
9
37.50%
Claude
80.65%
6
16
29.03% 9
Gemini 2.0 Flash
75.00%
9
45.83% 11
Gemini 1.5 Pro
42
100.00%
6
6
57.14% 4
LLaVA Onevision
32
100.00%
6
60.00% 4
Qwen2VL
41
73.33%
0
8
5
40.00% 6
Intern VI.2.5
49
100.00%
0
5
7
41.67% 5
Table 16: Results of each VLM on Task 3. Each model was prompted once generate 20 hypotheses. The table reports whether correct hypotheses was included in the response (1) or not (0).
BFF
4P
Claude 33
Gemizl 2.0
Gemini 1.5
LEEW-OV QweezVL
InterVVL
BP
GPT-4o Clunde 3.5
Genini 2.
Genizi 1.5
LiaVA OV
QwenZVL
ImenVL 2.5
51
53
0
S4
0
11
12
62
15
81
14
64
15
63
22
72
23
73
24
8
74
25
73
36
76
13
23
83
14
84
15
36
86
17
45
93
44
94
45
1
99
46
96
47
24

---

Bongard in Wonderland: Visual Puzzles that Still Make AT Go Mad?
V
O
BP#42
BP#58 Left: Solid guadrangles identcal..
BP#64 Left: locuted exension of
Left: Points inside figure outline stright line.
Right: Pvints inside the figure outfine are not striight linc.
Right: Solid dark quadrangles are different.
Right: circle located the estension of the ellipse axis.
A
A%
4
%
A
+
A A444
O
BP#65
BP#74
BP#81
triungles elongated botizontally.
grows
figures
figures
stright line.
trinngles elongated veticallly.
Right: tall grows from
Right: figures impossible to separate.
A
K
X
*
M
A
r
W
A
++
BP#82
BP#85
BP#87 Left: Four parts.
Left: The conves hull of the crosses forms an equilatennl trangl..
Left: Three parts.
Right: Five parts.
Right: Five parts.
Right: The convex bull of the crosses does not foom
equilateral triangle.
BP#93
Left: Branches collined circle.
Right: Reanches solid dark circle.
Figure 11: Most Challenging Bongard Problems. Ten Bongard Problems (BPs) that were not solved by any model across our evaluations. High-resolution images adapted from (Depeweg et al., 2024).
25


Reviews:
[{"review_id": "Z8I70gJp77", "reviewer": "Reviewer_fqym", "summary": "The paper evaluates current VLMs on Bongard Problems. Each Bongard Problem (BP) consists of 12 images divided into two sides — the left side and the right side — each side containing 6 images. The images on each side are characterized by a rule not shared by the other side. The aim of the problem solver is to identify the rule pair that applies to each problem — via a text description of the rules. The paper develops 3 tasks: (1) Task 1 focuses on the open-ended generation of the rule given the problem image. (2) Task 2 focuses on providing the ground truth rule pair in the context and then asking the model to classify each of the 12 images into one of the two given rules. (3) Focuses on asking for multiple rule-pair hypotheses for each problem, unlike Task 1 where the model needs to generate the correct rule pair in a single go. The paper also evaluates human performance on Task 1. The paper then reports the results of these tasks for various available VLMs and infers several interesting conclusions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and questions, we address them below.\n\n**(W1 - Differences to Malkinski et al.)**\n\nMalkinski et al. (2024) also evaluated VLMs on Bongard problems, concentrating on open-ended and classification-based settings. Our work shares their open-ended focus but goes further by adding two additional tasks (Tasks 2 and 3) to examine specific abilities we hypothesize are vital for solving Bongard problems. While Malkinski et al. compare synthetic Bongard problems to their real-world variant (Bongard-RWR), we target the original Bongard set, pinpointing especially challenging cases and identifying concept-detection inconsistencies. We also compare model output directly with human performance, offering more granular insights into how VLMs reason about these complex tasks. We have added this discussion to the related work section for greater clarity.\n\n**(W2 - Takeaway messages)**\n\nIn the following we outline the key takeaway messages of our work.\n\n- We introduce a typology of model behaviors for visual tasks, revealing both the strengths and weaknesses of current VLMs.\n- Models often fail even with multiple-choice answers, indicating challenges not only in discovering correct rules, but also in recognizing when they have the right one (Task 3).\n- We uncover a pronounced gap (Fig. 5) between solving Bongard problems (Task 1) and consistently identifying their relevant concepts (Task 2), highlighting an underexplored interplay between abstract reasoning and perception.\n- We identify 10 Bongard problems that no model solved under any task condition (Fig. 11), offering a strong basis for future benchmarking.\n- Our human study shows key differences between VLMs and people: participants collectively solved 95 Bongard problems (averaging 42 each, with best participant at 68), whereas all models combined solved only 66.\n\n**(Q1)**\n\nUnfortunately, the training data for both closed-source and open-source VLMs is not publicly available, so we cannot determine this with certainty. However, given that these large models are typically trained on massive corpora that include a substantial portion of publicly available data, it is likely that they have been exposed to Bongard problems during training - especially since the Bongard problems, along with their solutions, are publicly accessible (e.g., https://www.oebp.org/welcome.php). It would be interesting future work to design versions of BPs that are private or could get generated automatically to avoid the public exposure.\n\n**(Q2)**\n\nYes, models were sampled three times. Tasks 1 and 2 were considered correct if answers/images were correct in at least 2 of 3 attempts. For Task 3, we sampled once and checked if a correct hypothesis was among the 20 proposed.\n\n**(Q3)** \n\nPlease refer to W1 of reviewer uWew for more details.\n\n**(Q4)** \n\nOur goal in Task 2 is to examine whether models that successfully identify a discriminative rule (Task 1) also apply that rule consistently to each individual image. We hypothesize that a truly correct solution should translate into accurate classification of all images in a Bongard problem. However, our findings show that a model can solve a Bongard problem in Task 1 yet still classify certain images incorrectly in Task 2. For instance, GPT-4o solves BP#15 - distinguishing open from closed shapes - in all three Task 1 trials but fails to label an open triangle as open in Task 2. This indicates that even when VLMs manage to solve a Bongard problem, they may not reliably apply their “understood” rule at the level of individual images. While asking open-ended questions about each image (rather than providing ground-truth concepts) is an alternative approach, our intention here is to assess the consistency between rule discovery and rule application under controlled conditions.\n\n**(Q5)** \n\nWe agree that a model may sometimes approximate the correct solution without consistently applying it across all images. However, this is not necessarily the behavior we seek in VLMs, particularly in applications that demand reliability and robustness. This highlights the importance of considering the perceptual aspect together with the reasoning abilities when evaluating VLMs.  \n\nThat said, our comment was pointing to a complementary finding: that some ground-truth concepts underlying Bongard problems are well-detected by the models, even though they do not emerge as the rule for solving the BP in the general task. This may stem from confounding visual factors or the limited prominence of some ground-truth concepts. We find this to be an interesting observation, and think that it opens up promising directions for future work.\n\n**(Q6)**\n\nThe figure is rather illustrative, however we intended to give an impression of different possible diagrams occuring in Bongard problems (displayed on the cards).", "claims_and_evidence": "Yes. The paper claims that there is a significant gap between VLMs and human performance and this is made quite clear from the experiments.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes, the experimental design is sound. I will ask any doubts in the \"Questions for Authors\" below.", "relation_to_broader_scientific_literature": "The paper is about evaluating the limitations of VLMs in how well they can understand abstract concepts. For this, the paper looks at the Bongard problems. Among the works about VLMs that look at Bongard problems, this work appears novel. I request further clarity from the authors regarding this via the Cons listed below.", "essential_references_not_discussed": "I am not aware of such references.", "other_strengths_and_weaknesses": "## Pros\n\n1. Interesting findings and questions raised for future work.\n2. The paper is well-written. The way the metrics are reported is clear. The experiment section provides a large number of new and interesting insights.\n3. Identifying the most difficult problems not solved by any model could be a very useful resource for the community.\n\n## Cons\n\nI think there needs to be a more in-depth discussion of the differences between this paper and the paper of Malkinski et al. 2024. I don’t mean to imply that there aren’t differences but rather that the reader should be able to see them more clearly. Currently, the authors say “While they (Malkinski et al. 2024) provide meaningful insights, they only consider a classification setting for the evaluation and do not investigate the model behavior in more depth.”, but this seems not specific enough. For instance: Malkinski et al.’s paper also does direct generation (akin to Task 1, if I am not mistaken). \n\nIt may be nice to mention which conclusions could not have been arrived at with previous work’s experiments and what are their implications for the future. The results are very interesting as standalone statements, but I am not sure if I have been able to grasp the main (and novel) takeaway message about VLMs from the paper.", "other_comments_or_suggestions": "See Cons and Questions.", "questions_for_authors": "1. Are the solutions to Bongard problems public and present in pre-training datasets of the VLMs?\n2. Were multiple outputs sampled from the models to ascertain that a task is solved (i.e., similar to pass@$k$)?\n3. Was human study considered for Task 2?\n4. Task 2 is about exploring perception. What is the rationale for choosing this specific form of perception task (e.g., why not ask in an open-ended manner about the concepts present in each image)? I am not sure if I see a clear direct connection with the conclusion noted later “perception is a key issue for not identifying the correct rules of BPs”?\n5. Authors say in L355: “surprising gap between recognizing correct classifications and effectively applying that knowledge in problem-solving”. However, a solver might try to “guess” a rule even if it seems to apply to the majority of the images (if not all) on each side. Wondering about the author’s thoughts on this.\n6. Is there a technical value associated with the contents of Fig. 1 (e.g., the person’s face)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKWzFJJvs4", "reviewer": "Reviewer_R3ti", "summary": "This paper explores the performance of VLMs on Bongard problems. To test the abstract reasoning ability of VLMs, three different types of tasks are proposed: (1) open-ended solving of Bongard problems, (2) detection of specific concepts, and (3) formulation of hypotheses. Task 1 is to summarize the rules of the left and right panels of Bongard problems, or to select the rules from some options. Task 2 requires VLMs to identify whether a certain image follows a certain rule or concept. Task 3 tests the model's ability to generate Bongard problem rules. The performance on the above tasks can evaluate the robustness and reasoning ability of VLMs. The experimental results show that there is still a large gap between the reasoning ability of VLMs and humans. This work provides valuable insights for evaluating the concept learning and abstract reasoning abilities of current VLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We respond to your points in detail below.\n\n**(W1 - Selection of test data)**\n\nWe chose to work with the original Bongard problems introduced in [1], as they were specifically designed to test pattern recognition capabilities in machines, yet they remain unsolved by current AI systems. These problems were carefully crafted and, in our view, already present a significant challenge on their own. Importantly, we find that they strike a valuable balance between simplicity (in terms of visual size and structure) and conceptual difficulty, a point supported by our accompanying human study.\n\nWe agree that datasets such as CVR and SVRT are also interesting, as they share some conceptual elements with Bongard problems, including shape variation, counting, insideness, and contact. Exploring how VLMs perform on these datasets would be an exciting direction for future work.\n\n[1] Bongard, M. (1970). Pattern recognition. New York: Spartan Books\n    \n**(S1 - Add abstract visual reasoning tasks)**\n\nThank you for the thoughtful suggestion. We agree that Raven’s Progressive Matrices and odd-one-out tasks are important forms of abstract visual reasoning, and we appreciate the idea of transforming them into open-ended formats. However, as motivated in the paper, we believe that Bongard problems already provide a strong and sufficiently challenging foundation for evaluating the kinds of open-ended, concept-based reasoning we are interested in.\nIn fact, many of the underlying visual concepts of Raven and odd-one-out (e.g., sameness, symmetry, numerical relations) are already well-represented in the Bongard problems.\n\nThat said, we agree that adapting other reasoning formats into an open-ended, explanation-driven setup as the reviewer suggests could be an exciting direction. While not necessary for the scope of the current study, we see this as promising future work, especially for expanding evaluation diversity and probing generalization across reasoning formats.\n\n**(Q1 - same BPs variance)**\n\nThe \"same\" category includes only seven Bongard problems (see Table 4), which naturally increases the variance of participants’ results and makes it difficult to draw strong conclusions. It is possible that some individuals find the concept of \"sameness\" more intuitive than others, though further study would be required to explore this systematically.\n\nIn general, high variance across participants may stem from differing levels of task comprehension. For instance, the lowest-scoring participant - who did not solve any Bongard problems in the ‘same’ category - often formulated a rule that applied only to the left side of a problem while dismissing the right side as simply “not following that rule.” In #BP98, they labeled the classes as “triangle shapes” and “non-triangle shapes,” overlooking the crucial contrast between “triangles” and “quadrangles.” Such answers suggest they may not have realized that merely stating the inverse of a rule is insufficient to fully characterize a Bongard problem. Similarly, this participant used relative terms like “more circle shapes” vs. “fewer circle shapes,” highlighting a misunderstanding of the need for clearly defined classification rules independent of context.\n\n**(Q2 - black area in legend)**\n\nThe legend entry for \"solved\" is meaning filled rectangle (red or blue) rather than black filling. We see that this can be misleading. To improve clarity, we consider updating it to a half-red, half-blue filled rectangle. Does the reviewer think that this would improve clarity?", "claims_and_evidence": "The claims made in this paper are clear and supported by its experiments.", "methods_and_evaluation_criteria": "The datasets and evaluation criteria of this paper make sense for the problem.", "theoretical_claims": "This paper does not involve theoretical claims and proofs.", "experimental_designs_or_analyses": "This paper is reasonable and effective in experimental design and analysis.", "relation_to_broader_scientific_literature": "The main contribution of this paper is to verify abstract visual reasoning ability of current VLMs on Bongard problems. Previous work [1] verified abstract visual reasoning ability of VLMs on non-open-ended problems like Raven matrices and odd-one-out problems, which is different from this work. This paper analyzed the concept learning ability in Bongrad problems, which is not covered by previous works.\n[1] Cao, Xu, et al. What is the visual cognition gap between humans and multimodal llms?", "essential_references_not_discussed": "The related works that are essential have been discussed in this paper.", "other_strengths_and_weaknesses": "Strengths\n\nThis paper refines traditional Bongard problems and proposes three different tasks, which respectively verify the rule induction, concept learning and rule imagination abilities of VLMs. The performance of the above tasks can reveal different dimensions of VLMs' abstract visual reasoning ability. Therefore, this paper provides some insights into the reasoning ability of commonly used VLMs.\n\nWeaknesses\n\nThe main concern is about the data collection process. In this work, only 100 Bongard problems from existing work are selected as test data. Could the authors provide a detailed introduction and discussion on the data collection process? For example, for what considerations and what criteria were used to select these Bongard problems. Why not use larger Bongard problem datasets, e.g., CVR [1] and SVRT [2]. CVR and SVRT include a large number of program-generated Bongard problems, which should have annotations of rule descriptions and concept labels for the panel images.\n\n[1] Zerroug, Aimen, et al. A benchmark for compositional visual reasoning.\n[2] Fleuret, François, et al. Comparing machines and humans on a visual categorization test.", "other_comments_or_suggestions": "As important forms of abstract visual reasoning tasks, the authors can further incorporate raven matrices or odd-one-out problems into the test. These problems can also be transformed into open-ended forms, like Bongard problems, by describing the rules to complete the task instead of choosing the result from the options.", "questions_for_authors": "I notice in Figure 4 that the accuracy of the human answers to the questions varied quite a bit (e.g., from 0% to 100% in the \"same\" rule). I wonder why the participants have such a large variance in their results. Is it due to the difference of participants' abstract reasoning ability, or because the participants do not understand the form of the Bongard problems?\n\nThe bar chart on the right of Figure 4 does not show the black area representing \"solved\", so what does the \"solved\" black block of the legend mean?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFtf7QObW2", "reviewer": "Reviewer_uWew", "summary": "The paper benchmarks existing vision-language models using Bongard problems (BPs). It also performs a human evaluation for comparison. The paper tests not only whether a model can solve a given BP or not, but also whether the main concept in the BP can be recognized in the individual images in the BP, and whether the model can generate the correct solution when it is asked to generate a set of candidate hypotheses. The results are surprising, and show, for example, that models not only significantly underperform humans, but they also do not seem to correctly perceive the individual images even in cases where they correctly solve a problem.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed response and the constructive feedback! Below we address your concerns.\n    \n**(W1 - Human study for Task 2)**\n\nWe agree that analyzing human performance in Task 1 alongside Task 2 would be an interesting future direction in a different setting with higher conceptual ambiguity and novel concepts. However, our primary focus in this paper was identifying the VLMs' perception errors in relation to their performance in Task 1. \n\nWe hypothesize that humans would excel in Task 2, as previous literature has identified concept recognition, spatial reasoning, and relational abstraction as fundamental aspects of human cognition [1,2]. Given that Bongard problems involve discriminative rules, we argue that human performance should remain robust in this task.\n\nWhile rule verification tasks can, in principle, be challenging for humans in cases of rule ambiguity or when unfamiliar concepts are introduced, this is unlikely in the case of Bongard problems due to the discriminative nature of the rules. For example, in BP#16, asking a human whether a spiral turns clockwise or counter-clockwise (starting from the center) should be straightforward, as the rule is well-defined.\n\n[1] Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behavioral and Brain Sciences. 2017\n\n[2] Gentner, D. (2003). \"Why we’re so smart.\" Language in Mind: Advances in the Study of Language and Thought   \n    \n**(C1 - Footnote o1 prompting)**\n\nThe o1 evaluation is quite expensive, therefore we decided to evaluate only once for the concept detection experiment, as it would required 2400 more requests. If the reviewer thinks, it would be valuable to have the additional trials for o1 as well, we can still retrieve these results. \n\n**(C2 - BPs in training Set)**\n\nUnfortuantly, the opaque nature of the training processes, particularly for models developed by large corporations with proprietary datasets, makes it impossible to determine whether the models have actually been exposed to the BPs during training. However, even if such examples were present, the low overall performance of the models indicates that they have certainly not fully comprehended them. Your suggestion that the models might have learned shortcuts, could be one explanation why many Bongard problems solved in Task 1 remain unsolved in Task 2. It would be interesting future work to investigate this phenomenon on non-public test sets. Overall, this is an interesting hypothesis, and we included it in our discussion.\n\n **(Q1 - Representation of BPs (single images))**\n\nInteresting suggestion, we have investigated this with some of the models (c.f. table below). We see that the performance in Task 1 is comparable but more interestingly also the behaviour between Task 1 and Task 2 stays similar. This suggests that the image representation alone cannot be the reason for this discrepancy. We included these findings in the final paper.\n\n|  | GPT-4o | Claude 3.5 | \n| -------- | -------- | -------- | \n| Solved BPs Original Setup     | 25     | 31    \n| Solves BPs Single Images | 25 | 33| \n\nResults analogously to Figure 5: \n    \n| | T1 w/o T2 | T1 $\\cap$ T2 | T2 w/o T1\n| -------- | -------- | -------- | -------- |\n| GPT-4o (orig)     |13 | 11 | 13 |\n| GPT-4o (single imgs)     |12 | 12 | 12 |\n| Claude (orig)     |20 | 11 | 11 |\n| Claude (single imgs)     |20 | 12 | 10 |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. The paper performs an evaluation of existing, pre-trained models on tasks derived from Bongard problems to compare to human baseline and to assess the performance consistency between the different tasks.", "theoretical_claims": "N/A \n\nThis is an empirical study.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper provides an overview of existing work on Bongard problems and similar tasks to evaluate AI models. The references seem fairly comprehensive as far as I can tell. The study in this work differs in substantial ways from existing similar studies.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is very clear, well structured, and easy to follow, and it presents lots of interesting insights. \n\nThe comparison between Task 1 (solving BPs) and Task 2 (detecting a BP’s underlying concepts in individual images) is quite nice and the results are revealing. And it seems in line with the findings for Task 3. It is nice that it also highlights a danger in reading too much into the ability to solve any given problem (using the Task 1 setting). \n\nContrarily, it seems that the danger also applies to human evaluations. And it suggests a human evaluation for Task 2 would be very helpful to complete the picture and would significantly strengthen this study. Have the authors considered this?", "other_comments_or_suggestions": "Footnote 3: “Exception: In Task 2 o1 was prompted once.” Why was that? \n\nEven though the performance of models is fairly low, as the problems are publicly available (and have been for a long time) is there any chance of contamination affecting the results? It seems that even the inconsistency regarding Task 1 vs Task 2 might be explainable to some degree through contamination as well (with Bongard problems - or similar types of problems - seen during training enhancing the ability to generate a shortcut answer without truly perceiving details of the images)?", "questions_for_authors": "The presence of multiple panels in a single image could be difficult for existing models to process simply because information is highly local as a result. And it seems that in some cases this could make the resolution in which a given model perceives the local panels too small to perceive details (some models downsample any given images to a fixed resolution, such as 224x224, before processing them).\n\nCould this be a confounding factor (especially for Task 2 and the results of Task 1 vs Task 2)? One way to help ensure that this is not the reason for the observed failure cases would be to present the panels separately, each in an individual image, rather than within a single. Have the authors considered this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGgsNphS82", "reviewer": "Reviewer_5FyS", "summary": "This paper presents a case study of utilizing VLMs for solving Bongard problems, and identifies that it remains challenging for VLMs to reason some basic concepts in Bongard problems. The authors also conduct a comparison between VLMs' and human's reasoning abilities on Bongard problems.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and for considering it well written. We hope that the following responses will also convince you of the strength and value of its contributions.\n \n**(W1, Q1 - No new method)**\n\nWe respectfully disagree with the reviewer’s assessment regarding the lack of novelty, and note the other reviewers did not raise any concerns on novelty (with reviewer fqym explicitly stating that \"this work appears novel\"). While we do not propose a new technique to solve Bongard problems directly (an ambitious and still unsolved challenge) we introduce a detailed typology of model behaviours in such visual tasks, revealing strengths and limitations of current VLMs, helping us to understand their learned knowledge and abilities to reason.\n\nOur contributions include:\n- Introducing various prompt configurations (open-ended and multiple-choice) to explore how different prompt strategies affect VLMs’ performance on Bongard problems.\n- Presenting two novel problem setups (Task 2 and Task 3) designed to analyze both the perceptual and the robustness aspects of VLMs.\n- Conducting a human study involving 20 participants, which allows us to compare human performance with that of VLMs and thereby uncover systematic discrepancies.\n- Identifying 10 Bongard problems that no deployed model in our study could solve in any task setup, highlighting clear gaps for future research.\n- Demonstrating a substantial discrepancy between correctly identified rules in Task 1 and those that are correctly applied in Task 2, which unveals inconsitant model behaviour as the VLM can corretly find a rule but not use it reliably.\n\nBy documenting and analyzing these insights, we aim to inform the broader ML community about current limitations and future directions in solving Bongard problems. We believe this approach is in line with the guidelines’ emphasis on supporting new tasks, metrics, and problem framings, even when no novel algorithmic methods are introduced. We believe this set of contributions constitutes meaningful progress that the ICML community can build upon.\n      \n**(W2 - Key findings have been discussed in literature already)**\n    \n Even though there exists other works that investigate the shortcomings in abstract visual reasoning in VLMs, we think that several new insights can be drawn from our work, see the listed contributions above. For example our work identified a gap between problem solving (Task 1) and perception or rather applying rules correctly (Task 2) that is, to our knowledege, not yet discussed in the literature. \n \n**(Q2 - Compare to other methods)**\n\nWe appreciate your interest in seeing broader evaluation results. However, our primary aim is not to identify the single best method for solving Bongard problems, but rather to reveal critical limitations and uncover how current VLMs, which can be opaque in their reasoning, perform on these tasks. We aspire to highlight systematic insights that can guide future improvements in model design and evaluation.\n        \nThat said, we can discuss other methods that aimed at solving Bongard problems in more detail in related work. However they usually have a very different setup to VLMs and our aim is not to propose a method for solving Bongard problems, instead we use them as a diagnostic dataset to understand and analyse VLMs in more detail. \n\nIn that light we also consider the proposed references. Both present interesting approaches; however, they focus on the Bongard-LOGO dataset rather than the original Bongard problems considered in our study. While the approach in [1] could, in principle, be adapted to the original Bongard problems in an open-ended setting, [2] is specifically designed for classification tasks, making it difficult to apply directly to the open-ended nature of the original Bongard problems. Moreover, our primary focus is on evaluating the capabilities of VLMs independently of dedicated reasoning architectures, as their underlying mechanisms differ significantly. Nonetheless, we have included both works in our related work section to acknowledge their contributions.\n    \n**(Q3 - more theoretical insights)**\n\nWe are a bit puzzled by this remark. As we have highlighted above the contribution of this work is not to propose a novel AI method, but rather introduce and analyze a valuable dataset and typology of model behaviour for investigating this dataset in the context of VLMs. Can the reviewer clarify what they are referring to with \"theoretical insights of the method\"?", "claims_and_evidence": "The authors mainly made four claims: 1) Evaluation of VLMs on identifying underlying rules. 2) Comparisons of VLMs to Human's reasoning ability. 3) Exploration of the models’ pattern recognition abilities. 4) Examination of the ability of generating hypotheses.\n\nThose claims are mainly associated with the experimental evaluation on the Bongard problems. The authors did conduct those evaluations as shown in Sec. 4.", "methods_and_evaluation_criteria": "The authors mainly made use of existing VLMs for the evaluation and did not propose any new method. The evaluation critera are clear and suitable.", "theoretical_claims": "There are no theoretical claims. The authors mainly conducted an empirical study of existing VLMs on Bongard problems.", "experimental_designs_or_analyses": "The authors mainly evaluate existing VLMs but no proposed one. Also, no other existing solution models for Bongard problems are compared in the experiments.", "relation_to_broader_scientific_literature": "The main contribution of this paper is to unveil that current VLMs are not able well solve Bongard problems. But the authors did not propose any new approaches to better solve the problems. In literature, there are many existing solution models to solve the Bongard problems. But the authors did not compare with those methods, which will greatly limits the significance of the any discussions or conclusions drawn from the results.", "essential_references_not_discussed": "Many existing solution models for Bongard problems are not discussed or compared in the experimental validaiton. E.g.,\n[1] Take A Step Back: Rethinking the Two Stages in Visual Reasoning, ECCV, 2024.\n[2] Neural Prediction Errors enable Analogical Visual Reasoning in Human Standard Intelligence Tests, IMCL, 2023.", "other_strengths_and_weaknesses": "Strength: The paper is well written with clear logic flow and comprehensive evaluation on VLMs.\nWeakness: 1) This paper did not propose any new methods. The novelty of this paper is very limited. 2) The key finding, the incapability of VLMs in solving abstract reasoning problems, including Bongard problems, have been discussed in literature. This greatly limits the significance of this paper.", "other_comments_or_suggestions": "The authors may add some theoretical analysis in this paper to boost the technical aspects of this paper.", "questions_for_authors": "1. In case I miss any novel design in this paper, can the authors justify the novel contributions of this paper?\n2. Can the authors provide more evaluation results of other existing models, in comparisons to the results of VLMs and human's? Or, can the authors justify why other existing models are not chosen for comparison?\n3. Can the authors provide more theoretical insights of the proposed method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}]

Author rebuttal:

Meta-review: This study evaluates LLMs on classical Bongard problems, which require human-like reasoning abilities, and finds that LLM behavior diverges significantly from human reasoning patterns. While one reviewer questioned the novelty of this work, others provided positive feedback and acknowledged its value. Given these assessments, I recommend accepting this paper.
Decision: Accept (poster)