Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems
Authors: Szymon Pawlonka
Mikołaj Małkiński
Jacek Mańdziuk
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models
Abstract Visual Reasoning
Bongard Problems
Primary area: datasets and benchmarks

Abstract: Bongard Problems (BPs) provide a challenging testbed for abstract visual reasoning (AVR), requiring models to identify visual concepts from just a few examples and describe them in natural language. Early BP benchmarks featured synthetic black-and-white drawings, which might not fully capture the complexity of real-world scenes. Subsequent BP datasets employed real-world images, albeit the represented concepts are identifiable from high-level image features, reducing the task complexity. Differently, the recently released Bongard-RWR dataset aimed at representing abstract concepts formulated in the original BPs using fine-grained real-world images. Its manual construction, however, limited the dataset size to just $60$ instances, constraining evaluation robustness. In this work, we introduce Bongard-RWR+, a BP dataset composed of $5400$ instances that represent original BP abstract concepts using real-world-like images generated via a vision language model (VLM) pipeline. Building on Bongard-RWR, we employ Pixtral-12B to describe manually curated images and generate new descriptions aligned with the underlying concepts, use Flux.1-dev to synthesize images from these descriptions, and manually verify that the generated images faithfully reflect the intended concepts. We evaluate state-of-the-art VLMs across diverse BP formulations, including binary and multiclass classification, as well as textual answer generation. Our findings reveal that while VLMs can recognize coarse-grained visual concepts, they consistently struggle with discerning fine-grained concepts, highlighting limitations in their reasoning capabilities.

Published as a conference paper at ICLR 2026
BONGARD-RWR+: REAL- WORLD REPRESENTATIONS OF TINE-GRAINED CONCEPTS IN BONGARD PROBLEMS
Szymon Pawlonka' Mikolaj Malkiński! Jacek Mandziuk
1 WWrsaw University of Technology, Warsaw, Poland
2 AGH University of Krakow, Krakow, Poland
(s [ssymon pawpppawllooooo onka. stud, mikolaj. m..kolaj.malknnsk.. mwwkinkkiiiii...mmmmmmmmmmmm .okt,
d jacek meedziman @pw. edu.pl
ABSTRACT
Bongard Problems (BPs) provide challenging testbed for abstract visual reasoning (AVR), requiring models to identify visual concepts from just a few examples and describe them in natural language. Early BP benchmarks featured synthetic black-and-white drawings, which might not fully capture the complexity of real- world scenes. Subsequent BP datasets employed real-world images, albeit the represented concepts are identifiable from high-level image features, reducing the task complexity. Differently, the recently released Bongard-RWR dataset aimed at representing abstract concepts formulated in the original BPs using fine- the-grained real-world images. Its manual construction, however, limited the dataset size to just 60 instances, constraining evaluation robustness. In this work, we introduce Bongard-RWR+, a BP dataset composed of 400 instances that represent original BP abstract concepts using real-world-lie images generated via a vision language model (VLM) pipeline. Building on Bongard- RRWR, we employ Tixtral-12B describe manually curated images and generate aligned with the underlying concepts, use Flux. to synthesize images from these descriptions, and manually verify that the generated images faithfully reflect the intended concepts. We evaluate state-of-the-art VLMs across diverse BP formulations, including binary and multiclass classification, answer generation. Our findings reveal that while VLMs can recognize coarse- grained visual concepts, they consistently struggle with discerning fine-grained concepts, highlighting limitations in their reasoning capabilities.
descriptions
well textual
INTRODUCTION
Abstract visual reasoning (AVR) domain (Stabinger et al., 2021 van der Maas al. 202T) refers to visual tasks, solving which requires identifying and reasoning about abstract patterns expressed through image-based analogies. Classical AVR tasks and associated benchmark problems include Raven's Progressive Matrices (RPMs) (Barrett et al. 2018 Zhang et al. 2019), visual analogy problems (Hill et al. 2019: Webb et al., 2020), and more (Matkiński & Mandziuk] (2023). These tasks mainly focus on testing the ability of systematic reasoning and generalisation to new feature distributions. Nevertheless, they typically require supervised training on large-scale datasets. This stands in contrast to human intelligence assessments, which focus on rapid adaptation novel, potentially never-encoumtered problems based on prior knowledge.
distinct alternative to conventional AVR benchmarks is offered by Bongard Problems (BPs), originally proposed in 1970 (Bongard 1970). Each BP consists of two sides, each containing six images, separated according to an abstract rule. The solver's task is to infer this underlying concept and articulate it in natural language (see Fig. D. An alternative BP formulation relies on classifying novel test image (or a pair of test images) to appropriate side(s). Despite its deceptively simple format, the BP setting poses several unique challenges. First, it naturally constitutes few-shot learning problem (Fei-Fei et al.) 2006 Wang et al.] 2020b) identifying the separating concept requires generalization from just six examples per side. Second, the interpretation of images is inherently contextual (Linhares 2000). For instance, in Fig. [a| a visual feature such as curvature may appear salient on the left side, but only through comparison with the opposite side the true rule, difference

---

Published as a conference paper at ICLR 2026
in arrow directions, becomes evident. Third, solving BPs requires integrating visual perception with text generation, demanding bi- ti-modal reasoning capabilities.
Early BPs were created manually, resulting in only few hundred instances contributed by individuals (Foundalis 2006b). To scale beyond this limitation, Bongard-LOGG LOGO (Nic et al. (020) introduced a procedural generation method based the action-oriented LOGO language, enabling creation of a large set of problems. While this approach provided sufficient data for training deep models, the resulting BPs were restricted to synthetic black-and-whit drawings. Subsequent efforts have shifted BPs into the real-world domain. Bongard HOI (Jiang et al. 2022) utilized natural images of human- object interactions. Bongard- Seeerrdd-peWWoorrt World (Wu et 20044 employed an concepts to capture the complexity and ambiguity of real-workd scenes. Most recently, Bongard Bongard-RWR RWWR (Mafkiński et al., 2025) linked synthetic and world domains by using real images to represent abstract concepts found in synthetic BPs, facilitating direct comparison of model performance across both domains. Bongard-RWR, which focuses on abstract concepts, poses a greater challenge to contemporary models than Bongard HOI and Bongard- Soaaurd-ppeWWolddd which involve real-world concepts, despite all three using real-world images (see Fig. 2 for an illustration). However, since Bongard-RWR constructed manually, its small scale limits the robustness and breadth of possible evaluations, highlighting the need for a more scalable approach to dataset construction.
of free-form
To overcome the scalability limitations of Bongard -WRR, introduce Bongard -Rrar, new benchmark featuring real-world representations of selected synthetic BPs (see an overview in Table Unlike Bongard-RWR, which was constructed manually, our approach leverages recent advances in vision language models (VLMs), including Image-to- texxtt (I2T) and Text-to-Image (T2I) models, to automate dataset creation. For each BP the original Bongard-RWR, we (1) use Pixtral-12B (MistralAT 2024) to describe each image, (2) generate new image descriptions aligned with the matrix concept, (3) employ Flux.1-dev (Labs) 2024) to synthesize images from these descriptions, and (4) manually verify that generated images reflect the intended concept (see Fig.B) This pipeline enables construction of dataset comprising 400 BPs with the original abstract concepts preserved.
We conduct a comprehensive evaluation on Bongard-RWRK to systematically assess the reasoning capabilities of state-of-the-art VLMs across a range of problem setups. Specifically, we formulate: (I) binary classification tasks, where the model assigns either single test image pair of images to the correct side(s) of the matrix; (2) a multiclass classification task, where the model selects the concept that best matches the BP matrix from a set of candidates; and (3) a free-form text generation task, in which the model articulates the underlying concept in natural language. Beyond these primary tasks, we perform several ablations investigate factors influencing model performance. These include examining the effect of model size, comparing color vs. greyscale inputs, contrasting model performance on BPs constructed with real generated images, and varying the number of images per matrix side. Our findings show that, while current VLMs exhibit some capacity to identify high-level, coarse-graie goarse-grained concepts, they consistently struggle with discerning fine-grained concepts, highlighting gaps in their visual reasoning capabilities.
Contributions. In summary, this paper makes the following contributions:
1) We develop semi-automated pipeline to generate real-world-lin images of abstract concepts.
2) We introduce Bongard-RWi RWRt+ a new BP benchmark of 5 400 matrices generated with this pipeline.
3) We conduct extensive evaluations on the proposed dataset, demonstrating that current VL.Ms struggle to identify fine-grained visual concepts, revealing important limitations in their multi-image AVR abilities.
2 RELATED WORK
Overview of AVR challenges. Analogy-making is a cornerstone of human cognition, closely tied to fluid intelligence-thh ability to generalize learned skills to novel situations (Lake et al. 2017). the visual domain, such tasks present demanding challenges for representation learning (Chalmers et al.) 1992) and highlight the need to integrate perception and cogmition (Hofstadter
1995). motivating
diverse AVR benchmarks. Among these, RPMs (Raven 1936; Raven & Court
1998) are often
studied problem type (Hoshen & Werman 20177 Matkinski & Mañdziuk] 2024b
20244
2025 Mandziuk &
Zychowski 2019). typically involving small set of logic-based rules. Datasets like SVRT (Fleuret
2

---

Published as a conference paper at ICLR 2026
2
S
SOU
2
1
S
$
2
(a) Synthetic BP #52
(b) Bongard-RWV RWRR
(c) Bongard- Bongard-RWR+ (ours)
Figure 1: Bongard Problems. All matrices present the same abstract concept: Left side: Arrows
pointing in different directions. Right side: Arrows pointing in the same direction. (a) A manually- designed synthetic BP (Bongard 1970, Foundalis, 2006b). (b) manually-designes real-world
representation from Bongard- -ong RWR (Malkinski et al. 2025). (c) An automatically generated real-
world representation from Bongard-RWRS
8
P
d
q
A
20
V
4
3
D
C
4
4
4
X
(a) Bongard-LOGGO
(b) Bongard HOI
(c) Bongard- Open Bongard-OpenWorkk
Figure 2: Related BP datasets. (a) A synthetic BP from Longard-LOGe Nie et al. 2020): Left:
Shapes are the same. Right: Shapes are different. (b) A real-world BP from Bongard HOI (Jiang et al. 2022;; Left: A person driving a car. Right: Not a person driving car. (c) A real-world BP from Opeaard-Opeoorrre (Wu et al., 2024); Left: The top of snow-covered mountain. Right: Not the top of a snow-coverds mountain. Unlike Bongard-LOGO, which involves synthetic images unfamiliar to VLMs, or Bongard HOI and Bongard-Open World, which focus on coarse-grained concepts, Bongard- + is designed around abstract concepts expressed through realistic images that require fine-grate grained visual reasoning.
ctt al. 2011) and those based on BPs (Jiang et al. 2022; Malkiński et al. 2025 Nie et al. 2020
Wu et al.] 2024) introduce more abstract and diverse rules expressed through visual analogies in few-shot learning framework. Another line of work has proposed generative challenges, such as
ARC (Chollet 2019) and PQA (Qi et al. 2021). Several benchmarks (Bitton et al. 2023 Ichien et al.) Teney et al. (2200) construct AVR tasks using real-world images, enabling evaluation of models pre-trained large visual datasets. Our proposed dataset builds on this landscape by combining few-shot learning with real-world-lic images and supporting both classification and free-form text generation tasks, offering a diverse and challenging testbed for evaluating AVR models.
2021
BP solution methods. In response the wide range of AVR challenges, various approaches have been developed (Hernández-Oraal Orallo et al.) 2016 Malkinski & Mandziuk] 2024a Mitchell 202T). Specifically, for BPs, methods include program synthesis and inductive logic programming (Saito & Nakano 1996; Sonwane et al. [02I]. cognitive architectures (Foundalis) (006a), Bayesian infer- ence (Depeweg et al. 2018] 2024), convolutional networks (Kharagorgiev 2018), and meta-learning approaches such as prototypical networks (Snell et al. 2017). SNAIL (Mishra et al. 2018), or Meta- Baseline (Chen et al. 2021). Recently, LLM-based solutions combine image captioning models with
LLMs for text descriptions (Wu et al. 2024) or leverage VLMs that handle both modalities (Matkinski et al. 2025 W Wūst et al. 2025). To establish strong baselines, we evaluate state-of state-of-the-a th-tte-rrtt VLMs. Additionally, prior work has framed BPs in various settings, such binary classification where test images are assigned to matrix sides (Nie et al., [220 Kharagorgiev 2008),, or free-form text generation where the model articulates the underlying concept (Matkiński al. 2025 Wu et al.
3

---

Published as a conference paper at ICLR 2026
Table BP datasets. Bongard-RWR+ offers a broad set of matrices that uniquely combine generated real-world-liks images with abstract concepts.
Dataset
Images
Concepts
# Matrices
# Concepts
Synthetic BPs BBngard 1970
Synthetic
Abstract
394
388
Bongard-LOGee (Nie cf al. 2020
Synthetic
Abstract
12 000
627
Bongard HOI (Jiang al. 2022
Real-world
Real-world
53 000
242
Bongard- Langard-Open
Oaeee-peeWWWaaeee World (Wu et al.
2024)
Real-world
Real-world
010
1010
Bongard-RWFF (Malkinski
2025
Real-world
Abstract
60
55
Bongard- (ours)
Generated
Abstract
5 5400
49
12T
T2T
T2I
(CL,CR): (Large figures, Small figures)
A city view with tall buildings.....
T2I
A centered skyscrapper...
Ny A wind turbine...
BP
Describe
Augment
Generate
Review
Figure 3: Generative pipeline. Starting from a Bongard problem BP with concept (CL, Cn). the
pipeline: (1) describes each image using an 12T model to produce paired positive/negative captions L
and C (2) augments each positive caption with T2T model into N diverse descriptions { { 1. that preserve the underlying concept; (3) generates candidate images for each new description using T21 model; and (4) involves a human judge to review and filter the generated images. For readability, the figure illustrates the processing flow for the first image from the left matrix side.
! j=1
2024 To systematically evaluate model capabilities, cover diverse setups, including image (or description) to side classification, a multiclass concept selection task, and free-form text generation.
Data generation with T2I models. Generative T2I models have gained significant attention for their broad application. Prior work has applied T2I models to generate synthetic data for supervised learning (Fan et al. 2024 Ravuri & Vinyals 2019 Sariyildız et al.) 2023). contrastive representation learning (Jahanian al. [022] Tian al. |024]2222222 and data augmentation (Trabucco al.) 2022).
In contrast, we employ pre-trained T2I model to automatically generate real-worid-like images
depicting predefined abstract concepts, enabling scalable construction of Bongard- instances.
METHODS
To advance research on abstract reasoning, we introduce Bongard-RWR+, a dataset comprising 5 400 BPs in its main setting, with additional instances in ablation variants. The dataset supports range of task formulations, including binary and multiclass classification, and free-form text generation. Each instance is defined as BP (L, R, T, C'), where L (LI...., Lp} and R {R .R....., Rp} denote the left and right panel sets (each with P = 6 images), T {Tı, Tn) contains two test images (one per side), and C' (CL, Cn) specifies the underlying concept in natural language, with Ct,CCR describing the concept shared by all images on the left / right side, resp.
3.1 BONGARD-RWR+
To generate Bongard- we considered each BP from Bongard-RWR. For each image L; E L and R, E R, we generated paired positive and negative textual descriptions based on the side't con- cept: Describe L, CL) ( C and Describe R,, CR) (R, R2 using Pixtral-12B (Mis) tralAl| 2024). This 12T model was prompted to produce positive descriptions that faithfully captured the image's content and negative descriptions designed steer the T21 model away from depicting
4

---

Published as a conference paper at ICLR 2026
Cx
VLM
VLM
(a) Image-to-Sie Side (IIS)
(b) tmadessto-Sides (I2S)
(e) Concept Selection (CS)
R1
R2
1
R1
R3
T2T
VLM
R
(c) Description-to-Side (DIS)
(d) coseriptionssto-Sidee (D2S) (f) Concept Generation (CG)
Figure 4: BP formulations. We define six tasks of variable complexity: (a; IIS) assign a single test image to the left or right side; (b; 12S) assign a pair of test images to the respective sides; (c; DIS / d; D2S) use descriptions from an 12T model and classify with a T2T model; (e; CS) select the correct concept index lt such that C = C*; (f: CG) generate a natural language description of concept C.
the opposite concept. Each positive description was augmented with a Text-to- -ext (T2T) model into N = 15 alternative descriptions reflecting the side's concept: Augment( Cr. = {
N ) Each generated alternative description, paired with its corre- sponding negative prompt, was passed to the Flux. I -ddev model (Labs) 2024) to render 512 x 512 candidate images: Render (c,, ;. 2
and AR, Cn))
= Ligj and Render(RR,, R2 )= -R.j- Appendix F describes the employed model prompts. All generated images were manually reviewed to ensure they accurately reflected the intended concept (C'z or CR) without introducing elements from the opposite side's s concept. Images failing this criterion were discarded. Appendix E.2 presents further details.
From the candidate images } and {Rt,), J. we composed M = 10 left and right sides, denoted {I') and {R' R 1, by iteratively selecting subsets that maximized intra-set visual diversity. Each subset L', C {L, and R" C. {RE,,) contained 7 images (6 context images and test image), chosen iteratively to minimize total pairwise cosine similarity of ViT-L/14 embeddings (Dosovitskiy et al. (021]. To cover more images and increase dataset diversity, in each iteration we selected random image from each subset and removed it from the overall image pool the next iterations. Finally, each left side L'm, was paired with each right side R" to construct M 100 new BP instances aligned with the original concept. Algorithm details are explained in Appendix B Using this pipeline, we constructed 5 400 BPs corresponding to 54 Bongard- RWR matrices (IO0 new problems per source BP). The 6 remaining Bongard- RWRR matrices were discarded due to difficulties in generating a sufficient number of images faithfully depicting the underlying concepts.
We additionally introduce several dataset variants to support ablation studies. The first variant, Aingard-RWR//GS. consists of grayscale matrices. Since Bongard- RWR++ concepts are primarily structural, as they originate from black-and-white synthetic BPs, this variant isolates the role of color, which is expected to be non-essential for concept detection. The second variant, Bongard-RWR+H (P = 6), e.g., denoted Bongard- -orrd--RR+/L222 for P = 2, modifies the subset construction method by omitting the image removal step. In effect, contains fewer unique images (1 503 in Bongard-RWRRI/ RWR+/L6 vs. 4 157 in Hungard-RWR++ but exhibits lower average intra-side embedding similarity. Lower similarity implies greater visual diversity within a matrix, meaning the same concept is illustrated through more varied content for example, the concept "Vertical" may be instantiated as a tree, a building, or a standing figure-poter potentlpoy making the concept easier to identify. In contrast, high similarity may reflect repeated visual aspects (e.g., several images of trees), which may hint at unrelated concepts like "Nature" or "Green", making the intended concept harder to capture. We consider Bongard- Bongard-RWR+11 RWR+/LP variants with varying numbers of images per side (P), enabling analysis of how the number of demonstrations influences model performance. All variants are derived from the same image pool as the main dataset. Additional details are provided in Appendix
5

---

Published as a conference paper at ICLR 2026
3.2 PROBLEM FORMULATIONS
We cast BP into several task formulations of increasing complexity, as illustrated in Fig. 4 begin with binary classification settings. In the Image-to-Side (IIS) task, the model classifies single test image (Ty or TR) as belonging to either the left or right side. The Images-to-Sides (I2S) variant extends this by requiring the model to assign a pair of test images (Tz and TR). each from a different class, to their respective sides. To assess the effect of an intermediate image captioning step, we introduce Deseription-to-Side (DIS) and Descriptions-to-5
We
(D2S). These tasks first convert images into natural language descriptions using an 12T model: Describe [.) L.. Describe Rt) = R., (escribe(T.)) = T.. Vi E (L...., PJ. Vs E {L, R3. The resulting descriptions are passed to a T2T model for a binary classification, either for a single test description (DIS) or pair (D2S). In practice, I2T and T2T models may be implemented by a single VLM that processes both visual and textual inputs, though T2T can also be realized by text-only models (e.g., LLMs).
Beyond binary classification settings, we introduce the Concept Selection (CS) task, a multiclass classification setup where the model selects the correct concept C* describing the matrix from candidate set of concepts {C* K (C* E {Cx)). with K controlling task difficulty. To construct distractors (fCx C2 + C²),, we sample concepts from other matrices, ensuring that each candi- date represents a distinct concept to avoid ambiguity. Finally, we consider the most challenging formulation, Concept Generation (CG), where the model is required generate a free-form textual description of the concept underlying the BP matrix.
4 EXPERIMENTS
We evaluate 4 state-of-the-art open-access VLMs on Bon: InternVL2.. 78B
Randon goess
(IVL2.5) (Chen et al. 2024a bb. Qwen2- VL
72B- Instruct (Q2VL) Bai et al.
2023 Wang
06
2024), LLa Next 110B (L. VA) Tiang al.
04
et al.
2023 Liu et al. 2024a b). and MiniCPM-
2.6 8B (MCPM) (Yao et al. 2025] Yu al..
4
8
16
8
2
16
8 16
2025). Main experiments use the largest avail- able models, with smaller variants covered in ablations. All models are evaluated at a fixed decoding temperature of 0.5, using structured ISON output format enforced via the Outlines decoding backend (Willard & Louf 2023). Inference was performed on an internal computing cluster equipped with NVIDIA DGX A100 and H100 nodes. The largest models required up 24 hours of inference per task using 4 H100 GPUs.
IVL22.5 78R
Q2-VL 728
LLAVA 1108
MiniCPM
Figure 5: Concept Selection. Accuracy in the CS task on Bongard-RWR++ for K E {2, 4, 16).
We developed a Similarity Classifier (SC) to serve as non-parametric baseline. For image-based setups (IIS and 12S), it computes ViT-L/14 embeddings (Dosovitskiy et al.. 202 for all matrix images and the test image. For text-based tasks (DIS and D2S), embeddings of image descriptions are obtained with fine-tuned MiniLM (HuggingFace 2021 Wang et al. (020a)) Next, for each matrix side, the embedding that is most distant from the test embedding (based on Euclidean distance) is identified. The test input is then assigned to the side with the lower maximum distance (i.e., higher similarity). Appendix C.8 explains the reasons of SC strong performance.
Concept selection. We begin with the CS task, where the model selects the correct concept from a candidate set of size K E {2, 4, 8, 16). Increasing K reduces accuracy, confirming the growing difficulty of the task (Fig. 5). InternVL2.2 performs best across all variants. correctly identifying the concept in 91 % of cases for K = 2. However, its accuracy drops to 57% for K = 16, indicating that even the strongest model struggles when faced with multiple distractors. Meanwhile, both MiniCPM-o 2.6 and LLaVA-Next achieve only 19% for K 16, underscoring their limited capacity even in this discriminative setting, and highlighting the utility of the CS task as a benchmark for AVR. Notably, MiniCPM-o 2.6 matches VA-Net'ss performance despite a much smaller parameter count (8B vs. 110B), suggesting that size alone is insufficient to excel in the introduced challenge.
To better characterize model behavior, we categorized Bongard-RWRt matrix concepts into 9 semantic groups reflecting the key factor behind each concept: Size, Position, Count, Branching, Similarity, Contour, Shape, Rotation, and Angle. This grouping follows recent efforts advocating

---

Published as a conference paper at ICLR 2026
Intern 78B
Qwen2-VI. 72B
LLaVA-Next 100B
MiniCPM-o 2.6 8B
K
Size
Size
Size
Size
Aapk
Fosition
Pesition
Postion
Fostion
Rutalies
10
Robaiien
Rotasian
Rotatist
count
Dd
Count
Count
Oxunt
Shap
Shape
Shaple
Shape
Beaaching
Brunching
Contrur
Contour
Conooa
Contii
Siniluslt
Sunlurit
Similarity
Similaity
Figure 6:
Concept Selection. Accuracy in the CS task on Bongard-RWR+ for K E
for semantically grounded evaluations to isolate model strengths and weaknesses (Mitchel] 2021 Odouard & Mitchell 2022). The results are shown in Fig. [ For K 2. all models exhibit broadly similar profiles across concept groups. However, for more challenging setups (K >> 2), differences emerge. For instance, V1.25 retains high performance on Shape, Size, and Branching for =1,6, with accuracy near 75%. These categories involve high-level visual attributes that VI.Ms appear to capture reliably. In contrast, Intern VL2.5 accuracy drops below 50% for Contour, Rotation, and Angle. These concepts often rely subtle visual cues (Contour) or precise spatial relationships (Rotation and Angle), which current models struggle to capture reliably. We expand on this in Appendix [C10] Overall, performance patterns remain consistent across values of K, supporting the utility of such analysis as diagnostic tool for characterizing model capabilities.
Next, we consider the IIS and I2S tasks, where models must assign one or two test images, resp., to the correct matrix side. Notably, in I2S (and D2S). the model per- forms two independent binary classifications; while the prompt specifies that the test inputs belong to different classes (ma- trix sides), this constraint is not enforced the output format, causing some models to incorrectly assign both inputs to the same side. The results are shown in Table 2 For both tasks, VLMs performed below random accuracy in I2S. Strikingly. the simple SC base- line outperformed all VLMs, suggesting that the tested models fail to robustly identify the underlying concept that separates semantically the matrix sides. Compared the CS task, where models can rely on high-level heuristics to eliminate implausi- ble options, IIS and I2S require a deeper understanding of the depicted concept correctly classify
(mage(s)-to-sidees
Table 2: *-to-Side(s) classification. Results with IIS, 12S, DIS, and D2S on Bongard- -ongard-RWR+. Captions in DIS and D2S were produced with Intern V1.2.5 78B.
11S
12S
DIS
D2S
near chance, with several models falling
IVL2.5
0.50
0.39
0.57
0.49
Q2VL
0.49
0.44
0.58
0.42
LLaVA
0.50
0.50
0.54
0.43
MCPM
0.48
0.45
0.51
0.41
DS-RI
N/A
N/A
0.57
0.56
SC
0.52
0.54
0.49
0.50
new inputs. These results highlight a fundamental limitation in current VLMMs* AVR capabilities.
Given the difficulty of the introduced dataset, we hypothesized that decomposing the problem via an in- termediate captioning step may improve model performance. To test this, we generated image descriptions using each of the 4 VLMs and used these captions as input to the same set of models for solving the side prediction task. Since the prediction task is purely text- based, we also evaluated the DeepSeek-RI 70B (DS-RI) reasoning model (Guo et al.] 2025). which does not support visual input ("N/A" for IIS and I2S in Table 2) but offers strong language understanding. As shown in Fig. 7 final accuracy in the DIS task varies depend- ing on both the captioning (rows) and prediction (columns) models. Descriptions produced by InternV1.2. yield the highest overall ac- curacy, with Qwen2-VL also producing relatively strong captions. Interestingly, captions from MiniCPM-o 2.6 lead to better results than those from LLaVA-Next, despite MiniCPM-o 2.6 being much smaller, highlighting its promising image-to to-ge-to-text capabilities. Among prediction models (columns), Qwen2-Ve VL achieves the best average performance, followed by Intern indicating relatively strong text-based reasoning abilities of both models. DeepSeek-RR while known for strong perfor-
Description(()-tosiieee
IVL22.
QIVI.
1LAVA
MCPM
DS-R1
0.55
0.56
0.52
0.50
0.55
0.51
Q2VL. 0.54
0.56
0.57
0.33
0.50
0.55
LLAA
0.54
0.54
0.50
0.50
0.52
0.52
MCPM 0.5
0.55
0.55
0.51
0.50
0.54
Figure 7: Description-to- Side. Accuracy in the DIS task on Bongard-RWR+ for a pair-wise combination of I2T description models (row-wise) and T2T prediction models (column-wise).
7

---

Published as a conference paper at ICLR 2026
1.0
1.0
0.8
0.5
HEAD
04:00
0.6
0.47
0.48
0.6
44
0.28
0.24
0.17
0.19
0.19
0.4
06.30
0.08
8B
268 360
78B
7B
726
328
728
F10
SB
2
4
8
16
4
6
16
8
16
4
16
IVL25
LLaVa
MMnCPM
IVL2.5
VL 72B
LLaVA 1108
MinKFPM
(a) Varying model size on Bongard-RWR+
(b) Bongard-RWR
Figure 8: Concept Selection sensitivity analysis. (a) Accuracy of models of varying sizes for
K 16. (b) The impact of using grayseale images for K (2, 4, 8, 16). Differences w.r.t. color
images are shown using lighter colors and are annotated in parentheses above the corresponding bars.
VL2. 78B
Qwen2-VL VL 72B
Next 110B
MiniCPM-o
-3300
-000
-01
005
16
4
16
2
8
16
16
16
16
Figure 9: Functional equivalence of generated images and real-world images. Performance trends the CS task (K E (,,4,8116) on Bongard- -WWR and Bongard-RWR+1L3 for P = 2,3 panels
mance on standard benchmarks, ranks third, revealing potential limitations in reasoning over image descriptions. Using captions generated by the best pest-performin model, Intern VL2.5, we report DIS and D2S task performance in Table Compared to IIS and I2S, most models perform above chance in these settings, confirming that the intermediate captioning step helps models better ground their predictions.
Does AVR performance scale with model size? Prior works have shown that VLM performance tends to scale with model size (Hoffmann et al. 2022 Kaplan et al. 2020). To examine whether this observation holds for AVR, we evaluated smaller model variants in the CS task on Bongard-RWRR Specifically, we tested 8B, 26B, and 34B variants of Intern IL2emV 7B, 32B, and 72B versions of LLaVA-Next and 7B variant of Qwen2-VL. We excluded additional MiniCPM-o 2.6 variants, as the 8B model already showed weak performance and no larger variants are currently available. As shown in Fig. [aa accuracy generally increases with model size. Among the smallest configurations (7B/8B), Intern VL.55 performs best, followed by Qwen2-VL, mirroring the relation observed for their largest versions (70+B). Notably, both Intern VL2.5 8B and Qwen2-VL. 7B outperformed MiniCPM-o 2.6 8B, underscoring the generally stronger AVR capacity of these two model families.
Is image color necessary for concept recognition? To assess the role of color in abstract reasoning. we evaluated all VLMs on Bongard-RWR+/GG using the CS task. As shown in Fig. 8b model performance remains comparable, or even improves, on grayscale images. For instance, LLaV Ve-Next consistently achieves higher accuracy across all K (e.g. +5 p.p. for K = 2 concepts). This outcome aligns with the dataset's 's design the underlying concepts are derived from structural properties of white synthetic BPs, where color is not semantically relevant. In this context, color may act visual distractor that adds additional complexity. especially for lower- wwwer-performing models.
Are generated images as effective as real ones? Since Bongard-RWR-. relies on generated images, a key question is whether such images are as effective as real ones for evaluating visual reasoning. To investigate this, we compared model performance on the introduced dataset with that on Bongard- RWR, which consists exclusively of real-world images. However, given that Bongard- only 60 instances, we constructed matrices based on smaller subsets of images per side to expand the dataset. For each BP in Bongard-Re -WWR, we considered all pairs of P-element subsets from both
includes
2 new matrices per BP, i.e., 36, .,,,, 5, resp. We applied this method to 54 Bongard Rongard-RWR BPs that were used to generate Bongard-RWR+, and then selected P = 2, 3 to ensure sufficient dataset
matrix sides (analogously as described in Section 3.1 yielding (p)
225, 400, 225, 36 matrices for P = 1,...,
8

---

Published as a conference paper at ICLR 2026
Table I1S and DIS on Bongard-RWR+/LP.. Results in IIS and DIS tasks with varying numbers of images per side P. Image captions in the DIS task were produced with Intern V2..5 78B.
(a) Image-to-Sice (IIS)
(b) Description-to-Side (D1S)
P
IVL.2.5
Q2VI.
LLaVA
MCPM
SC:
IVI.2.5
Q2VI.
LLaVA
MCPM
DS-R1
SC
2
0.51
0.51
0.49
0.50
0.54
2
0.56
0.55
0.53
0.50
0.56
0.47
3
0.54
0.54
0.51
0.50
0.59
3
0.60
0.58
0.56
0.51
0.62
0.51
0.54
0.54
0.51
0.51
0.59
4
0.61
0.63
0.58
0.54
0.64
0.51
0.54
0.55
0.52
0.58
0.62
0.60
0.57
0.52
0.64
0.52
0.55
0.57
0.51
0.51
0.59
6
0.67
0.67
0.61
0.53
0.70
0.54
size, and capped the number of sampled matrices per source BP at 100, resulting in 5 400 matrices per P. We evaluated all 4 VLMs in the CS task on these Bongard-RWR variants and compared the results to those on Bongard-RWR R which uses analogous matrix construction approach but with generated images. As shown Fig. model performance follows similar trends on both datasets accuracy decreases consistently as K iecreases, regardless of whether the images are real or generated. This supports the validity of our coneration-based approach for capturing the challenges of AVR.
Do models learn from demonstrations? As P increases, models are presented with more demon- strations of the underlying concept. While this could enhance concept identification, it also increases the amount of visual information to process. Interestingly, 9 VL benefit from larger P in the CS task, e.g., Intern accuracy improves by 2, 6, 8 p.p. for Bongard-RWR+/// with Bongard-RWRR//13.
shows that Intern VL2.5 and Qwen2-
K 2, .., 16, resp. when comparing the results In contrast, the performance of LL-Ve-eext and MiniCPM-o 2.6 shows no consistent improvement, suggesting these models may struggle leverage additional examples.
To explore this further, we evaluated models on Bongard-RWR+/LLP with 2 using I1S and DIS tasks 3)). In I1S, Intern V1.2.5 and DL. show mild upward trend with increasing P. with Qwen2-Va VL achieving the accuracy of 57% at P = 6, and Intern close behind at 55%. In DIS, accuracy generally improves with showing that the models can effectively utilize additional concept demonstrations in text-based setting. Notably, DeepSeek-R R1 outperforms others for every P > 2 (and is on-par with Intern VL2.5 for 2), demonstrating its strength this setting. Additionally. DIS scores are consistently higher than IIS scores, showing that the models benefit from the explicit image captioning step. Nevertheless, performance remains modest on both tasks, in I1S being clearly inferior to the SC baseline, pointing to persistent AVR gap of the current models.
Finally, models' accuracy on Bongard-RWR-a scores 50% and 57% in IIS and DIS, resp. on Sangard-RWR. (cf. Table 2 but reaches 55% and 67% on Bongard-RWR+H analyze in detail in Appendix C
exceeds that on Bongard- - (e.g., Intern
This gap stems from the differences in dataset construction, which we
Visual diversity facilitates concept recog- nition. Concept diversity can be ex- pressed through both larger number of examples per side (P) and greater variation among the images themselves. To measure diversity in a complementary way, we com- puted the maximum cosine similarity be- tween ViT-L/14 image embeddings within each side of BP, then took the negative of the larger value across both sides as the BP's diversity score. BPs were partitioned into three equally sized bins (Low, Medium, High diversity), and we analyzed model performance the most challenging CS variant (K = 16).
InternVE22.5788
Qwen2-VT 72B
LLvV--ee 1100
MiniCPM-o26 SB
0.58
0.62
0.50
0.43
0.2
0..16
0.20
0.11
0.30
0.20
0.0
Line Modian High
Line Sindiam High
Low Mediam High
Low Menium High
Figure 10: Accuracy vs. image diversity (CS. KK=6).
As shown in Fig. 10 performance is lowest on Low-diversity matrices (e.g.. Intern VL2.5 78B scores 0.50), where images are similar and depict the concept in similar ways, making it harder to identify. In contrast, performance highest High-diversity matrices (e.g., Intern V1.2.5 78B scores 0.62,

---

Published as a conference paper at ICLR 2026
+12 p.p. vs. Low-diversity), where the concept is expressed through more varied visual content, making it more apparent. This indicates that diversity helps disambiguate abstract concepts.
Noisy images hinder concept recognition. To quantify the impact of manual filtering on model performance, we conducted targeted experiment introducing controlled noise into the dataset. Specifically, we constructed Bongard-RWy R-erdd--WR-------- 11/////--mmmmee....- a variant Bongard-RWR+/122 in which each BP side contains one image that passed human filtering and one image that was rejected. This yields BP instances where a subset of images violate the intended concept, producing ambiguity that would arise if human verification was removed.
We evaluated all four models on this impure dataset in the CS task (see details in Appendix C.5) Across K E {2, 4, 8, 16} levels, we observe mean absolute accuracy drops of 3.85, -8.60, 5.73. and -4.60 for MiniCPM-o 2.6 8B, Intern VL2.5 78B, Qwen2-VL. VL 72B, and LLaVA-Nex 1 10B, resp. This consistent reduction in accuracy supports our hypothesis: including unfiltered images degrades dataset quality and harms model performance, demonstrating that the human review step meaningfully improves data quality.
Concept generation. We tested model ability to generate free-form concept descriptions in the CG task. Models were prompted to describe the concept behind each matrix, using either raw image inputs or Intemm captions. Responses were evaluated using standard NLP metrics: BLEU (Papineni et al. 2002), METEOR (Banerjee & Lavie 2005). ROUGE, (Lin 2004), CIDEr (Lin 2004), and Score (Zhang* et al.) 2020). The results, reported in Appendix C show that the models consistently achieved low scores on all metrics, with no clear best-performing model. Manual inspection of selected outputs further confirmed that current models generally struggle to articulate the BP concepts, highlighting this task as a particularly challenging direction for future research.
5 CONCLUSIONS
We introduced Bongard-RWEL large-scale dataset for evaluating AVR capabilities using generated real-world-like images of abstract concepts from classical BPs. The dataset supports diverse BP task formulations, including multiclass concept selection, binary side classification, and free-form text generation. Our results show that while current VLMs demonstrate certain ability to identify coarse- grained concepts, they consistently struggle with fine-grained concept recognition. Although explicit image captioning and increased visual demonstrations can improve performance, even the strongest models fall short in more demanding setups, notably unconstrained textual answer generation.
Limitations. Our dataset generation pipeline still requires human oversight to ensure adherence to intended concepts; advancing generative modeling is needed improve scalability. Our evaluation in- cludes selected VLMs; broader community involvement is essential to benchmark additional methods on Bongard- - We extend discussion on limitations and their mitigation in Appendix
Ethics statement. We note that images generated by the T2I model can reflect social, cultural, or geographical biases. Although our data generation algorithm incorporates diversity safeguards, some biases may still be present in the dataset. To address this, we conducted a limited-scale bias audit, which we further discuss in Appendices and E.
In addition, since our pipeline (Fig. 3 involves manual review, we recognize the potential for annotation-related bias. To mitigate this, each generated image was independently reviewed by two expert annotators, who verified whether the image faithfully expressed the intended concept. Inter- annotator agreement reached Cohen's 's 0.64 (substantial agreement), with annotators agreeing on the same label in 77.3% of cases. Disagreements were resolved through discussion, and images lacking consensus were discarded. Overall, 30.2% of generated images were excluded through this filtering process. A more detailed description of the review process and an annotator agreement report, are provided in Appendix E.2
All images in our dataset are generated with Flux. 1 -eee; none of them was taken from public image databases libraries. The dataset was released under the CC BY 4.0 License, this licensing is
10

---

Published as a conference paper at ICLR 2026
compatible with the terms of the T21 model used for image generation ! We imposed an additional usage restriction by adding the following note: "While Bongard-WWR- is released under the CC BY 4.0 License, it is explicitly not intended for use in high stakes human assessment contexts, including but not limited to standardized testing, educational admissions, employment screening, or cognitive evaluation".
All datasets developed in this work are publicly available via the HuggingFace Datasets platform Code for dataset generation and experimental evaluation available on GitHul under the MIT License.
Reproducibility.
ACKNOWLEDGMENTS
This research was carried out with the support of the Laboratory of Bioinformatics and Computa- tional Genomies and the High Performance Computing Center of the Faculty of Mathematics and Information Science Warsaw University of Technology.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-VI: VL: versatile laiiuala model for understanding, localization, text reading, and beyond. arXiv:2308.12000
2023.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metrie for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 511-520. PMLR, 2018
Valerio Biscione, Dong Yin, Gaurav Malhotra, Marin Dujmovic, Milton I. Montero, Guillermo Puchla, Federico Adolfi, Rachel F Heaton, John F. Hummel, Benjamin D Evans, et al. MindSet: Vision, A toolhox for testing DNNs key psychological experiments. 2XXx::2404.05290, 2024.
Yonatan Bitton, Ron Yosef, Eliyahu Strugo, Dafna Shahaf, Roy Schwartz, and Gabriel Stanovsky, VASR: Visual analogies of situation recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1): 241-249, 2023.
Mikhail Moiseevich Bongard. Pattern Recognition. Spartan Books, 1970.
Xu Cao, Yifan Shen, Bolin Lai, Wenqian Ye. Yunsheng Ma, Joerg Heintz, Jintai Chen, Meihuan Huang, Jianguo Cao, Aidong Zhang, and James Matthew Rehg. What is the visual cognition gap between humans and multimodal L.MMs? In Second Conference on Language Modeling, 2025.
David J Chalmers, Robert French, and Douglas R Hofstadter. High-level perception, representation, and analogy: A critique of artificial intelligence methodology, Journal of Experimental & Theoretical Artificial Intelligence, 4(3):185-211 1992.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta- Baseline: Exploring simple meta-lcarning for few-shot lcarning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9062-9071, 2021.
Zhe Chen. Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. 1rX0v2241..05271, 2024a.
Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
CENSSR.
md

---

Published as a conference paper at ICLR 2026
François Chollet. On the measure of intelligence. crXin:/911.015472200
Stefan Depeweg, Constantin A Rothkopf, and Frank Jäkel. Solving bongard problems with visual language and pragmatic reasoning. arXiv:1804.044333 2018.
Stefan Depeweg, Contantin A Rothkopf, and Frank Jäkel. Solving bongard problems with visual language and pragmatic constraints. Cognitive Science, 48(5)=e13432 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3382-7392, 2024.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(():594-6) 611, 2006.
François Fleuret, Ting Li, Charles Dubout, Emma K Wampler. Steven Yantis, and Donald Geman. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108 (43):17621-17625, 2011.
Harry E Foundalis. Phaeaco: A cognitive architecture inspired by Bongard's problems. PhD dissertation, Indiana University, 2006a.
Harry E Foundalis. Index of bongard problems. http://wwwwccccac..cccc.eeeeeaea bppps///ssppppicxx htm. 2006b. Accessed: 2025-05-01.
www.
Isaac R Galatzer-Levy David Munday, Jed McGiffin, Xin Liu, Danny Karmon, Iia Labzovsky, Rivka Moroshko, Amir Zait, and Daniel McDuff. The cognitive capabilities of generative Al: A comparative analysis with human benchmarks. arXiv:2410.07311, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-RI: Incentivizing reasoning capability LLMs via reinforcement learning. arXiv:2501. 12948, 2025.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding disentangled attention. In International Conference on Learning Representations, 2021.
Decoding-enhancee enhanced BERT with
Michelle Helms-Lorenz, Fons J.R Van de Vijver, and Ype Poortinga. Cross-cultural differences in cognitive performance and Spearman's hypothesis: g or c? Intelligence, 31(1):9-21 2003.
José Heraáneez-Orallo, Fernando Martinez-Plumed, Ule Schmid, Michael Siebers, David Dowe. Computer models solving intelligence test problems: Progress and implications. Artificial Intelligence, 230:74-107,
2016.
Felix Hill, Adam Santoro, David Barrett, Ari Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2019.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 30016-30030, 2022.
Douglas Hofstadter. Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought. Basic books, 1995.
Dokhyam Hoshen and Michael Werman. IQ of neural networks. arXiv:1710.01692,200
HuggingFace.
https: / /tapss:///ggg////e www.hhggggggacc..ooooco
------1-----6------ L6-v2 2021.
Accessed: 2025-05-01.
Nicholas Ichien, Qing Liu, Shuhao Fu, Keith J Holyoak, Alan Yuille, and Hongjing Lu. Visual analogy: Deep learning compositional models. 1rXXv::/05507064 2021.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning. In International Conference Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2026
AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, Lengyel, G Lample, Saulnier, et al. Mistral 7B (2023), arXiv:2310.06825, 2023.
Huaizu Jiang, Xiaojian Ma, Weili Nie, 7hiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-Ho Benchmarking few-shot visual reasoning for human-ohject interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19056-19065, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei. Scaling laws neural language models. arXiv:2001. 2020.
Sergii Kharagorgiev. Solving bongard problems with deep learning. https://k10v wwt.kv.....hubb.... github. io/ 2018/02// 005//22225/600 aooolliinnn-oogaar--rroooboooo wittmmswwittttme deep- learning/ February 2018. Ac- cessed: 2025-05-01.
Black Forest Labs. Flux. httpps//gittub..cmm/llettt 05-01.
2024. Accessed: 2025-
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Che Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
Alexandre Linhares. A glimpse at the metaphysics of bongard problems. Artificial Intelligence, 121(1-2): 251-270, 2000.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the 2024a.
the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306,
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024b.
Mikolaj Malkiáski and Jacek Mandzzuk. A review of emerging research directions in abstract visual reasoning. Information Fusion, 91:713-736, 2023.
Mikolaj Malkiński and Jacek Mandziuk. Multi-label contrastive learning for abstract visual reasoning. IEEE Transactions Newral Networks and Learning Systems, 35(2):144-11553 2024a.
Mikolaj Malkiáski and Jacek Mańdziuk. One self-configurable model to solve many abstract visual reasoning problems. Proceedings of the AAAI Conference on Artificial Intelligence, 14297-11355, 2024b.
Mikolaj Malkiáski and Jacek Maádziuk. Deep learning methods for abstract visual reasoning: A survey on a progressive matrices. ACM Computing Surveys, 57(7):1-36, 2025.
Mikolaj Matkiński, Szymon Pawlonka, and Jacek Mańdziuk. Reasoning limitations of multimodal large language models. case study of bongard problems. In International Conference on Machine Learning, pp- 42795-42833. PMLR, 2025.
Jacek Mańdziuk and Adam Zychowski. DeepIQ: human-inspired AI system solving IQ test problems. In 2019 International Joint Conference on Neural Networks, pp. 1-8. IEEE, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Chen, and Picter Abbeel. simple neural attentive meta-learner. In International Conference on Learning Representations, 2018
MistralAT. Announcing Pixtral 12B. https: htt.///sstralll.cmmmmm aii htt.a/aaaaa//eess///traaall--- 2025-05-01.
2024. Accessed:
Melanie Mitchell. Abstraction and analogy- in artificial intelligence. Annals of the New York Academy of Sciences, (1):9)):79-)) 10199-00122000000111111111 2021.
Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongard-LOGes A new benchmark for human-level concept learning and reasoning. Advances Neural Information Processing Systems, 13:16468-16480 2020.
Richard E. Nisbett, Kaiping Peng, Incheol Choi, and Ara Norenzayan. Calure and Systems of Thought: Holistic versus Analytic Cognition, pp. 956-985. Cambridge University Press, 2008
Victor Vikram Odouard and Melanie Mitchell. Evaluating understanding on conceptual abstraction benchmarks. arxiv:2206. /4187, 2022.
13

---

Published as a conference paper at ICLR 2026
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Yonggang Qi. Kai Zhang. Aneeshan Sain, and Yi-Zhe Song. PQA: Perceptual question answering. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12056-12064, 2021.
James C Raven. Mental tests used in genetic studies: The performance of related individuals on tests mainly educative and mainly reproductive. Master's thesis, University of London, 1936.
John Raven. The raven progressive matrices: A review of national norming studies and ethnic and sociocconomic variation within the united states. Journal of Educational Measurement, 26:1 16, 09 2005. doi: 10.1111/j. 1745-3984. 1908844.999..ee 1989.bb00314.x.
John C Raven and John Hugh Court. Raven's 's progressive matrices and vocabulary scales. Oxford pyschologists Press Oxford, England, 1998.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in Neural Information Processing Systems, 32:12268- 1222222227999,--1111122222929999 2019.
Eleanor H Rosch. Natural categories. Cognitive psychology, 4(3):328-350, 1973.
Kazumi Saito and Ryohei Nakano. concept learning algorithm with adaptive search. In Machine intelligence 14: applied machine intelligence, pp. 347-363, 1996.
Mert Bülent Sarıyıldız Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenct clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8011-8021, 2023.
Pushkar Shukla, Aditya Chinchure, Emily Diana, Alexander Tolbert, Kartik Hosanagar, Vineeth N. Balasub- ramanian, Leonid Sigal, and Matthew A. Turk. Mitigate one, skew another? Tackling intersectional biases text-to-image models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 12453-12472, Suzhou, China, November 2025. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30:4080-4099 2017.
Athary Sonwane, Sharad Chitlangia, Tirtharaj Dash, Lovekesh Vig, Gautam Shroff, and Ashwin Srinivasan. Using program synthesis and inductive logic programming solve bongard problems, arXiv:21 2021.
Sebastian Stabinger, David Poer, Justus Piater, and Antonio Rodríguez-Sáncee Sánchez. Evaluating the progress of deep learning for visual relational concepts. Journal of Vision, (1((1) 2021.
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. V-PROM: A benchmark for visual reasoning using visual progressive matrices. Proceedings of the AAAI Conference on Artificial Intelligence, 14007):10011-22773 2020.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. StableRep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, (0548382-484002 2023.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 15887- 15898, 2024.
Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov, Effective data augmentation with diffusion models. In International Conference Learning Representations, 2024.
Han LJ van der Maas, Lukas Snock, and Claire E Stevenson. How much intelligence is there in artificial intelligence? A 2020 update. Intelligence, 87:101548, 2021.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Dwwn2-VV.: Enhancing vision-language model's perception of the world any resolution. 2rXiv:2409.12199 2024.
Wenhui Wang, Furu Wei, Li Dong. Hangbo Bao, Nan Yang. and Ming Zhou, MiniLM: Deep self attentttonion distillation for tank-agi compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020a.
14

---

Published as a conference paper at ICLR 2026
Yaqing Wang. Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3)11334, 2020b.
Taylor Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Re O'Reilly, and Jonathan Cohen. Learning representations that support extrapolation. International Conference on Machine Learning, pp- 10136-10146. PMLR, 2020.
Brandon T Willard and Rémi Louf. Efficient guided generation for LLMs. arXiv2330700900. 2023.
Rujie Wu, Xiaojian Ma. Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, and Yizhou Wang. Bongard- OpenWorld: Few-shot reasoning free-form visual concepts in real world. International Conference on Learning Representations, 2024.
Antonia Wüst, Tim Woydt, Lukas Helff, Inga lbs, Wolfgang Stammer, Devendra Dhami, Constantin Rothkopf, and Kristian Kersting. Bongard in wonderland: Visual puzzles that still make Al go mad?? In International Conference on Machine Learning, pp. 681 18-68142. PMLR, 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Chi Chen, Haoyu Li, Weilin Zhao, et al. Efficient GPT-4V level multimodal large language model for deployment on edge devices. Nature Communications, 16( 16(1),550, 2025.
Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang. Taiwen Hc, et al. RLAIE-V: Open-source AI feedback leads super GPT-4V trustworthiness. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19985-19995, 2025.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chus Zhu. RAVEN: A dataset for relational and analogical visual reasoning. Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q Weinberger, Yoav Artzi BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. 2020.
15

---

Published as a conference paper at ICLR 2026
A BROADER IMPACT
Our findings highlight that state-of-the-art VLMs continue to struggle with tasks that are relatively straightforward for humans. As public discourse increasingly focuses on the capabilities and risks of AI systems, benchmarks such as Bongard-RWRe play crucial role in dispelling misconceptions about current model capabilities. Our release of code and datasets under an open license promotes open research and lowers the barrier of entry for researchers to evaluate and improve their methods. However, we also acknowledge potential risks. in online remote IQ assessments, allowing individuals inflate their scores. Such misuse may lead to unfair advantages in job recruitment or other competitive settings where cognitive ability plays a role. This underscores the importance of raising public awareness about both the capabilities and potential applications of such tools.
effective AVR solver could potentially be misused
Prior work in cognitive psychology shows that cultural background can indeed shape reasoning strategies and outcomes (Hellm-Lorennz et al., 2003), sometimes leading to qualitatively different solution approaches (Nisbett al., 2008). In partieular, studies RPMs discussed the variation between ethnic groups within the United States (Raven 2005). While little work has examined BPs in this context, related studies in concept learning and categorization suggest that cross-cultural differences can in fact influence how abstract visual concepts are formed (Rosch) 1973). Building on this line of work, we believe that our Bongard-RWEL dataset, which uses real-world images rather than abstract patterns, can serve as complementary resource for studying how multi-image reasoning generalizes across different cultures.
B DATASET VARIANTS
Algorithm 1 The generation Bongard- RWR++ images.
1: Input: A set of Bongard- RWWR problems BPx E RWR
2: Output: Generated images G (GL, GR)
3: N + +5
4:
5: for (L,, RE, Tk, Ck) E RWR do
6:
LA Lx U.KLL
7:
R4 + R4 UTEUR
8:
for S& (In, Rig do
9:
for Sk.i E Sz do
10:
(S SK,) + Describe(S (Sk.i, Csy)
(S f== t Augment(St, (St Osa))
12:
for j 1, I,.., N do GS t ( k.i.j ST.))
14:
15:
end for end for
17: end for
end for
Image generation. T outlines the process of generating Bongard-RWR+ images. Each BPx (I4,, Rg, Tk: Cx) from Bongard-RWR is processed by iterating over both matrix sides, T and Rg. Test images Ti, are processed based on their corresponding side. For each image S₁,6 on a given side S₁ (either L3 or Rgg,, positive (S and a negative (S textual description are generated based on the side's concept Cs, (either Cr., or CR.)). The positive description then augmented N = 15 times to produce a diverse set of semantically consistent variations St,,5- J = 1,..., N. Each augmented description is paired with the corresponding negative description and passed to a T2I model to generate reviewed manually by a human expert, and those that did not faithfully reflect the intended concept were discarded.
image G.k.i, (either GLR,i, G.....). All generated images were
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Bongard- R matrix construction.
Input: Generated images G (GL, GR), Boolean image removal flag F, Number of images
per side P
2: Output: Set of generated matrices R
3: M + 10
4: RWR*
5:
6: for GTg, RR E G
7:
for GS2 E (GT,,. GR do
8:
Eg < GellmageFnhedding((GSs.))
9:
GS +
10:
[GSk,1, Ex,1} + SetAllSubsets(GSy, Ex, P)
11:
for 1,.... .M do
12:
Find the most diverse subset
8 arg mi.. max [E,,11 Ex,1j i, 1...., {i
14:
GS GSt G.S.t
15:
16:
if F then
17:
Find the most redundant image and remove it from the overall image pool
18:
p f arg = (E Ek,ej j I,..., P A i
19:
GS& GSS (A.)
20:
Ex < Ex. - (Ex.t.p)
RemmovesbseasCootannniigee (GaniaaggGGSS.... Ek, GSk,t // Exit early if there is not enough images to construct the next subset
{GSSk Ek}
22:
if [GS P then
24:
break end
25:
26:
// Exit early if the images become too similar
27:
<- Mean(1 |[xx, E|l (,j 1, .... |E + j))
Ex,
if s 2 0.85 then
29:
break
30:
end end if
31:
32:
end for
33:
end for
34:
35:
RWR- R U (GL GR
36:
end for
17

---

Published as a conference paper at ICLR 2026
Bongard-RWR+ construction. Algorithm 2 ooutines the construction of Bongard - matrices from a pool of generated images G = (GL, GR). For each BP,, both sides GL₃ and GRg, denoted as GS are processed independently, to construct M 10 visually diverse subsets per side GS Each side's ' image pool GS& is first embedded using ViT-I//14 (Dosovitskiy et al.. 2021). producing embeddings Fg.- The algorithm arranges images into P-element subsets, denoted GSk,I. along with their embeddings E In each of the M iterations, the algorithm selects the most visually diverse subset, defined as the subset whose maximum pairwise cosine similarity among its embeddings is minimal. This selected subset G is then added to GS If the image removal flag F is enabled, the algorithm identifies the most redundant image in the selected a.bset-ile.. the one with the highest average similarity to the other images and removes it from the overall image pool GS&. The loop terminates early if either (1) the remaining pool contains fewer than images or (2) the overall average similarity among remaining images exceeds a threshold, indicating reduced diversity. Finally, the cartesian product GL₁ x GR is computed to generate all possible pairings and added to the output set RWR + Examples of resultant BPs are illustrated in Fig. 22
Bongard- R construction. We constructed grayscale variant of Bongard- -weerrddRRR+,... denoted Bongard-RWR+/GS, by converting all images in the original set to grayscale. The overall matrix construction procedure remained unchanged. Representative examples are shown in Fig. 23
Bongard-RWWE R construction. We also constructed Bongard-RWR+/LPP a dataset variant in which the image removal step is disabled (i.e., the flag F in Algorithm 2 set to false). Without this removal step. the greedy subset selection procedure for building GS & often draws from a smaller set of unique images within GSS,. In effect, matrices in Bongard-RWR+11 exhibit lower average intra-side embedding similarity but are composed of fewer distinct images. The increased visual diversity within each matrix can facilitate recognition by presenting concepts in broader range of views. However, because the same images are reused more frequently across matrices, there is overall less image-level diversity compared to Bongard-RWR++ We constructed dataset variants with P 2...., 6, denoted Bongard-RWR+/L2, Bongard-RWR+/133, etc. Example BPs from Bongard- - are shown in Figs.
28
Construction of Bongard-RWR+/TW
We also developed Bongard-RWR+/TIE a dataset variant in which BPs are divided into train, validation, and test splits (TVT), enabling the evaluation of supervised approaches. To construct these splits, first consider the six images on each side of a Bongard-RWR matrix, and the corresponding seventh test image. Images 1 4, along with the side's test image, are grouped into shared pool used to construct BP contexts for all three splits and to define test targets for the training split. Images 5 and are exclusively allocated as test images for the validation and test splits, resp. This partitioning strategy ensures relatively high image diversity in the training set while enabling evaluation on -iitr-i-iitiitttionn images in the validation and test splits. Bongard- RWT+ comprises 12 150 matrices (7 290 1 215 / 3 645 in train / validation / test splits); we additionally provide an extended variant, Bongard- BungarddRWR+/TVVR -WTraae,,, consisting of 86 400 BPs (51 840 8 640 / 25 920 in train / validation test splits).
and Bongard-RWRR-TTTTLLare
C EXTENDED RESULTS
C.I IS IMAGE COLOR NECESSARY FOR CONCEPT RECOGNITION?
To extend the grayscale image analysis from the CS task, we evaluated model performance on Boggard-RW++/GS with the IIS and I2S tasks. As shown in Fig. [T] grayscale images led to improved accuracy in certain cases, likely by encouraging the model to focus on structural cues. Nevertheless, performance remained close to the random guess level across both tasks, reiterating their inherent difficulty.
C.2 ARE GENERATED IMAGES AS EFFECTIVE AS REAL ONES?
To quantify the relationship presented in the paper (Fig. 9 more rigorously, we conducted a statistical analysis across all four models for both counts of panels per side (P 2, 3) and all difficulty levels (K 2, 4, 8, 16). We computed Pearson correlations and fitted linear regression models to assess
18

---

Published as a conference paper at ICLR 2026
Image- Tonge--T--Siee
Images-To-Shie To-Sides
1.0
RWR+
RWR+
RWR+/GS
RWR+GS
0.6
(+0.01)
0.51
(+0.01)
0.50
(40.00)
0.50
(+0.01)
0,49
(+0.12) 12)
(+0.06)
0.50
(+0.00)
0.50
(+0.04)
0.49
0.4
0.2
0.0
IV1.2.5
Q2-VL
IVL.2.5 Q2-VI.
MCPM
Figure 11: Solving
with I1S and 12S. The impact of using grayscale images on accuracy in the IIS (left) and I2S (right) tasks.
Intern VL2.5 78B
Qwen2-Ve VL 72B
LLaVA-Next 110B
MiniCPM-o 2.6 8B
1.0
RWR
RWR
0.8
RWR+L
RWR+/L
RWR+L
RWR-/L
0.6
+0.04
+0.05
+0.03
+0.02
+0.01
+0.00
-0.01
+0.01
+0.00
+0.01
+0.01
-0.01
0.4
0.2
0.0
2
4
3
4
3
4
2
3
4
Figure 12: Functional equivalence of generated images and real-world images. Performance trends in the IIS task on Bongard-RWY and Bongard-RWR+/LP for varying number images per side P = 2, 4.
how well performance on the generated-image dataset (Bongard-RWR+//PP) predicts performance on the real-image dataset (Bongard-RYy -WRR.. The results are summarized in Table
All models exhibit strong correlations between real-image and generated-imes image dataset performance (r > 0.99), with R² consistently above 0.987. For example, InternVL2. 78B for = shows 0.994 and regression slope of 1. 1.204, meaning that, e.g., 10 p-p. gain Bongard-RWR+/IL.. corresponds to roughly a 12 p.p. gain on Bongard-RWR. Error measures including mean absolute difference (MAD 7.0) and RMSE (< 2.09) confirm that -hngarddRWR+/LPPP is slightly easier set, on average, but the relationship is stable and predictable across difficulty levels.
These results demonstrate that the generated-image dataset preserves the underlying difficulty of the real-image dataset. Consequently, performance improvements on Bongard- R can be expected translate reliably improvements on real-image BPs, supporting the validity of using generated images for AVR evaluation.
Furthermore, we evaluated model performance on Bongard-RWR and Bongard-RWRR/1L in the IIS task for varying number of images side P = 2, 3, 4. As shown in Fig. 12 results are comparable across both datasets, regardless of whether the model processes real-world or generated images.
To further analyze the types of errors models make real vs. generated images, we compared per-concept performance on Bongard-RWR (real) and Bongard-RWR+/1PP for P 3. Across the 54 concepts present in both datasets, performance was largely similar: in 40 concepts (74%) the difference was < 10 p-p., in 7 concepts (13%) differences were within 10 - 20 p.p.. and for 7 concepts (13%) differences exceeded 20 p.p. Notably, in all larger-difference cases, models performed better on average on Bongard- Bongard-RWRen RWR+/LP than on Bongard- RWR. In the small-difference cases, models performed better Bongard-RWF for 5 concepts and Bongard- - for 2 concepts. Overall, the results indicate that despite certain variation, the two datasets present a comparable level of difficulty.
19

---

Published as a conference paper at ICLR 2026
Table 4: Performance correlation on generated and real-world images. Pearson correlation (r), coefficient of determination (R²), regression parameters, mean absolute difference (MAD) and RMSE across K E 4,, 16) on Bongard-Ryy Rergard--WWaaae and Bongard-RWR+HH for P (2.3).
Model
P
R²
Slope
Intercept
MAD
RMSE
MiniCPM-o 2.6 8B
0.997
0.994
0.971
-0.005
1.75
1.38
MiniCPM-o 2.6 8B
0.999
0.998
0.902
0.030
2.25
0.80
InternV1.2.5 78B
2
0.994
0.987
1.204
-0.190
5.00
2.04
Intem 78B
0.993
0.987
1.419
-0.377
7.00
2.09
Qwen2-VL 72B
1.000
0.999
1.005
00.015
1.25
0.43
Qwen2-VL 72B
1.000
0.999
1.004
-0.035
3.25
0.43
LLaVA-Next 110B
0.994
0.987
0.867
0.054
3.00
1.98
LLa 110B
3
1.000
1.000
0.936
0.040
1.50
0.26
Table 6: Concept Generation: Image-based on Bongard- RWR+.
BLEU,
BLEU2
METEOR
ROUGE,
CIDEr
PBERT
HEFRT
FBERT
Intern V1.255 78B
0.063
0.011
0.056
0.159
0.017
0.114
-0.049
0.027
Qwen2-VLL VL 72B
0.048
0.004
0.043
0.121
0.007
0.066
-0.102
-0.025
LLaVA-Next 110B
0.071
0.008
0.061
0.174
0.006
0.128
-0.037
0.040
MiniCPM-o 2.6 8B
0.077
0.008
0.063
0.181
0.004
0.127
-0.035
0.041
These findings are consistent with the experiment on the CS task presented in the main paper and reinforces claim that generated images equally effective real-world images for probing visual reasoning capabilities of VLMs.
C.3 Do MODELS LEARN FROM DEMONSTRATIONS?
We analyzed how the number of images per side (P), repre- senting concept demonstrations, affects model performance. Extending the IIS and DIS analysis in the main paper, evaluated model accuracy on Bongard-RWe - using the CS task across varying values of P. As shown in Table performance generally improves with more demonstrations, particularly for stronger models such Intern Vn.ern which achieves its best result at P = 6. However, not all mod- els exhibit this trend Qwen2-VL performs similarly for P = 3 and P = 6. while MiniCPM-o 2.6 achieves its peak performance for P 5. These results suggest that while additional demonstrations can be beneficial, some models may not be able to fully exploit them.
Table 5: Concept Selection on Bongard-
RWR+/LP for K 16 concepts.
P
IVL.255 Q2-VI.
LLaVA
MCPM
2
0.46
0.37
0.18
0.18
3
0.54
0.41
0.19
0.17
4
0.46
0.39
0.19
0.18
5
0.58
0.39
0.19
0.19
6
0.62
0.41
0.20
0.18
C.4 ADDITIONAL BASELINES
We evaluated the performance of supervised learning methods on the IIS and DIS tasks. The models operated on embeddings produced by pre-trained models. For IIS, raw images were encoded using ViT-L/14 (Dosovisskiy et al. 2021). while for DIS, image captions produced by InternV1.2. were embedded using fine-tuned MiniLM (HuggingFace 2021 Wang et al.) (220a).. We considered an MLP with a single hidden layer applied concatenated embeddings, Wild Relation Network (WReN) (Barrett et al. 2018), which processes embedding pairs and aggregates them via summation, and SNAIL (Mishra et al., 2018). an attention based meta-learner, As shown in Table 12 all models performed near the random guess level both Bongard-RWR+/TH and Bongard Bongard-RWRR/TV RWR+/TVT. Large, demonstrating that the proposed BPs present significant challenge not only for VLMs but also for standard supervised learning approaches.
20

---

Published as a conference paper at ICLR 2026
Table Concept Generation: Image-based on Bongard-RWR</L6..
BLEU1
BLEU2
METEOR
ROUGEl
CIDEr
PBERT
ROEKT
FBURT
InterVLL2.5 78B
0.060
0.011
0.053
0.149
0.021
0.111
-0.045
0.028
Qwen2-VL. 72B
0.042
0.005
0.039
0.110
0.008
0.046
-0.118
-0.042
LLaVA-Next 110B
0.072
0.007
0.061
0.172
0.008
0.133
-0.031
0.045
MiniCPM-o 2.6 8B
0.076
0.009
0.064
0.182
0.005
0.144
-0.024
0.054
Table 8: Concept Generation: Text-based Bongard-RWR+.
BLEU1
BLEU2
METEOR
ROUGE,
CIDEr
PBERT
RBERT
FBERT
Intern 78B
0.122
0.021
0.063
0.158
0.017
0.142
0.073
0.107
Qwen2- VL 72B
0.132
0.021
0.066
0.161
0.018
0.145
0.090
0.117
LLaVA-Next 110B
0.111
0.018
0.064
0.165
0.013
0.147
0.062
0.103
MiniCPM-o 2.6 8B
0.105
0.019
0.064
0.170
0.019
0,141
0.033
0.085
DeepSeck-RI 70B
0.129
0.013
0.060
0.138
0.015
0.069
0.050
0.060
C.5 NOISY IMAGES HINDER CONCEPT RECOGNITION
Fig. 13 compares model performance between Longard-RWR+HH and
in the CS task across K levels. Substituting an accepted image with rejected one consistently reduces performance across models, with only a single case of comparable results (LLaVA-Next, K 8). The effect is strongest for InternVL2.. (mean drop of 8.6 p.p.) and Qwen2-VL (5.7 p.p.), moderate for LLaVA-Next (4.6 p.). and smallest for MiniCPM-o 2.6 (3.9 p.p.).
C.6 PERFORMANCE ON CONCEPT GENERATION TASK
Tables evalluaee model performance on the CG task, where models prompted produce free-form descriptions the underlying concept for each matrix. Inputs were either raw images or image captions produced by VL2.55 Experiments were conducted both Bongard-RWR+ and Bongard-RWR++/ W+/66 datasets. We assess generated descriptions using standard NLP metrics: BLEU (BLEU, and BLEU,) (Papineni al. 2002), METEOR (Banerjee Lavie 2005), ROUGE, (Lin 2004 CIDEr 2004). and SERTScore (precision PBEKTY recall RBERT, FBERT) iZaagg* et al.
2020 For BERTScore, we apply baseline rescaling and the DeBERTa model (He al., 2021] which has strong alignment with human evaluations. Higher scores indicate better performance across all metries.
MiniCPM-o 2.6 consistently performs best in image-based setups (Tables 6 ranking first in 5 of 8 metrics on Bongard- -ongard-RWR+ and on Bongard-RWR+/LG though differencee with other models small. In text-based setups (Tables 8 D. Qwen2-VL leads in 5 of 8 metrics Bongard-RWR. and 4 of on Bongard-RWR++Hle compare pep-performin models on Bongard-RWR+ with the best results reported for Bongard- Open World (Wu et al. 2024, Appendix E, Table 4), as shown in Tables I0 and IT] Regardless of input representation, models achieve significantly higher scores Bongard-Open World than Bongard-RWR+, indicating that the concepts in our dataset are more challenging to recognize even for ftate-oftthe-ee models.
again with relatively small gaps to competing models. We further
C.7 ASSESSING DIFFICULTY OF BONGARD-RWR+ VARIANTS
We compared model performance on range of tasks using both Bongard-RWR+ and Bongard- RWR+/L6, with the results shown in Table 13 Overall, the models consistently perform better on Bongard-RWR-HH across both image- and text- strategies. In particular, DeepSeek-RR -R presents notable gains in text-based setups, achieving 70% accuracy with DIS and 72% using
basel imd https://hugge
21

---

Published as a conference paper at ICLR 2026
Table 9: Concept Generation: Text-based on Bongard-RW
BLEU1
BLEU2
METEOR
ROUGE,
CIDEr
ABERT
ROERT
FbERt
InternVL.225 78B
0.123
0.023
0.065
0.162
0.019
0.154
0.087
0.120
Qwen2-VL 72B
0.132
0.023
0.066
0.160
0.018
0.153
0.095
0.124
LLaVA-Next 110B
0.114
0.020
0.064
0.164
0.012
0.149
0.070
0.108
MiniCPM-o 2.6 8B
0.110
0.024
0.066
0.169
0.022
0.146
0.046
0.094
DeepSeek-RI 70B
0.128
0.015
0.060
0.134
0.018
0.072
0.052
0.062
Table 10: Concept Generation: Image-based on Bongard-RWR+ vs. Bongard-Open World. World. Comparison of concept descriptions generated from images in Bongard-RWR+ (using MiniCPM-o 2.6 8B) and Bongard- OpedddppWWWWWn World (using GPT-4V (Wu et al. 2024, Appendix E, Table 4)).
BLEU1
BLBU2
METEOR
ROUGE,
CIDEr
Bongard-RWR+
0.077
0.008
0.063
0.181
0.004
Bongaadd-OpenWee World
0.190
0.073
0.111
0.188
0.527
D2S. We attribute these improvements the greedy matrix construction strategy used Bongard- RWR+/L6, which prioritizes subsets that minimize intra-side embedding similarity. This encourages the selection of more visually diverse images to represent concept, making the underlying shared aspects more distinguishable and thus easier to recognize for the models.
C.8 PERFORMANCE OF THE SIMILARITY CLASSIFIER
The Similarity Classifier achieves high results in I1S experiments, as it exploits a fundamental property of BPs: defining concept must hold for all images use of a maxxaggregattin heuristic, which assigns the test input to the side for which the maximum distance between individual image embeddings and the test image embedding (most outlying pair) is smaller. Fig. 15 illustrates the strength of the underlying design concept of SC.
given side. This motivates the
C.9 QUALITATIVE ANALYSIS
To understand how concept difficulty varies across input modalities, we analyzed the average number of correctly solved instances per concept for all models. Fig. presents the distribution of concept difficulty for both I1S and D1S tasks.
For I1S, each concept was solved 79 - 109 instances (out of 200) on average. This relatively narrow range indicates that most concepts remain uniformly challenging for models in the visual domain, which is consistent with their near-random performance levels. One noteworthy case is "Shading thicker the right side" (concept #63), which appears among top-5 solved concepts in I1S (109) but is relatively difficult in DIS (92), highlighting the relevance of visual input in identifying fine-grained concepts.
In contrast, DIS exhibits much broader difficulty range, from 70 to 182 solved instances, reflecting high performance on certain concepts that appear be easier. Notably, the top-5 solved concepts include counting-based ones such as "three parts" or "one figure", which may be easier to capture in the text domain.
Tables 14 and 15 present the top correctly recognized concepts per model for the 11S and DIS tasks, resp. In the image-based setup, we observe diverse distribution of top concepts across the models, with little overlap. This variation suggests that each model relies on distinct reasoning strategies when processing visual input. For example, Intern shows some capacity for identifying rules involving contour, similarity, and shape, whereas Qwen2-V VL performs better concepts related to count and angle. In contrast, the text-based setup yields more consistent results across models. Counting-relatee concepts dominate the top- cop-perforning examples, followed by shape and size.
The contrast between IIS and DIS results highlights modality speciifc strengths, as the models tend to prefer different concept groups depending whether they reasoning over images text.
22

---

Published as a conference paper at ICLR 2026
Table 11: Concept Generation: Text-based on Bongard-RWR+ vs. Bongard-Openworld.
Comparison of concept descriptions generated from image captions in Bongard-RWW R (using Qwen2-VL VL 72B for prediction and InternVL.2.5 78B for captioning) and Bongard- Open World (using
ChatGPT for prediction and BLIP-2 w/ Fine-tuning for captioning (Wu et al., 2024 Appendix E, Table 4)).
BLEU1
BLEU2
METEOR
ROUGE,
CIDEr
Bongard-Ryy + Longard-OpennWeee
0.132
0.021
0.066
0.161
0.018
0.441
0.292
0.222
0.417
1.714
Table 12: Baseline performance. Test accuracy of supervised learning models on Bongard- using image and text-based input representations.
RWR+/TVT and Boongad-RWR+/TVV Each experiment was repeated 10 times with different seeds; the table reports mean + std.
Bongard-RWED RWR+
Image
Text
Image
Text
MLP
0.49 + 0.00
0.49 = 0.00
0.47 + 0.01
0.49 0.00
WReN
0.49 0.01
0.48 = 0.01
0.48 0.01
0.48 0.01
SNAIL
0.48 + 0.01
0.51 = 0.01
0.48 + 0.01
0.48 0.01
These trends are further illustrated in Fig. 14 which shows average accuracy per concept group. Most models, Intern VL2.5 and Qwen2-VL in particular, exhibit performance spikes in size, count, and shape groups with the DIS task compared to I1S. This discrepancy indicates weak connection between visual and textual processing pathways in multimodal systems and suggests that improving the integration between these modalities could further boost model performance via knowledge transfer.
1.10 SOURCES OF ERROR AND FAILURE MODES
In Section 4 (Concept Selection) we noted that concepts such as Contour, Rotation, and Angle pose particular difficulties because they rely subtle geometric current models struggle to capture reliably. To further explore these weaknesses, we deconstructed the concept-group aggregates (Fig. 6) and reviewed model performance on specific concepts in the CS task with K = 4. For example, Intern V1.2.5 achieved consistently strong results on all concepts within the Size, Shape, and Branching groups, but struggled with certain concepts from the three more difficult groups mentioned above (Contour, Rotation, and Angle):
precise spatial relations that
1. For Angle (69% average accuracy), the model performed particularly poorly on concept #76 "Long sides concave Long sides convex" (16%), which requires abstracting beyond surface-level features to recognize a geometric property across diverse real-world scenes.
2. For Rotation (73%). performance dropped on concept # #17 "An acute angle directed inward No acute angle directed inward" (43%), again pointing to difficulties in detecting precise spatial arrangements.
3. For Contour (56%), concept #3 "Outline figures vs. Solid figures" (22%) was especially challenging, suggesting difficulties in recognizing fine- rree-grined details such as object bound- aries.
Solving these tasks requires abstraction beyond immediate object features precisely the type of reasoning our benchmark aims to measure.
We also investigated typical model error patterns in the CS task by manually reviewing prediction explanations. Several recurring failure modes emerged: (1) predicting a concept that applies only to a subset of images on a given side, (2) confusing the sides images belong to (3) predicting a concept for side while ignoring that it also applies to the opposite side, (4) relying on shallow surface-level features rather than deep image understanding required to detect fine- grained concepts.
Fig. 17 presents the examples.
23

---

Published as a conference paper at ICLR 2026
1.0
REST
0.8
(MU77)006
0.75
[+020]
0.93
0.56
(+0.83)
(AMMDZ)
0.6
0,46
0.51
0.51
0.30
0.4
I125
(H002)
2
16
2 4
4
4
8
16
IVL2.5 78B
Q2-VL 72B
LLaVA 110B
MiniCPM SB
Figure 13: Model performance on Bongard- - and the corresponding gain over Bongard- RWR+/L2-Impon Impure in the CS task for K {2, 4, 8, 16}. Substituting a human- auman-approved image with one rejected during filtering hinders concept recognition.
Table 13:
Relative difficulty of Bongard- Kongard-RWR+/LA. vs. Bongard- RWR+. Each entry reports
accuracy on Bongard- Bangard-RWR+Hle RWR+/L6, with the absolute change Bongard-RWER shown in parentheses.
Image
Text
CS, K = 16
IIS
12S
DIS
D2S
InterVL2.55 78B
0.62 (+0.05)
0.55 (+0.05)
0.48 (+0.09)
0.67 (+0.110) 0.64 (+0.15)
Qwen2- VL 72B
0.41 (+0.03)
0.57 (+0.08)
0.52 (+0.08)
0.67 (+0.09) 0.56 (+0.14)
I.LaVA-Next 110B
0.20
(+0.01)
0.51 (+0.01)
0.49 (-0.01)
0.61 (+0007)
0.52 (+0.09)
MiniCPM-o 2.6 8B
0.18
(-0.01)
0.51
(+0.03)
0.51 (+0.06)
0.53 (+0.02)
0.45 (+0.04)
DeepSeck-RI 70B
N/A
N/A
N/A
0.70 (+0.13)
0.72 (+0.16)
Similarity Classifier
N/A
0.59 (+0.07)
0.68
(+0.14)
0.54 (+00055
0.59 (+0.09)
C.11 FRONTIER PROPRIETARY MODELS
To contextualize the performance of open VLMs, we conducted a limited evaluation of three state-of- the-art proprietary models- Claude Sonnet 4.5, Gemini 2.5 Pro. and GPT 5.1 5. on Nongard-RWRR across the IIS, DIS, and CS (K 16) tasks. As shown in Fig. 18a these models consistently outperform open ones (cf. Fig. 5 and Table 2). with particularly strong and stable performance on the Shape, Size, and Count concept groups (Fig. (88). Their advantage is most pronounced in the inion-intensive IIS task, where all open models fall to near-chance accuracy despite performing reasonably well on the same groups in CS and, to a lesser extent, DIS. Nevertheless, even the best-performing proprietary model (Gemini 2.5 Pro) reaches only 65% on IIS, 70% on DIS, and 82% on CS, indicating that Bongard - presents significant challenge also for frontier models.
D LIMITATIONS AND FUTURE WORK
Our dataset generation pipeline requires manual verification to ensure that generated images accurately reflect the intended concepts. During initial experiments, we identified limitations in the ability of T2I models to reliably render certain abstract fine-grain fine-grained visual concepts. For instance, the concept "there are (no) inside figures of the second order" from Bongard-RWR could not be accurately represented by the T2I model, preventing us from including in Bongard-RWER We believe future advances in accurate representation of fine-graing grained concepts in generative models will be critical for improving scalability of datasets like Bongard-RWWR RWR++
Our main experiments evaluated the ability of VLMs solve BPs, alongside selected supervised
learning baselines described in Appendix C.4 The results highlight clear limitations in the current models' capacity for abstract visual reasoning, especially when dealing with fine- ine-grained concepts, raising the need for more sophisticated approaches. In particular, multimodal reasoning models that integrate visual and textual information may offer a promising direction, analogous to how reasoning models like DeepSeek-RL operate in the text domain. We envision the introduced datasets as valuable benchmarks for tracking progress in AVR capabilities and, more broadly, in multi multi-image reasoning.
Multiple recent studies confirm the limitations of VLMs highlighted in our work. MindSet: Vi- sion (Biscione et al. 2024) tests models on 30 psychological findings inspired by human visual
24

---

Published as a conference paper at ICLR 2026
Intern InternVL2.5 78B
Qwen2-VL 72B
LLaVA-Next 110B
MiniCPM-o 2.6 8B
Size
Angle
Angle
Angle
Angle
Position
Rosition
Position
Position
Rotatign
LD
Rotations
L.
Rotatign
Rotatioh
u
0.5
DD
Count
Count
Count
Count
Shap
Shape
Shape
Shap
Branching
Scanching
Branching
Branching
Contour
Contour
Costour
Similarity
Similarity
Similarity
Similarity
CS
DIS
IIS
DIS
IIS
CS
DIS
I1S
CS
DIS
IIS
Figure 14: Model performance the CS, 11S and D1S tasks across concept groups Bongard- RWR+. The presented CS results are for
20
5
Figure 15: An I1S example illustrating the strength of the Similarity Classifier Left: Polygons. Right: Curvilinear figures. The right side shows a test image, and the left side presents a plot of distances from the test image to images on both sides the ViT-L/I4 embedding space. The sculpture is farther from the brick wall in the Left side than from any image in the Right side, leading to higher probability of being classified as a curvilinear figure. In contrast, Intern VL2.5 relies on local features and doesn't 't take into account all side images, resulting in an incorrect classification: "The test image shows symmetric ice sculpture, matching the symmetric patterns seen in the LEFT class images. Conversely, the RIGHT class images feature more natural, organic, architectural elements that lack symmetry."
perception, revealing fundamental differences between human and machine vision. VCog- VCog-Bench Bench (Cao et al. 2025) evaluates models on abstract and commonsense visual reasoning as well as visual question answering. showing that VLMS consistently struggle with multi-image reasoning tasks. Evaluations on the WAIS-IV test indicate that VLMs underperform in perceptual reasoning tasks (Galatzer-Levy al. 2024). systematic review of AVR performance across such benchmarks could provide more comprehensive understanding of current model capabilities and limitations.
Our evaluation of bias and fairness (Appendix reveals that, although Flux. 1-dev offers certain demographic variation, it disproportionately generates White figures and tends to produce racially homogeneous crowds. These results are consistent with prior studies on demographic biases in Text-to-Iman Image models (Shukla et al. 2025), highlighting the need for further research on diversity and fairness in Text-to-Image generation. Nevertheless, we do not expect these demographic imbalances to affect the benchmark's validity for its primary purpose: evaluating abstract visual reasoning. The concepts underlying Bongard- Rongard-RWR++ RWR+ (e.g., spatial relations, counting, rotations) are independent of demographic attributes such as gender, race, or age. Human figures serve only as possible way for expressing an abstract rule, not as entities whose demographic characteristics are relevant to the task.
Although we consider the expansion of Bongard- RWR dataset (60 instances) to Bongard- - (5400 instances) significant step forward, some Bongard-RWF matrices and many original Bongard
25

---

Published as a conference paper at ICLR 2026
Fnn ocuis ungle dirccee rnears
200
One figure
178
Shactng tiicerantne 1ignt ride
236
Friee cotts
133
of gymmery
Large fg.
fpures
angle
Ten ciriee cline
cynmerty
cade
Pouume
1:7
Acuic angle
large hele
Pthggore
1.05
Smieth contour figune
1110
Ons Ine
104
Figurra orgaaee vortically
113
iteiical Soures
1344
F'x txrper
112
Thrme (ttettalleeeeeee ejemens
101
One line
Twee cricke dese such
Flungli abrvee ciicee
Thres pants striggtt Iiee
201
Ousline dirc
sbraght
155
Ashap praectton
Arrono pointing in diffirennt seciion An ac-te angle nected Insand
(witicl mectonglee or contad lieee
1000
107
One figm
1000
The dha coas not boanch
102
argic irid: the orde
10
Frie soints en sto gft line aharra projection
102
Trangles
Irsitta ralde the outine stralght
Tha npli the dircle
Segmente
There branches second orde.
Emsty
Jiree a cetiocosinggg
14
Accie angle
57
Riuress are sillar
Convex Fauirs
57
More sclti blacce indea
93
Verceal reckang les or laontaal all paess
Long sdaa consore
25
Phinta ralee the Hig col no and stniggt
circtec strightt the
Figuree eiviler
the cutse paraiel
na dée tmoches the accond ander
hutchal
hale
sitee
Ourine figime
Extensione oewegeers
soltt
four
Angghh
dargutal figurs
Fou
Smooth contsurtgues
A lin with eetfcroecing
tnds of the ouree are opart
14
Eengated gares
Lane tooa length
Exes the cuive braatat
(a) Image-to-Side
(b) Description-to-SSdee
Figure 16: Per-concept difficulty distributions. Average number of correctly solved instances (out of 200) per concept across all models Bongard- for (a) I1S and (b) DIS tasks.
problems remain unrepresented in the extended Bongard-RWR+ dataset. As we note in Appendix E.2 this is mainly due to the limited number of images passing review and the difficulty of fully capturing complex synthetic concepts in the real-world images (e.g.. "spiral curls clockwise/counterllociii We believe that expanding Bongard-RWRR RWR++ to cover the missing concepts is a challenging yet promissing research direction, and encourage the community to contribute to this endeavor. Below, we outline several directions that look promising.
(1) Advancing T2I models. In our current pipeline, human verification served primarily quality control step. In practice, we filtered approximately 30.2% of generated images, which indicates that the majority of outputs were already acceptable. This suggests that the pipeline could operate with minimal human intervention or potentially in fully automated manner as generative models become more reliable at following complex prompts and avoiding rendering unintended concepts. Our approach is therefore forward cowarddcompatible with future advances in generative modeling.
(2) Reducing demographic bias. The demographic skew in our dataset (Appendix E) stems from the underlying T2I model. While our pipeline does not amplify these biases, future refinements could mitigate them directly. One approach is explicitly encode demographic diversity into prompts during both the Augment and Generate stages for example, by specifying varied gender, racial, or age attributes when appropriate for expressing an abstract concept. Additionally, greater diversity could be achieved by leveraging broader set of 12T, T2T, and T2I models within the pipeline, thereby reducing reliance on any single generative model.
verification. In our current pipeline, T2I model generates images from textual descriptions, and humans verify whether the outputs faithfully depict the intended concept. A natural extension is to introduce automated cycle-consistency check: after generating image from a source caption, an 12T model could re-caption the image to produce a reference caption. A T2T model would then compare the source and reference captions to assess semantic alignment.
(3)
26

---

Published as a conference paper at ICLR 2026
Table 14: Top correctly recognized concepts for each model using the IIS task on Bongard-RWRR
Model
Accuracy
Concept Group Left-side Rule
Right-side Rule
0.64
Contour
Shading thicker the right side
Shading thicker the left side Four identical elements
0.61
Similarity
Three identical elements
IVL2.5
0.61
Shape
circle
No circle
0.59
Count
Three parts
Five parts
0.58
Angle
Acute angle
No acute angle
0.65
Count
Empty picture
Not empty picture
0.62
Angle
A sharp projection
No sharp projection
Q2-VL
0.61
Angle
Convex figures
Nonconvex figures
0.59
Position
Axes of symmetry
No axes of symmetry
0.59
Branching
The chain does not branch
The chain branches
0.55
Angle
sharp projection
No sharp projection
0.55
Rotation
An acute angle directed inward
No angle directed inward
0.55
Similarity Position
All figures of the same color Points inside the figure outline are on a straight line Figures are similar
Figures of different colors
LLaVA
0.54
Points inside the figure outline are not on a straight line Figures are not similar
0.54
Similarity
0.61
Similarity
All figures the same color
Figures diffcrent colors Two figures
0.60
Count
One figure
0.58
Similarity
Figures similar
Figures are not similar
MCPM
0.53
Angle
Convex figures
Nonconvex figures
0.53
Branching
There are no side branches of the second order
There are side branches of the
second order
1.00
Count
One figure
Two figures
0.93
Size
Large figures
Small figures
Gemini 2.5 Pro
0.93
Shape
Triangles
Circles
0.92
Angle
sharp projection
No sharp projection
0.91
Similarity
All figures of same color
Figures different colors
0.97
Count
One figure
Two figures
0.92
Similarity
All figures the same color
Figures diffcrent colors
Claude Sonnet 4.5
0.91
Angle
A sharp projection Triangles
No sharp projection
0.90
Shape
Circles
0.88
Count
Three parts
Four parts
1.00
Count
One figure
Two figures
0.98
Position
Triangle inside of the eirele
Circle inside of the triangle Circles
GPT
0.95
Shape
Triangles
0.94
Angle
sharp projection
No sharp projection
0.93
Similarity
All figures of the same color
Figures of different colors
Images whose re-captioned descriptions fail to preserve key elements of the source prompt could be filtered automatically.
(4) Property-based testing. A more structured direction for automation is to move beyond free-form captions and operate on explicit image properties. During the Augment stage, the T2T model could be prompted not only to produce a natural language description of the scene, but also output set of properties that the generated image should satisfy to faithfully express the intended concept. An initial closed vocabulary could include attributes such size, shape, count, rotation, lighting, camera perspective, color, and distance. Alternatively, the model could generate open-vocabulary instance-specde -saanccc properties tailored to each concept. After image generation, an I2T model could verify whether these properties are present in the output. Images that fail to meet the specified property set (or fall below chosen threshold) could be filtered automatically.
We note, however, that while such model-based approaches could improve scalability, some human oversight will likely remain important in the near term ensure high data quality.
27

---

Published as a conference paper at ICLR 2026
Table 15: Top correctly recognized concepts for each model using the DIS task on Bongard-RWRR.
Model
Accuracy
Concept Group Left-side Rule
Right-side Rule
0.98
Count
One figure
Two figures
0.93
Count
Three parts
Four parts
IVL2.5
0.86
Shape
Triangles
Circles
0.84
Size
Large figures
Small figures
0.78
Count
Three parts
Five parts
0.96
Count
Three parts
Four parts
0.95
Count
One figure
Two figures
Q2-VL
0.85
Shape
Triangles
Circles
0.83
Size
Large figures
Small figures
0.81
Shape
Polygons
Curvilinear figures
0.86
Count
Three parts
Four parts
0.85
Count
One figure
Two figures
LLaVA
0.81
Shape
Polygons
Curvilincar figures
0.78
Size
Large figures
Small figures
0.71
Count
Three parts
Five parts
0.80
Count
Three parts
Four parts
0.73
Count
One figure
Two figures
MCPM
0.59
Count
Three parts
Five parts
0.59
Similarity
Three identical elements
Four identical elements
0.57
Similarity
All figures of the same color
Figures of different colors
1.00
Count
One figure
Two figures
0.98
Count
Three parts
Four parts
DS-R1
0.85
Shape
Polygons
Curvilinear figures
0.84
Shape
Triangles
Circles
0.78
Size
Large figures
Small figures
1.00
Count
One figure
Two figures
1.00
Count
Three parts
Four parts
Gemini 2.5 Pro
0.98
Position
Triangle inside of the cirele
Circle inside of the triangle
0.97
Similarity
All figures the same color
Figures diffcrent colors
0.96
Size
Triangle larger than circle
Triangle smaller than circle
0.99
Count
One figure
Two figures
0.98
Similarity
All figures of the same color
Figures of different colors Curvilincar figures
Claude Sonnet 4.5
0.94
Shape
Polygons
0.90
Count
Three parts
Four parts
0.88
Contour
Large total line length
Small total line length
1.00
Count
One figure
Two figures
0.98
Position
Triangle inside of the circle
Circle inside of the triangle
GPT-5.1
0.96
Similarity
All figures of the same color
Figures of different colors Curvilinear figures Four parts
0.95
Shape
Polygons
0.94
Count
Three parts
E BIAS AND FAIRNESS
E. DEMOGRAPHIC BIAS AND DIVERSITY
Our data generation pipeline includes several design choices aimed at promoting visual diversity and thereby mitigating potential biases. First, augmentations made Algorithm[ intrroduce variation in visual scenes, thus increasing conceptual and demographic diversity. For example, description originally referencing "A man standing with a surfboard" was augmented to "A woman standing with surfboard" (Fig. 19 Second, in Algorithm 2 we explicitly encourage visual diversity within BPs representing the same concept. This is accomplished through iteratively selecting image subsets that maximize intra-set visual diversity, as measured by the total pairwise cosine similarity of ViT-L/14 embeddings. Furthermore, after each matrix is constructed, we randomly exclude one image from the selected subset from future sampling, thus increasing the total number of unique images used.
28

---

Published as a conference paper at ICLR 2026
(a) Outline vs. solid figures
(b) Large small total line length
Figure 17: Failure mode examples, (a) Intern prediction: "Convex hull figure elongated Convex hull of figure compact"; explanation: "The images the left side feature objects or shapes that are more elongated, such as the person's 's face, the mirror, the stones, and the clock. [...]". Assessment: The omission of the pie in the description of the left side illustrates error type (I). Furthermore, while faces and mirrors indeed appeared on the left side, the stones and clock were on the right, exemplifying error type (2). (b) Qwen2-VL's prediction: "Closed lines vs. Open lines"; explanation: "The left side of the image contains natural landscapes [...]". Assessment: Although some left-side images indeed contain landscapes, landscapes also appeared on the right side, exemplifying error type (3). The explanation also reflects type (4) error, as it relies on surface-level features while ignoring the underlying geometric property.
Gemini 2.5 Pro
Claude Sonnet 4.5
GPT-5.1
Anpie
Anple
Anule
beition
Redison
Rodtion
Count
Coux
Model
CS
DIS
I1S
Stap
Shop
Gemini 2.5 Pro
0.82
0.70
0.65
Bozching
Bendhng
Bonching
Corrrol
Cvnnril
Ormni Sinlluity
GPT 5.1
0.81
0.67
0.65
Sinilatty
Claude Sonnet 4.5
0.76
0.67
0.63
DIS 115
D15
118
DIS
11
(a)
(b)
Figure 18: Proprietary model results Bongard-RWR+ in the CS (K -16), DIS, and I1S tasks.
Despite these methodological safeguards, Bongard-RWR+ may still inherit biases from the underlying models used in the generation pipeline, particularly Flux.I-dew I-dev for Text-to-Image generation. Since these models are trained on large-scale web data, they may reflect social, cultural, geographic biases present in that data. To better understand these risks, we conducted a limited-scale bias audit. Two expert annotators independently reviewed a subset of 800 images from Bongard- R depicting humans, and to each of them assigned labels related to Gender (Female, Male, Hard to tell), Race (Asian, Black, White, Hard to tell), Age (Adult, Child, Hard to tell), and Image type (Single, Pair, Crowd, Hands, Legs, Other). We assessed the annotators agreement computing Cohen's x for each the categories (Gender 0.82, Race 0.84, Age 0.78, Image type =
type= 0.89).
The performed analysis (Fig. 20 revealed that Flux. Flux.1-dee 1-dev produces relatively balanced Gender distribution (45.1% Female, 36.6% Male, 33.0% Hard to tell). However, the model shows racial annotated as White, rising to 98. 88.7% in images limited to human hands. Adults were far more frequent (67.9%) than Children (12.8%). Crowd scenes were largely racially homogeneous, with 82.2% appearing consist of single race.
imbalance: 79.9% of generated humans
29

---

Published as a conference paper at ICLR 2026
(a) "A man standing with surfboard.." was aug- mented to "A woman standing with a surfboard..""
(b) "A woman wearing retro glasses with a pink out- line..." was augmented to "A child wearing square glasses with yellow outline..".
(c) "A hearty bowl of Italian minestrone soup..." was augmented to "A bowl of That green curry with jas- mine rice..." "
(d) "A herd of deer standing together in dense forest..." augmented "A group of kangaroos hopping closely a grassy plain...".
Figure 19: Examples of (a) Gender, (b) Age, (c) Culture and (d) Geographic diversity augmenta- tions used in Algorithml
Sex
Age
Race
Hands
800
203
600
300
(36.6%))
(35.0%5
200
400
100
200
102
(22.2%)
200
50
(12.8%)
(6.05
48
(11.0%5
(11.0%))
(0.7%) (0.7%) (2.0%
Female
Black H2T
Figure 20:
Distribution of the four selected human categories in Bongard-RWR+ images (Gender, Race, Age, Image type). The blue dashed line indicates the expected value under equal group representation. H2T used an abbreviation for Hard tell.
E.2 ANNOTATION RELIABILITY AND BIAS
In our pipeline (Fig. 3p. the human role is purely discriminative rather than generative and limited filtering generated images. This substantially reduces the manual burden compared to annotation- heavy approaches such object detection image segmentation. Nevertheless, this dicriminative step may potentially introduce some bias. To mitigate this risk, each image generated by Algorithm I was independently reviewed by two expert annotators, who assigned one of the labels: Left, Right or None. This way, each annotator verified whether an image (1) adhered to the concept of its side of the matrix and (2) did not adhere to the concept of the opposing side. Images failing either criterion were flagged for exclusion.
Inter-annotator agreement reached Cohen's & of 0.64, indicating substantial consensus. Annotators disagreed side assignment (Left vs. Right) in 5.0% of the cases, disagreed between None vs. either Left or Right in 17.7%, and agreed on the same label 77.3%. Disagreements were resolved through discussion. In of the lack of an agreement, the image generated images were discarded. On average, we removed 52% of images for concepts that were ultimately rejected, compared to 23% of them for concepts that retained. Table 16 the
discarded. Overall, 30.2%
30

---

Published as a conference paper at ICLR 2026
(a)
Left:
Four
parts.
(b) Left: Ends the curve are parallel. Right: Ends of the curve are perpendic-
(c) Left: One line. Right: Two lines.
(d) Left: Convex figures. Right: Nonconvex fig- ures.
Right: Five parts.
ular.
Figure 21: Examples of images dropped during review, (a) Depicts six petals, and therefore does fit either side of the problem. (b) The ends of the chain extend beyond the image. making the concept ambiguous. (c) In front of the car there is single white line, but in the background two yellow lines are visible. (d) The blood vessel appears circular, which implies convexity; however, when viewed in three dimensions, it is concave.
percentage of generated images discarded during human filtering, broken down by concept group. Rotation concepts show the highest rejection rate (39%), followed by Contour and Count (both 30%), indicating these most challenging for the T2I model render faithfully. In contrast, Branching and Similarity have the lowest rejection rates (14%), suggesting they are easier for the pipeline to capture reliably. Overall, this strict filtering also explains why some concepts (e.g., "spiral curls bhockwise/counterrlowwwwse ") did not reach the target of 100 matrices, due to the limited number of images passing review.
Additionally, we performed a more detailed per-concept analysis, focusing on the top five concepts with the lowest and highest im- age dropout. Concepts requiring strong object recognition skills had the lowest dropout (under 5%). such as: "A sharp projection vs. No sharp projection", "Triangles vs. Other figures". "Three parts vs. Four parts". "Vertical hatched lines vs. Horizontal hatched lines", and "Triangle inside of the circle vs. Circle inside of the triangle". To understand why images were rejected, anno- tators explained their decisions for the highest-dropout concepts. The main reasons for rejection can be grouped into three cate- gories: (1) the image did not satisfy the concept constraints, (2) the image was ambiguous (e.g., conflicting cues in the foreground (3) the image did not provide enough infor- mation to judge the concept. For example, in "Ends of the curve are parallel vs. Ends of the curve are perpendicular", images were rejected mainly because at least one end of the curve lay outside the frame, preventing clear recognition of the concept. Selected examples are presented in Fig. 21
Table 16: Percentage of dis- carded generated images per con- cept group.
Concept Group
Rejection Rate
Rotation
39%
Contour
30%
Count
30%
Shape
27%
Position
26%
Size
19%
Angle
19%
and background),
Branching
14%
Similarity
14%
F MODEL PROMPTS
Prompts and correspond to the Describe, Augment, and Render steps in Algorithm 1 resp. Prompts
T0 and 12 are used for the CS, I1S, DIS, I2S, and D2S tasks. Prompt outlines the shared task introduction for both IIS and DIS, while Prompt 9 provides the common introduction for I2S and D2S. Prompts include 2 illustrative examples to help models understand the task format. Importantly, these examples use different class labels to avoid biasing the model toward always selecting the same answer in the examples. When applicable, models are instructed to respond in a structured JSON format. The validity these outputs is enforced using the Outlines decoding backend (Willard & Louf 2023
31

---

Published as a conference paper at ICLR 2026
Prompt 1:
Generate both positive and negative image descriptions.
You are given image and gereral concept. Your task is refine the concept into two concise, visually descriptive prompts: one that aligns with the image (positive prompt) and one that contrasts with it (negative prompt) Focus on making each prompt specific, the image, and reflective of the core idea. You don't t need to match every detail--just convey the main visual concept.
clearly grounded
### Example:
Image: Euman Legs wearing socks with vertical lines Concept: Vertical lines.
Positive prompt: Socks with vertical lines
Negative prompt: Socks with horizontal lines
Now, it's your turn: Concept: !concept)
### Instruction ons:
1. Generate positive and negative prompt based on the provided image and concept.
2. Answer using the fol following format: Positive prompt:
Negative pronpt:
G THE USE OF LARGE LANGUAGE MODELS
We used LLMs solely assist with polishing the manuscript, limited grammar correction and stylistic refinement. No model contributed to research ideas, analysis, or conclusions. our experimental pipeline, however, VLMs, LLMs, and T21 models served the primary tools for data generation. Specifically, they were employed for (1) producing image descriptions, (2) augmenting captions, and (3) generating synthetic images. addition, we evaluated the AVR abilities of several VLMs and LLMs.
32

---

Published as a conference paper at ICLR 2026
Prompt 2:
Augment positive image description.
You are tasked with modifying given prompt for diffusicn model, Your primary objective is to preserve the specified **concep*** while le altering unrelated details or environments.
You are encouraged to create diverse, creative, and unique
- augmentations that stay true to the concapt but introduce variety in interpretation.
### Example:
Prompt: "An empty white bowl with thin black rim placed on solid blue background."
Concept: "Empty picture"
Output Augmentations:
"An empty red plate on a wooden table."
"A clear glass cup sitting on marble countertop.",
"A white ceramic vase on patterned fabric."
]
Now, it's your turn:
Prompt: [pronpt]
Concept: (concept)
### Instructions:
1. Generate (* (ut_aaugmenttiionss unique augmentations for the provided prompt.
2. Cutput the resulta the following JSON string array format:
<<augmented__promp___1
<<ausmennted__roopp_222
Ensure that each augmentation aligns with the concept anc introduces creative variations in other details.
Prompt 3: Positive prompt for rendering new images. Negative prompt was provided unchanged.
Sharp, high qual ity image of <posi five prompt>.
33

---

Published as a conference paper at ICLR 2026
Prompt 4: Concept Selection (CS) task prompt.
You are vision understanding module expert) designed provide short, clear and accurate answers. The goal in solving Bongard Problem is to identify concept that differentiates the left and right matrix sides. You will receive list of concepts. Only one of them correctly describes the image.
Select the concept that correctly describes the image. Respond JSON using the following format.
EXAMPLE START
REQUEST
"concepts": [
"ieft"; "Parallel lines",
"right": "perpendicular" lines",
"label":
"ieft"; "Negative slope", "right": "Positive slope", "abbel" 2
RESPONSE
dexplanation": "The image presents several lines that don't intersect.",
"label":
EXAMPLE END EXAMPLE STARE
REQUEST
"concepts": [
:iftt": "Two shapes", "right": "Six shapes", "label":
"Left": "A cycle", "right": "No cycle" "label": 2
"Left": "Similar shapes", "right": "Different shapes", "Label"; 3
"left": "Small variance", "right": "Large variance", "Label"; 4
RESPONSE
"explanation": "All images on the left side feature circles of similar size, while the images on the right side present triangles, squares, and lines.", "labei":
EXAMPLE END RECUEST
<request> RESPONSE
34

---

Published as a conference paper at ICLR 2026
Prompt 5:
Shared task introduction used in both IIS and DIS settings.
You are a vision understanding module (an expert) designed to provide short, cLear and accurate answers. The goal solving Bongard corresponcing class. There two classes: LEFT and RIGHT. All images belonging to the
Problem is to classify a test image
LEFT class represent common, shared concept which is not present in any image from the RIGHT class, and vice versa all images belonging to the RIGHT class represent common, shared concept which is not present in any image from the LEFT class.
Prompt Image-to-Side (I1S) task prompt.
Solve the provided Bongard Problem. Respond in json using the following format.
EXAMPLE STARE RESPONSE
"concept"; "small big", "explanation": "The test image shows small shape, similarly all images on the 1eft side. Conversely, the images on the right side feature big shapes.", "answor"; "LEFT"
EXAMPLE END EXAMPLE STARE RESPONSE
"concept"; "two circles vs one circle", " "explanation" "The donut sits alone the primary round element
in the scene, matching the RIGHT examples where single rounded object takes visual focus amid other details, like the soup bowl or drum.", "answer"; "RIGHT"
EXAMPLE END RESPONSE
35

---

Published as a conference paper at ICLR 2026
Prompt 7: Description-to-Side (D1S) task prompt. (1/2)
Use the descriptions below to solve the provided Bongard Problem. Respond in JSON using the following format.
EXAMPLE START
REQUEST
"1eft rrr_deerripiiinsee
"A wired telephone with coiled cord resting on wooden - table",
"Power lines stretching across a clear blue sky with birds perched on them",
"A clothes drying rack with colorful clothes hanging under the
sunlight",
"A swing hanging from tree branch in park with child playing nearby",
"A needle with thread passing through fabric, placed on a sewing table",
"A pair of shoes with reatly tied laces placed on a tiled floor"
"right descriptions";
"A smartphone with a cracked screen lying on desk with
4 visible damage",
"A bird flying freely in the sky with its wings spread wide",
"A pile of folded clothes neatly stacked shelf in wardrobe",
"A park bench aurrounded by fallen leaves in autumn setting",
"A piece of fabric with intricate embroidery placed on sewing table",
"A pair of slip-on sandals placed on carpeted floor near doorway"
"test description"; "A sailboat with its sails fully open gliding across a calm lake under clear sky"
RESPONSE
"concept"; "line is present "explaration"; "The first test image presents a sailboat with ropes - hanging the sails. The ropes are lines, the same as items on 4 the LEFT, like the telephone cord, power lines, and shoelaces.". "answer": "LEFT"
line is present",
EXAMPLE END
36

---

Published as a conference paper at ICLR 2026
Prompt 8: Description-to-Side (D1S) task prompt. (2/2)
EXAMPLE START REQUEST
"left "left_descriptiine descrrptioonn" [
"Two bagels on a cutting board beside a knife", "A pair of headphones resting in on desk with the earcups facing up",
"Two apples placed next to each other
picnic blanket", "A pai of camera lenses lined up on shelf", "A bicycle lying on the ground with both wheels visible", "A snowman with two stacked parts, one above the other"
"right_descriptionns"
"Two forks placed on plate",
"A dartboard mounted on a brick wall with scores written on two
- papers beside it",
"A spotlight shining onto a dark floor surrounced by cables", tray of gingers bread men",
"A pie baking in the oven turntable with hand reaching toward
"A record apinning it",
"A teacup seen from above with steam rising",
1,
"test_ "rest_descriptoon" "An octagonal dessert plate holding single donut with sprinkles"
RESPONSE
"concept": "two circles vs one circle", "explanation "The donut sits alone as the primary round element in the scene, matching the RIGHT examples where single rounded object takes visual focus amid other details, like the soup bowl or drum.", "answer"; "RIGHT"
EXAMPLE RND REQUEST <request> RESPONSE
Prompt 9;
Shared task introduction used in both I2S and D2S settings.
You are a vision understanding module (an expert) designed to provide short, clear and accurate answers. The goal solving Bengard Problem is to classify each of the two test images the classes: LEFT and RIGHT. Each corresponding class. There class (either LEFT
RIGHT), All the LEFT class represent common, shared concept which is not present in any image from the RIGHT class, and vice versa all images belenging to the RIGHT class represent common, shared concept which is not present in any image from the IEFT class. The two test images belong to different classes.
image belongs to exactly
images bet onging
37

---

Published as a conference paper at ICLR 2026
Prompt 10: Images-to-Sioee Sides (12S) task prompt.
Solve the provided Bongard Problem. Respond in json using the following format.
EXAMPLE START RESPONSE
"concept": "small vs big", "first":
"explanation"; "The test image shows small shape, similarly as all images on the left side. Conversely, the images on the right side feature big shapes.", "answer": "LEFT"
b "second":
"explanation"; "The test image shows big shape, similarly as + all Lmages on the right. The images on the left, on the other hand, feature small shapes.", "answer": "RIGHT"
EXAMPLE END EXAMPLE START RESPONSE
"concept": "two circles one circle",
"first":
"explanation" "The donut sits alone as the primary round element in the scene, matching the RIGHT examples where single rounded object takes visual focus amid other details, like the soup bowl or drum.",
}, "answer": "RIGHT"
"second":
"The tennis net and two balls represent two circular elements in the scene, aligning with the LEFT examples where pairs of circular objects are present, such as the bagels, apples, or camera lenses.", "answer"; "LEFT"
EXAMPLE END RESPONSE
38

---

Published as a conference paper at ICLR 2026
Prompt 11: Descriptions-to-Sides (D2S) task prompt. (1/2)
Use the image and the descripti ons below to solve the provi ded Rongard Problem. Respond in JSON using the following format. EXAMPLE START REQUEST
"left deseeeeectppoons
"A wired telephone with coiled cord resting wooden table",
"Fower lines stretching across a clear blue sky with birds perched on them",
"A ciohes drying rack with colorful clothes hanging under the sunlight",
"A swing hanging from tree branch park with child playing nearby",
"A needle with thread passing through fabric, placed on sewing table",
"A pair of shoes with reatly tied laces placed on a tiled fLoor"
1.
"right_ dight_ddescriptionss"
"A smartphone with a cracked screen lying desk with visible damage",
"A bird flying freely in the sky with its wings spread wide",
"A pile of folded clothes neatly stacked on shelf in
- wardrobe",
"A park bench surrounded by fallen leaves in an autumn setting",
"A piece of fabric with intricate embroidery placed on sewing table",
"A pair of slip-on sandals placed on carpeted floor near doorway"
1.
"first": sailboat with its sails fully open gliding across calm Lake under clear sky", "second": "A Launch boat with an engine cruising through the water leaving a trail of waves behind"
RESPONSE
"concept"; "line is present
line is present",
"first":
"expianation". "The first test image presents a sailboat with ropes hanging the sails. The ropes are lines, the same as items on the LEFT, like the telephone cord, power lines, and shoelaces.", "answer": "LEFT"
"second";
"expianation"; "The second image shows a motorized launch boat, + which typically docen't have sails or ropes visible. This aligns more with the RIGHT side examples where no lines are present, like the smartphone, slip-on sandals, or park bench."
"answer"; "RIGHI"
EXAMPLE END
39

---

Published as a conference paper at ICLR 2026
Prompt 12: Descriptions-to-Sides (D2S) task prompt. (2/2)
EXAMPLE STARD REQUEST
"Left_ descriptions":
"Two bagels on a cutting board beside a knife", "A pair of headphones resting on up",
with the carcups facing "Two apples placed next each other
picnic blanket", "A pair of camera lenses lined up on shelf", "A bicycle lying on the ground with both wheels visible", "A snowman with two stacked parts, one above the other"
"right dessstttteessrriioonnss..
"Iwo forks placed on plate",
"A dartboard mounted on a brick wall with scores written on two
papers beside it",
"A spotlight shining onto a dark floor surrounded by cables", "A ple baking the oven next to a tray of gingers bread men", turntable with hand reaching toward
"A record spinning
it",
"A teacup seen from above with steam rising"
1,
"first": "An octagonal dessert plate holding a single donut with sprinkles",
"second": tennis with balls lying nearby"
RESPONSK
"concept": "two circles vs one circle", "first":
"explanation"; "The donut sits alone the primary round element in the scere, matching the RIGHT examples where single rounded object takes visual focus amia other details, like the soup bowl or drum.", "answor"; "RIGHT"
b.
"second":
"expianae ; "The tennis net and balls represent circular elements in the scene, aligning with the LEFT examples where pairs of circular objects are present, such as the bagels, apples, or camera lenses.", "answor"; "LEFT"
EXAMPLE END REQUEST
<request> RESPONSE
40

---

Published as a conference paper at ICLR 2026
SLIM
(a)
(b)
Figure 22: Bongard-RWR++ (a) Left: Figures elongated vertically. Right: Figures elongated
horizontally. (b) Left: Smooth contour figures. Right: Twisting contour figures.
AAAA
XXXX
(a)
(b)
Figure 23: Bongard-RWWR (a) Left: Extensions segments cross at one point.
Right:
Extensions of segments do not cross at point. (b) Left: Three parts. Right: Four parts.
(b)
Figure 24: Bongard-RWR+/122. (a) Left: Empty. Right: Not empty. (b) Left: Large figures. Right:
Small figures.
41

---

Published as a conference paper at ICLR 2026
BLNU BLD- YOU ELILY
20
HHald pou
calaue
Swarlond you,
(b)
Figure 25: Bongard-RWRR/L33, (a) Left: Vertical hatched lines. Right: Horizontal hatched lines. (b)
Left: Triangles. Right: Quadrangles.
EUROPEAN
LNGT
NATIONAL
FESTICS
199
Figure 26: Bongard-RWR+I1 (a) Left: An acute angle directed inward. Right: No angle directed inward. (b) Left: Neck horizontal. Right: Neck vertical.
(a)
(b)
Figure 27: Bon (a) Left: One figure. Right: Two figures. (b) Left: More solid black
circles. Right: More outline circles.
42

---

Published as a conference paper at ICLR 2026
Figure 28: Bongard-RWR+/L.. (a) Left: A large hole. Right: A small hole. (b) Left: Shading thicker on the right side. Right: Shading thicker on the left side.
43


Reviews:
[{"review_id": "8wYG0DGWta", "reviewer": "Reviewer_kdgb", "summary": "The paper introduces Bongard-RWR+, a large-scale benchmark (5,400 matrices) for abstract visual reasoning (AVR) that preserves the *fine-grained concepts* of classic Bongard Problems while using real-world-like images synthesized by a VLM-assisted pipeline. Starting from the concepts in Bongard-RWR, the authors (i) describe exemplar images via an I2T model, (ii) augment descriptions with a T2T model, (iii) render images with a T2I model, and (iv) perform human verification. The benchmark supports six task formulations—binary and paired image(s)/description(s)-to-side(s), multiclass concept selection, and free-form concept generation—and includes grayscale and varying-shots variants to analyze the role of color and number of demonstrations.\n\nAcross 4 strong open VLMs, results reveal a consistent pattern: performance is decent on coarse concepts (e.g., shape/size) but drops on fine-grained cues (e.g., contour, angle, rotation). Caption-then-reason pipelines help, and concept-selection accuracy scales with model size, but free-form concept generation remains weak. The authors further test functional equivalence between generated and real images and show broadly similar scaling trends, supporting the dataset’s validity.", "questions": "1. Have you considered automatic checkers that evaluate whether generated concepts *logically partition* the two sides (e.g., via structured parsers / learned validators), beyond surface n-gram overlap?  \n2. What fraction of generated candidates fail human vetting per concept family (e.g., angle vs. shape)? Any systematic failure patterns in the T2I step?  \n3. How sensitive are results to chain-of-thought or *explicit analogy* prompting in I1S/I2S (e.g., “explain the difference, then decide”) and to few-shot textual exemplars?  \n4. Could the benchmark be extended to interactive *concept discovery* (iteratively request descriptions or crops) and do models improve with interaction?  \n5. Beyond the bias audit, do you plan to ship per-matrix *concept tags* and *hardness annotations* so researchers can target specific failure modes?\n6. The manuscript shows “ICLR 2025” in the PDF header but the submission targets ICLR 2026.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clever use of I2T/T2T/T2I with human vetting yields large, diverse matrices that preserve fine-grained Bongard concepts rather than only coarse, object-level cues.  \n- Binary/paired side assignments, concept selection, and free-form concept generation give a multifaceted picture of AVR.  \n- Grayscale shows color is a distractor; more demonstrations help certain models; generated vs. real yields similar trends, supporting external validity.  \n- Per-concept-group breakdowns (size/shape vs. contour/angle/rotation) isolate where VLMs fail, informing future model design.  \n- A simple similarity classifier is competitive in image-based tasks, underscoring that current VLM prompting often fails to internalize the concept—a useful, humbling baseline.", "weaknesses": "- Caption/augmentation quality inherits biases and blind spots of the I2T/T2T models; although human verification mitigates this, a quantitative *masking* or *counterfactual* stress test of pipeline robustness would help.  \n- BLEU/ROUGE/CIDEr/BERTScore only weakly capture conceptual correctness and fine-grained relations; a concept-aware rubric (or human evaluation on a subset) would better reflect success/failure in CG.  \n- Most core experiments use a small pool of open VLMs; including stronger closed models (or stronger open ones as they appear) would calibrate the difficulty spectrum more completely.  \n- While the release mitigates this, adding inter-annotator agreement for the acceptance filter directly in the main paper (not just appendix) would strengthen transparency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMy0MG0SMW", "reviewer": "Reviewer_sMpD", "summary": "The paper introduces Bongard-RWR+, a new large-scale benchmark for abstract visual reasoning, expanding on the Bongard Problems by generating 5,400 instances using a vision-language model pipeline. Despite promising results with coarse-grained concepts, state-of-the-art VLMs struggle with fine-grained reasoning tasks, highlighting significant gaps in their capabilities for multi-image abstract visual reasoning.", "questions": "1.\tThe paper claims that the main advantage of Bongard-RWR+ lies in its scalability. However, the image selection still relies on manual review, and according to Appendix E.2, as much as 30.2% of images were discarded. This significantly weakens the argument of \"automation\" in dataset construction. Why not further quantify the impact of manual intervention on the results? Is there any evidence of systematic bias in image selection due to reviewer subjectivity? \n2.\tThe paper only compares InternVL2.5, Qwen2-VL, LLaVA-Next, and MiniCPM-o. However, stronger closed-source models, like GPT-4V and Gemini, have been used in several AVR benchmark tests. Why were these stronger models not included in the comparison? \n3.\tThe authors intentionally maximize intra-side visual diversity (e.g., in Bongard-RWR+/LP) to aid concept identification. However, this strategy risks introducing distracting, spurious concepts. For instance, representing \"Vertical\" with a tree, a building, and a person may also activate unrelated concepts like \"Nature\" or \"Architecture,\" potentially confusing the model. The claim that greater diversity makes concepts \"easier to identify\" is not systematically verified. The authors should analyze if an optimal level of diversity exists by examining whether accuracy on the Bongard-RWR+/LP variants plateaus or even decreases as P (and thus diversity) increases. \n4.\tAlthough the paper claims that “models perform poorly with fine-grained concepts,” there is no comparison with human-level benchmarks. The absence of such a comparison makes it hard to interpret whether the models’ performance is truly “bad.” \n5.\tAs shown in Figure 16 and Appendix E, many discarded images were due to unclear structure, background confusion, or perspective errors. This suggests that the main issue might lie in the ambiguity or noise within the images themselves, rather than the models' poor reasoning capabilities. The authors should provide a quantitative analysis of “image quality vs. accuracy” to demonstrate that the problem lies in reasoning rather than perception.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces Bongard-RWR+, an innovative extension of the Bongard Problems, using a semi-automated pipeline with vision-language models (VLMs) to generate large-scale, fine-grained real-world images, addressing the limitations of manual datasets like Bongard-RWR.\n2. The methodology is solid, ensuring image diversity and offering valuable insights through ablations on model size, color, and image diversity.\n3. The paper is clear and well-structured, with effective visuals. Its significance lies in advancing AVR, providing a scalable benchmark that exposes gaps in current VLM capabilities and guides future research in multimodal reasoning.", "weaknesses": "1.The paper lacks a detailed exploration of how the semi-automated pipeline can be further refined for more reliable image generation. Manual filtering still plays a key role, and automated verification could improve scalability and reduce bias.\n2.Although the experiments cover multiple tasks, the evaluation of fine-grained reasoning is limited. It would benefit from including more diverse models or comparing performance with human-level reasoning.\n3.Lastly, a deeper analysis of errors, especially in fine-grained tasks, is needed. Understanding specific challenges faced by the models would help guide future improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X4Wu7BZBxo", "reviewer": "Reviewer_SHcE", "summary": "This paper introduces Bongard-RWR+, a dataset of 5,400 Bongard Problems that uses AI-generated real-world-like images to represent abstract visual concepts from classical Bongard Problems. The authors develop a semi-automated pipeline using vision-language models to generate images, then evaluate state-of-the-art VLMs on multiple task formulations including binary classification, multiclass concept selection, and free-form text generation. Their experiments reveal that while current VLMs can recognize coarse-grained visual concepts, they consistently struggle with fine-grained concept recognition, highlighting limitations in their abstract visual reasoning capabilities.", "questions": "no", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a relevant problem by scaling up the Bongard-RWR dataset from 60 to 5,400 instances through a semi-automated generation pipeline, which represents a reasonable engineering contribution to the abstract visual reasoning benchmark landscape. The experimental evaluation is comprehensive, covering multiple task formulations (binary classification, multiclass selection, text generation) and including useful ablation studies on factors like model size, color versus grayscale, and number of demonstrations. The paper is generally well-structured and clearly written, with detailed appendices documenting the generation process, prompts, and bias analysis. The findings consistently demonstrate that current VLMs struggle with fine-grained visual reasoning, which confirms existing concerns about their capabilities.", "weaknesses": "The paper's core limitation is that it provides dataset scaling rather than methodological innovation. The reliance on generated images raises validity concerns, especially given significant demographic bias (79.9% White figures) and the number of exclusion of original concepts due to generation failures, suggesting the approach cannot capture full abstract reasoning complexity. The experimental analysis is shallow—it confirms known VLM limitations without investigating why models fail, lacks detailed error analysis or failure mode characterization, and dismisses notable performance differences  between real and generated images rather than examining their implications. The supervised baselines achieve random performance but receive minimal analysis, missing opportunities to understand whether difficulties stem from data, architecture, or fundamental reasoning gaps. Additionally, the finding that explicit image captioning improves performance suggests models may be exploiting caption artifacts rather than demonstrating genuine visual reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2GI8HRY8P", "reviewer": "Reviewer_c11F", "summary": "This paper is about Bongard Problems (a classical test of abstract visual reasoning).\n\nThe first contribution is a pipeline to generate such problems at scale, with synthetic photo-realistic images.\n\nThe second contribution is an evaluation of existing models on the resulting set of new problems.", "questions": "N/A\n\n---\nTips for the presentation:\n\n- Typo on the 2nd line of the abstract\n- Figure 5: may be interesting to mark the \"random choice\" (\"chance\") baseline for each K, e.g. as a small gray line?\n- Inconsistent capitalization in paragraph headers: e.g. L309 \"Concept selection\" (no need to capitalize \"selection\", it's still a simple noun even if it's the name of an experiment), L342, L357, L465\n- Figure 8: I don't think these are \"ablation\" experiments; ablation means *taking away* (some part of the model, usually). A correct name for these experiments could be, for example, a \"sensitivity analysis\".\n- In the experiments, it may be worth reminding a few more times what some key symbols mean (K, P), for the readers who will scan through the paper without reading everything top to bottom. Another option may be to replace these \"unguessable\" symbols by something more explicit (mabye something like N_candidates? I'm not totally sure it's a good idea, maybe it's too cumbersome). Or maybe include/define these symbols in one of the figures that describe the dataset/setup?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a gap in the literature: there was no large dataset of such photo-realistic BPs.\nBPs are a classical test so there is interest in keeping to use this paradigm to test AI models.\nThe literature review does a good job of reviewing the extensive related work in abstract visual reasoning.\n\nThe evaluation uses several (4) current large models. It includes several formulations of the task, and multiple interesting variations and ablations (model size, color vs. grayscale,", "weaknesses": "I don't see any clear weakness to this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)