Evaluating Compositional Scene Understanding in Multimodal Generative Models
Authors: Shuhao Fu
Andrew Jun Lee
Yixin Anna Wang
Ida Momennejad
Trevor Bihl
Hongjing Lu
Taylor Whittington Webb
Venue: Accepted by TMLR ()

Abstract: The visual world is fundamentally compositional. Visual scenes are defined by the composition of objects and their relations. Hence, it is essential for computer vision systems to reflect and exploit this compositionality to achieve robust and generalizable scene understanding. While major strides have been made toward the development of general-purpose, multimodal generative models, including both text-to-image models and multimodal vision-language models, it remains unclear whether these systems are capable of accurately generating and interpreting scenes involving the composition of multiple objects and relations. In this work, we present an evaluation of the compositional visual processing capabilities in the current generation of text-to-image (DALL-E 3) and multimodal vision-language models (GPT-4V, GPT-4o, Claude Sonnet 3.5, QWEN2-VL-72B, and InternVL2.5-38B), and compare the performance of these systems to human participants. The results suggest that these systems display some ability to solve compositional and relational tasks, showing notable improvements over the previous generation of multimodal models, but with performance nevertheless well below the level of human participants, particularly for more complex scenes involving many (>5) objects and multiple relations. These results highlight the need for further progress toward compositional understanding of visual scenes.

Published in Transactions on Machine Learning Research (04/2025)
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Shuhao Fu*, Andrew Jun Lee*, Anna Wang
Deportment of Psychology, University of California, Los Angeles
Ida Momennejad Microsoft Research, NYC
Trevor Bihl
Air Force Research Laboratory
Hongjing Lu
hongjing cuucla.edu edu Department of Psychology, Department of Statistics, University of California, Los Angeles
Taylor Webb
taylor. wewebbbbgaaa com
Microsoft Research, NYC
Equal contribution
Reviewed on OpenReview:
Abstract
The visual world is fundamentally compositional. Visual scenes are defined by the compo- sition of objects and their relations. Hence, is essential for computer vision systems to reflect and exploit this compositionality to achieve robust. and generalizable scene under- standing. While major strides have been made toward the development of general-purpose, multimodal generative models, including both text-to-image models and multimodal vision- language models, it remains unclear whether these systems are capable of accurately gen- erating and interpreting scenes involving the composition of multiple objects and relations. In this work, we present an evaluation of the compositional visual processing capabilities in the current goneration of texl-to-image (DALL-E 3) and multimodal vision-language models (GPT-4V, GPT-4o, Claude Sonnet 3.5, QWEN2-VL- and compare the performance of these systems to human participants. The results suggest that these systems display some ability to solve compositional and relational tasks, show- ing notable improvements over the previous generation of multimodal models, but with performance nevertheless well below the level of human participants, particularly for more complex scenes involving many (> 5) objects and multiple relations. These results highlight the need for further progress toward compositional understanding of visual scenes.
and InternVL2..53881
1 Introduction
A fundamental aspect of human visual processing compositional nature. By composing objects and relations in novel ways, we have the capacity to imagine and accurately perceive an infinite variety of visual
scenes, even highly improbable ones (Spelke & Kinzler 2007 Hafri & Firestone 2021 Cavanagh
2021

---

Published in Transactions on Machine Learning Research (04/2025)
This ability to exploit the compositional nature of visual scenes is an important priority for the development of general-purpose, human-like computer vision systems.
Recent progress in multimodal generative models has led to the development of systems with an impressively general range of visual capabilities, including systems for text-to-image generation (Radford et al., 2021 and multimodal vision-language models (Achiam et al. 2023). Notably, these systems have been touted for their compositional abilities, as evidenced by their ability to generate novel and unusual scenes depicting, for instance, 'a baby daikon radish in tutu walking dog', the famous 'avocado chair' (Heaven 2021 On the other hand, these systems also sometimes display basic failures of compositionality. For example, it was reported that the text-to-image model DALL-E 2 was incapable of reliably generating 'a red cube on top of a blue cube' (Ramesh et al. 2022), and followup work found that DALL-E 2 could not reliably generate images to match prompts describing scenes involving relations (Conwell & Ullman 2022 Marcus et al.
p.22p. Similarly, while the advent of multimodal vision-language models such as GPT-4 has been heralded as a major step toward the development of systems with general-purpose visual understanding (Wul et al.
2023 some recent. studies have identified major shortcomings in their ability to accurately interpret scenes, especially for scenes involving the composition of multiple objects and relations (Mitchell et al. Fahmanzadehgervi et al. 2024).
visual 2023
In this work, we assess the extent to which the current generation of multimodal general ive models has made progress toward developing a capacity for compositional image understanding. Our experiments are divided into two broad sections. In the first section, we evaluate the ability of the text-to-image model DALL-E 3 to generate images based on spatial and agentic relations. We first assess the degree of improvement for relational prompts (e.g., 'a blanket covering box') used in previous work (to test. DALL-E 2) (Conwell & Ullman 2022,, and then test novel variants of these prompts, including reversed prompts that evaluate improbable scenarios (e.g., 'a rabbit chasing tiger'), and compositional prompts involving multiple relations among several entities. We carry out online experiments with external raters to assess the extent to which the generated images accurately depict the objects and relations described in the prompts. In the second section, we evaluate the ability of multimodal vision-language models (GPT-4v, GPT-4o, Claude 3.5 Sonnet, TWEN2-VL-72B2 and InternVL2.5-38B) to infer relational patterns from a set of example images. We test problems involving both synthetic images (the Synthetic Visual Reasoning Test (SVRT) (Fleuret et al., 2011g) ) and real-world scenes (using the Bongard-HOI dataset (Jiang et al. 2022 and perform direct comparison with human behavioral data.
Our results indicate that multimodal generative models have indeed made some progress toward composi- tional scene understanding, with DALL-E 3 displaying improvements over DALL-E in its ability to generate images from relational prompts, and the evaluated multimodal language models all displaying some capacity to infer relational patterns. These successes are tempered, however, by persistent failures of compositionality, While DALL-E 3 is capable of generating images based more common relational prompts, performance degrades for reversed prompts involving less common scenarios, or prompts involving multiple relations. Similarly, all of the evaluated multimodal language models perform well below human participants in their ability to infer relational patterns, with performance falling to chance levels for problems involving many (> 5) objects. Overall, these results suggest that the current generation of multimodal generative models do not possess a robust capacity for compositional scene understanding.
A summary of our contributions provided below:
We propose two new datasets for evaluating the robustness of relational text-to-image generation through the use of reversed and compositional relational prompts.
We perform comprehensive human behavioral evaluation of relational image generation in state- of-the-art text-to-image model (DALL-E 3).
We perform a comprehensive evaluation of relational concept learning in five multimodal language models (GPT-4v, GPT-4o, Claude 3.5 Sonnet, QWEN2-VI-72B, and InternVT.2.5-3BBB) for both real-world and synthetic images, and directly compare the performance of these models with human behavior.
2

---

Published in Transactions on Machine Learning Research (04/2025)
2 Evaluating Relational Image Generation in Text-to-lmage Models
We evaluated the ability of the text-lo-image model DALL-E 3 to generate images based on prompts that described scenes involving relations. We performed three human experiments to evaluate the validity of images generated by text-to-lmage models. Section 2.1 describes experiment evaluating DALL-E 3 on the relational prompts used in Conwell and Ullman's 's study of DALL-E 2 (Conwell & Ullman 2022 Section 2.2 reports an experiment involving reversed versions of these prompts, intended to Lest DALL-E 3's ability lo generalize to unusual scenarios (e.g., rabbit chasing a tiger"). Section 2.3 describes an experiment involving
prompts with many objects and multiple relations, testing DALL-E's ability to compositionally generalize by combining relations into more complex configurations.
2.1
Image Generation with Basic Relational Prompts
2.1.1
Methods
We first assessed the ability of DALL-E 3 to generate images based on basic relational prompts, using the same prompts as those used in a previous study evaluating DALL-E2 (Conwell & Ullman 2022). We adopted
a design similar to the one used in previous work (Conwell & Ullman 2022], asking human participants to judge whether images matched the corresponding text prompts.
We used a set of 75 prompts introduced by Conwell & Ullman (202)) evaluate relational image generation. Each prompt involved two entities with a single relation between them (e.g., 'a tiger chasing a rabbit"). We found that it necessary to modify some prompts duc to policy violations with DALL-E 3 (c.g., prompts involving 'kicking' or "hitting'). 25 prompts were modified with slight wording changes, while respecting the criteria used to create prompts described by Conwell & Ullman (2022) (see Section A.2 Prompts were broadly classified into those that involved physical relations (in, on, under, covering, near, occluded by, hanging over, and tied to), and those that involved agentic relations (pushing, pulling, touching, chasing, hugging, helping, and hindering). We refer to this set of text prompts adopted from Conwell & Ullman (002)) as basic relational prompts.
A blanket covering box
A box under a bow!
child pulling box
woman
hindering robot
Examples with high agreement
Examples with low agreement
Figure 1: Examples of images generated by DALL-E 3 for basic relational prompts (prompts the describe scenes involving relations), using the 'natural' style. See Figure 14 for more examples.
Images were generated by prompting DALL-E 3 (Betker et al. 2023), text-to-image model developed by OpenAI, through the Microsoft Azure A (version 2024-02-01 01 for all experiments). DALL-E 3 instructed to use the exact prompt provided, and not to revise the prompt before image generation (prompts are sometimes automatically revised by GPT-4 before image generation). The final prompt used by DALL- E 3 was checked against the original prompt, and, if necessary, DALL-E 3 was prompted again, until the original and final prompts were identical. We generated 10 images for each prompt, with the following hyperparameters: quality set to 'standard', style set. to 'natural', and image size set to 1024 x 1024.
3

---

Published in Transactions on Machine Learning Research (04/2025)
We performed human behavioral experiment to assess the extent to which the images generated by DALL- E 3 matched the prompts, using design similar to Conwell & Ullman (2022 Participants were presented with a prompt, along with a collection of 10 images generated by DALL-B E 3, and asked to select all of the
images that matched the prompt. More details can be found in Section A.3
1.0
Model
DALL-E 2
DALL-E 3
0.8
Agreement
0.6
0.4
0.2
0.0
ocoluded by tied lo
pulling
hanging over
hitting
kicking
hindering
pushing
helping
under
touching
covering
near
in
on
Relation
Figure 2: Results for basic relational prompts. The Y axis indicates the agreement between prompts and the images generated by either DALL-E or DALL-E 2, as judged by human participants. Results for DALL-E 3 (in blue) are from the present study. Results for DALL-E (in red) are from Conwell & Ullman (2022.. Each point reflects the average agreement for an individual image. Horizontal lines indicate the mcan agreement for each relation, and boxes indicate 95% confidence intervals.
2.1.2 Results
Figure 2 shows the results for the experiment with basic relational prompts. 'Agreement' reflects the pro- portion of participants who indicated that the generated images matched with the corresponding prompts, where 1 means that all participants indicated that an image matched the prompt, and 0 means that no participants indicated that an image matched the prompt. Results are grouped according to the specific relations used each prompt.. Results from our experiment with DALL-E 3 are presented in blue, alongside the original results from Conwell & Ullman's 2022) exporiment with DALL-E 2 in red.
We found that DALL-E 3 displayed significantly improved capability, relative to DALL-E 2, to generate images based on basic relational prompts (paired t-test, DALL-E 3 vs. DALL-E 2: t(49) 8.552, p < .001), with higher agreement for all relations tested. For some rolations in particular, including spatial relations such as 'on', 'near', or 'in', DALL-E 3 displayed major improvements, with average agreement of greater than 0.8, compared with the very low agreement (< 0.4) observed in images generated by DALL-E 2. We did not observe any major differencee in performance between physical (in, on, under, covering, near, occluded by, hanging over, and tied to) and agentic (pushing, pulling, touching, chasing, hugging, helping, and hindering) relations. Figure shows some examples of the images generated by DALL-F. 3.
4

---

Published in Transactions on Machine Learning Research (04/2025)
To assess the extent to which the pattern of results might be inffuenced by the binary decision in the human behavioral task (i.e., participants only indicated whether or not an image matched the prompt), we also performed an additional experiment involving more fine-grained rating (a Likert scale from to 7). This experiment yielded qualitatively similar results (see Figure 13 in Section A.3). Overall, the results of our experiments with basic relational prompts indicated improved performance in DALL-E 3 relative to DALL-E, 2, with some relations in particular showing major improvements.
2.2 Image Generation with Reversed Prompts
2.2.1 Methods
Next, we investigated whether DALL-E' 3 was able to generate images based on prompts in which the subject and object were reversed. For instance, the basic prompt 'a tiger chasing a rabbit' was changed to 'a rabbit chasing a tiger', switching the roles of the two objects (see Section A.2 for more details about these prompts). The goal of this experiment was to evaluate whether DALL-E 3 was capable of generating relational scenes involving unusual arrangements, as this is typically taken be a key capacity enabled by compositionality (i..., unusual scenes can be imagined by combining familiar elements). Images were generated by querying DALL-E 3 in the same manner as the previous experiment, except that the style parameter was set to 'vivid'. We found that DALL-E 3 is more likely to generate matched images for reversed prompts using the 'vivid' style than the 'natural' style. We then performed a behavioral experiment to assess the agreement of the images with their prompts, using a similar design to the experiment with basic relational prompts. Due to the introduction of new policy warnings from reversed prompts, we made 15 more prompt revisions to 75 prompts from Experiment 1, with modifications to already modified prompts and 8 modifications to new ones. This set of 75 was then filtered down 30 final prompts along with their reversed variants (a total of 60 prompts). More details can be found in Section A.3
A spoon on bowl
A bowl on spoon
A tiger chasing a rabbit
A rabbit chasing tiger
Examples with high agreement
Examples with agreement
Figure 3: Examples of images generated by DALL-E 3 for basic relational prompts and their corresponding reversed prompts using the 'vivid' style. See Figure 15 for more examples.
2.2.2 Results
Figure phouss the results for the experiment with reversed prompts. DALL-E 3 displayed significantly worse agreement for reversed prompts than it did for the original basic prompts (paired t-test, basic vs. reversed prompts: t(29) = 3.668, p 0.001), though the extent of this effect differed between different relations. Some relations displayed almost no difference in performance between the basic and reversed prompts, including 'ncar', 'helping', and 'touching', while other relations displayed major difference in performance, including 'on', 'in', and 'chasing'. Notably, two out of the three relations for which performance was not strongly affected by prompt reversal, 'near' and 'touching', are symmetric in nature, so reversing the order of subject and object should not affect the mcaning of the prompt, whereas the three relations for which performance was strongly affected, 'on', 'in', and 'chasing', are all asymmetric. It is also worth noting that effect of prompt reversal particularly pronounced for the relation 'chasing', likely due to the fact that
5

---

Published in Transactions on Machine Learning Research (04/2025)
the reversed prompts described very unusual scenarios (such as rabbit chasing tiger). Figure examples generated by DALL-E 3 for basic and corresponding reversed prompts.
3
shows
To systematically assess the impact of prompt likelihood on the validity of the generated images, we measured the plausibility of each text prompt using the GPT-3 language model from OpenAI (the 'davinci-002-1" engine, available through the Microsoft. Azure APT). We used log likelihood (L.L); the sum of the log- probabilities of each token in a sentence, as measure of semantic plausibility (Kauf et al. 2024 Hu & Levy, 2023). We performed correlation analysis comparing LL with average agreement (between the prompt and the generated images). We found that LL had statistically significant (but modestly sized) correlation with agreement (spearman correlation r(598) = 0.090, 0.027), consistent with the interpretation that lower agreement for reversed prompts was driven by the lower likelihood of these prompls in DALL-E 3's training data. These results suggest that, although DALL-E shows an improved capability to generate relational scenes relative to DALL-E 2, is not able robustly generalize this capability to unusual scenarios that are less likely to be present the model's training data.
1.0
Prompt Type
Basic
0.8
Reversed
Agreement
0.6
0.4
0.2
0.0
near
on
in
helping
touching
chasing
Relation
Figure 4: Results for compositional relational prompts. The Y axis indicates the agreement between prompts and the images generated by either DALL-E 3, as judged by human participants. Each point reflects the average agreement for an individual image. Horizontal lines indicate the mean agreement for each relation, and boxes indicate 95% confidence intervals.
2.3
Image Generation with Compositional Prompts
2.3.1 Methods
Finally, we investigated DALL-E I's ability to generate images based on more complex prompts involving the composition of two relations. key feature of image compositionality the ability generate new visual scenes by combining multiple objects connected through various relationships. To test this capacity in DALL-E 3, we generated 60 prompts, each of which involved three entities linked by two relations (e.g.. 'a spoon tied to box, with the box near cylinder'). These prompts were divided into five categories: 10 prompts involved two instances of the same physical relation; 10 prompts involved two different physical relations; 10 prompls involved two instances of the same agentic relation; 10 prompls involved two different agentic relations; and 20 prompts involved one physical relation and one agentic relation. For each prompt, we generated 10 images, yielding 600 images in total. Images were generated by querying DALL-E 3 in the same manner as the previous experiment. We performed a behavioral experiment to assess the agreement of the images with their prompts, using similar design to the previous experiments (see Section A.3
6

---

Published in Transactions on Machine Learning Research (04/2025)
A bowl hanging over a teacup with teacup hanging over car
A child hugging man with the man pulling a box
A spoon in bowl with the bowl a box
An iguana pushing a robot with the robot a bowl
Examples with high agreement
Examples with low agreement
Figure 5: Examples of images generated by DALL-E 3 for compositional relational prompts using the 'vivid' for more examples.
style. See Figure
2.3.2 Results
1.0
0.8
Agreement
0.6
0.4
0.2
0.0
helping near naar in
in in
uedt helping hurronng on solping
rusging fied to hariging over huigiing near hilderi
under cinichingg chaming&
Windervig haar pulling
harrging over in
occlidted pusshing in
hunging puiling
hinderni hanrging Over & chaisnn fiee helping & pichiily
& tescching hindering
pullig & hughing
holping
cov novaniigg under
near feelpng
gustiing
lied to covaring ruiling
pussting prahiing tie chassng chaning
hanging over covering hanging oivrr ocolle aldted pushing
Perhhing toacchng chnnng
ocotuded by siiiing
pvey
oudideed by pulling helping finderin biriin chassu aoon dailiin pulling
accudded by
parthn & coverne miining anddrr under
ellided by under helpinn puiling misnung puiliing
acsluded by Pxclidedd
under à
Figure 6: Results for compositional relational prompts. The Y axis indicates the agreement between prompts and the images generated by DALL-E 3, as judged by human participants. Each point reflects the average agrecment for an individual image. Horizontal lines indicate the mean agreement for each relation, and boxes indicate 95% confidence intervals.
Figure 0 shows the results for the experiment with compositional relational prompts. Performance varied widely across relation combinations, with some combinations displaying relatively high agreement (o.g. 'in & in' or 'helping & near"). However, the majority of relation combinations displayed relatively low agreement, suggesting that DALL-E. had significant difficulty generating images that required the composition of

---

Published in Transactions on Machine Learning Research (04/2025)
multiple relations. Figure 5 shows examples generated by DALL-E 3 with text prompts involving multiple
objects and relations.
2.4 Summary of Relational Image Generation Results
1.0
0.8
Agreement
0.6
0.4
0.2
Basic
Reversed Prompts
Compositiinal
Figure 7: Summary results for relational image generation experiments with DALL-E Error bars reflect 95% confidence intervals.
Figure 7 shows summary of the results for the relational image generation experiments. These results highlight two major findings. First, DALL-E 3 displays a notably improved ability to generate relational scenes relative to DALL-E 2. However, DALL-E 3's performance is dograded significantly for prompts in which the order of subject and object are reversed (often yielding unlikely scenes), or prompts involving the composition of two relations. These failure modes indicate that DALL-E 3's capacity for scene generation is not robustly compositional, as compositional approach would enable entities to be combined in novel ways, and more complex scenes to be formed through the composition of multiple relations.
3 Evaluating Relational Concept Learning in Multimodal Language Models
We evaluated the ability of two multimodal variants of GPT-4, spt-4-vision-previer and gpt-4o, to perform few-shot relational concept lcarning. We evaluated these models using two datasets: Bongard-HOI (Jiang et al. 2022), involving action- and event-based relations (human-object interaction) naturalistic scenes, and the Synthetic Visual Reasoning Test (SVRT) (Fleuret al. 2011), involving visuospatial relations synthetically generated images. Both datasets test the ability learn abstract relational concepts from relatively small number of demonstrations. To ensure that the learned concepts gemuinely relational, these datasets employ 'hard negatives' - incorrect answers that share object-level features with the relational concept, but differ at the relational level (e.g., for the concept 'human rides bicycles', hard negative will involve a human and a bicycle with a different relation).
3.1
Learning Relational Concepts from Real-World Scenes
3.1.1 Methods
We first evaluated relational concept learning with real-world scenes, using the Bongard- dataset (Jiang ct al. 2022). This dataset contains 167 distinct concepts involving human-object intoractions. Each concept is defined by a subject--erbboojjee triplet, where the subject is always a human, the verb indicales an action (e.g., 'hold' or 'inspect'), and the object a common item (e.g., 'bicycle' or 'apple"). Each concept is illustrated by positive examples and negative examples, each presented a soparate image (Figure 8 In the standard evaluation methodology, 6 labeled images from each class are presented (12 total), and the remaining positive and negative images presented for classification.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Example Images
Target Images
ride bicycle
~ ride bicycle
Figure 8: Example relational concept from Bongard-HOO. Positive instances feature a human riding a bicycle, while negative instances feature a human and a bicycle with a different relation.
We evaluated gpt-4- vissii-rreiooowww preview and gpt-4o on problems taken from the Bongard-HOl test set, and com- pared their performance to human performance reported with the original dataset, well the HOTTrans baseline model (the best performing model in that work) (Jiang et al. 2022 These models were evaluated using the Microsoft Azure API. Due to the limited number of images that could be presented in the context window for these models (10 images maximum), evaluation was performed by presenting 9 labeled example images (randomly selecting either positive examples and 4 negative examples, or vice versa) followed by a single query image (randomly selecting either positive or negative example). Given the large size of the dataset (13,941 problems), we sampled 10 problems per concept for our evaluation (1,670 problems in total). Some problems resulted in content policy violations. After excluding problems that triggered policy violations, spt-4-vision-previee was tested 1,149 problems (involving 161 concepts) and gpt-4o was tested 1,156 problems (involving 160 concepts). Temperature was set to 0 when evaluating both models, top-p was set to 1, and the 'detail' parameter was set to 'high..
3.1.2 Results
1.0
1.0
0.9
0.9
Acccuracy
Acccuracy
0.8
0.8
0.74
0.7
0.6
0.6
0.5
Humens
gpt-4o
gpt-4v
HOlTrans
Original
Swapped
(a) Performance on Bongard-HOU
(b) Swapped class labels
Figure 9: Bongard-HOO results. (a) Few-shot classification accuracy for two variants of GPTT4 (gpt-4o and pot-4-vision-previee shortened to gpt-4v) vs. best previous model (HOITrans) and human performance. (b) Performance of gpt-4v original problems vs. problems with swapped class labels. Error bars rellect binomial 95% confidence intervals.
Figure 9a shows the results for our experiments with the Bongard-HOO dataset. Both GPT-4 variants significantly outperformed HOITrans, the best performing model from previous work (one sample proportion tests, gpt-4o vs. HOITrans: (1) 80.06, p 0.0001, 4pt-4-vision previ- vs. HOITrans: x1) 75.27, 0.0001), but performed significantly worse than the human participants on this task (one sample
9

---

Published in Transactions on Machine Learning Research (04/2025)
proportion task, human vs. gpt-4o: (1) 335.78, < 0.0001, human vs. spt-4-vision-ppevieer: x² 352.61, < 0.0001).
One possible concern with these results is that the Bongard-HOO dataset is publicly available on the internet, and therefore may in principle have been present in the training data for these models. To determine whether performance on this task depends on prior training, we performed an experiment in which the labels assigned classes were swapped (positive examples, with a label of 1, became negative examples, with a label of O, and vice versa). We tested ppt-4-vision-previie on the swapped-labels task, and compared performance to the standard version of the task (Figure 9b). There was no difference in performance for these two versions of the task (logistic regression, swapped labels vs. original labels: = 0.83, 0.41), indicating that performance on this task was not due to memorization of a publicly available dataset. However, it is also important to consider that this dataset features common relational configurations (e.g., a man riding a bicycle) that are likely to have been present some form the training data for these models. Thus, while we have ruled out the possibility of literal memorization, there still concern that performance may depend to some extent on general familiarity with the relations and configurations present in these problems. To address this, in the next section, we present results from a synthetic dataset (SVRT) with novel relational concepts not likely to be present in the training data.
3.2 Learning Relational Concepts from Synthetic Scenes
3.2.1 Methods
Same
Inside- Outside
Equal Pair Distance
Not In Between
Equally Far Satellites
Mirrored Objects
Shape
Positive
O
D
P
Negative
4 4
PV
P 4
P
Figure 10: Examples of relational concepts in SVRT. Each concept shows a positive and negative instance.
We evaluated relational concept learning with synthetic scenes using the SVRT datasct (Fleuret et al. 2011 This dataset features abstract relational concepts instantiated with synthetic figures, making it difficult to rely on previous experience with common real-world relational concepts. Additionally, many SVRT problems feature relational concepts involving a larger number of objects and relations, making it possible to test for the ability to learn more complex relational concepts.
The SVRT dataset consists of 23 distinct relational concepts, each involving between 2 and 6 objects. These concepts range from simple pairwise relations (e.g., same shape) to more complex configurations of multiple objects and relations (e.g., mirrored objects) (Figure 10p. We used the original codebase from Fleuret et al.) (2011) 2 to generate novel positive ive and negative instances for each concept. This allowed us to ensure that the images would not have been present in the models' training data.
We evaluated spt-4-vision-previeer and gpt-4o on these problems using the Microsoft Azure API. To evaluate broader set of models, we also evaluated Claude Sonnet 3.5 (through the Anthropic API), and two open- source VLMs: QWEN2-VL-72B and InternVL2.5-38B (these were evaluated locally a workstation with 4 NVIDIA GPUs). For each problem, we presented 1-9 few-shot examples, consisting of mixture of positive and negative instances (in random order). Each example was presented in separate image, and was accompanied by a message indicating the class (0 or 1). After presenting the few-shot examples, we presented a target image (randomly sampling either a positive or a negative instance) and prompted the
hhttps://leuurtt.rrgg. hitp///giiiiiie/eeeeee wwwan//ggwb//ggggeo http://wwwwbbbtwbb/gggg////////e suma
10

---

Published in Transactions on Machine Learning Research (04/2025)
model to make classification. For each relational concept (out of 23), and each number of few-shot examples (1-9), evaluated each model 10 times, using completely different images each time, resulting in a total of 2,070 tests. The models occasionally made formatting errors in its responses. In these cases, we re-prompted the model until it produced a correctly formatted response. Temperature was set to 0 when evaluating both models, top-p was set to 1, and the 'detail' parameter was set to 'high!.
We also performed comparison with human behavioral data from Lee participants were presented with SVRT problems in interactive episodes. For each episode, participants received series of positive and negative examples illustrating relational concept, and attempted to classify each example. After cach response, they received feedback indicating the correct classification. The previous examples from the current episode remained on screen, and were sorted based on class. The episode continued until participants made 7 correct classifications in a row. See Lee et al. (2023) for more details on the human behavioral experiment, and Section A.7 for more details on the analysis comparing VLMs with human performance.
(023): In that experiment,
3.2.2
Results
1.00
1.00
Humans
claude-3-S-sonnet intemVL2.5-388
Humans
daude-3-5-sonnee
0.95
gpt-4o
gpt-4o
IntemVL2.5-355
0.90
got-6-wision-previie
Dwen7-V1-77R
0.90
gnt-d-visoon-prewiee
Ower2-V1-776
0.85
0.80
0.80
Accuracy
0.75
Accuracy
0.75
0.70
0.70
0.65
0.65
0.60
0.60
0.55
0.55
0.50
0.50
0.45
0.45
0.40
0.40
3
9
Number Labeled In-Context Examples
Number Objects Concept
(a) Performance by number of in-context examples
(b) Performance by object count
Figure 11: SVRT few-shol classification accuracy for humans, gpt-4o, upl---vision-previee. claude 3.5 sonnet, Qwen2-VL-7P and InternVL2.5. -2.5--3BBBBBB by (a) number of in-context examples and (b) number of objects in the concopt. Points are averages across all SVRT concepts and attempts, coupled with binomial 95% confidence intervals. Curves are logistic regression fits with binomial coufidence interval bands. Dashed black lines indicate chance performance.
Figure I shows the comparison of human participants with GPT-4 on the SVRT. Human participants
significantly outperformed all models (logistic regression, pite---isiion-peeiiew:: 3 0.63, 12.01, p
0.0001; gpt-4o: B = 0.66, 12.65. p < 0.0001; claude 3.5 sonnet: B3 0.44, 8818. < 0.0001;
Qwen2-VL-72B: B = -0.70, = 13.52, < 0.0001; InternVL2.5-88 3 = -0.52, = 9.82, < 0.0001).
Both human participants and all VLMs displayed improved performance with more in-context examples (human: = 0.17, 2 = 15.57, p < 0.0001, spt-4-vision-previery
3 = 0.078, = 4.47, < 0.0001, gpt-4o: I 0.085, 2 4.88, p- < 0.0001: claude 3.5 sonnet: 8 = 0.096, 2 = 5.32, p < 0.0001; Qwen2-VLL-22B: 0.053, = 3.08, 0.0021; InternVL2.5-38B. 3 0.038, = 2.18. p 0.030). (Figure Ila), and degraded
performance function of the number of objects in relational concept (human: B - 6.18,
0.0001, previie:
-0.18, 5.12,p p 0.0001, gpt-4o:
-0.16, 4.69, 0.0001;
claude 3.5 sonnet: 3 0..114,.2 = 3.29. = 0.0010; Qwen2-VL--22B B = -0.16, = 4.61, < 0.0001: InternVL22.-38111 8 = -0.19, = 5.62,p < 0.0001) (Figure Iib.. Notably, performance for both GPT-4 variants and the two open-source VLMs was statistically indistinguishable from chance levels for problems with objects (gpt-4-vision-previce 72B: (1) 0.83. p 0.36; InternVL2.5-38B. x²(1) 1.070. 0.30). although Claude Sonnet showed
x1) = 0.3. p = 0.58; gpt-4o: x1) = 0.3. = 0.58; Qwen2-VI.
performance above chance for these problems (x²(1) -6.5,, 0.0089). Thus, while the VLMs that
11

---

Published in Transactions on Machine Learning Research (04/2025)
investigated displayed some capacity to learn relational concepts in this task, and sensitive to the same factors as human participants (number of in-context examples and number of objects per image), performance was nevertheless well below human level, and they were generally not capable of solving more complex problems involving a greater number of objects and relations. In the Appendix, we also present additional experiments that explore the role of chain-of-thought prompting (section A.8), and interactive testing of GPT-4 (section A.8.1j, finding that neither of these factors improve performance on this task.
4 Related Work
4.1 Datasets for Evaluating Compositionality in Neural Networks
A number of previous studies have investigated the ability of neural networks to process compositional scenes. Prior to the development of large-scale pre-trained models, several datasets were proposed for evaluating compositional visual processing and reasoning deep learning models, including Visual Genome (Krishna et al. 2017), CLEVR (Johnson et al., 2017). PGM (Barrett al. 2018), and OOD (Geirhos al.) 2021), typ- and compositional generalization (but with large, task-specific training
ically emphasizing scts).
4.2 Evaluating Compositionality in CLIP and Text-to- Image Models
More recent research has focused on evaluating the zero-shot and few-shot capability of pre-trained multi- modal generative models to process and generate compositional scenes. Lewis el. al. (2022 investigated the compositional processing capabilities of Contrastive Language-Image Pre-'Training (CLIP) (Radford et al., 2021), on which both text-to-image models and multinodal language models are based, finding that CLIP could compose concepts at the single-object level, but could not reliably compose multiple objects or rela- tions. Marcus et al. (2022) and Conwell & Ulman (002) investigated the ability of DALL-E 2 to generate images based on compositional and relational prompts. Our study adds to this work by investigating the ability of DALL-E to generate compositional scenes, finding that it has improved performance relative to DALL-E 2, but is still not robustly compositional, especially in settings with many objects and relations. Concurrent work from [Cowwell al. (2024) also shows that DALL-E 3 struggles with prompts that involve logical operators such as negation.
4.3 Visual Reasoning in Multimodal Language Models
Other recent studies have investigated the ability of multimodal language models (e.g., GPT-4v) to process compositional scones. Both Yiu et al. (2024) and Mitchell et al. (2023) evaluated GPT-4v on visual analogy problems. Yiu et al.] (2024) found limited success (GPT-4v was able to solve visual analogies for certain types of relations), whereas Mitchell al. (2023) found that GPT-4v performed very poorly on problems from the Abstraction and Reasoning Corpus (ARC) (Chollet) 2019). even for problems that human participants perform nearly perfectly on. Our work adds to these studies by investigating the ability of these models to infer relational concepts in a few-shot setting (whereas analogy involves the inference of relational pattern from just single example), finding limited success in this domain.
It is also worth noting that previous work has found that both convolutional neural networks and vision transformers can perform nearly perfectly on the SVRT with very large number of training examples (Kim) et al. 2018 Messina et al., 2021bba) b)). In this work we were interested specifically in the ability solve SVRT problems with relatively small number of examples (as is done with human participants). which remains
challenging setting /Vaishnav & Serre 2022 Webb et al.
2024b Mondal et al. 2024
5
Discussion
In this work, we have presented a broad-ranging evaluation of compositional scene understanding in multi- modal generative models. In our experiments with text-to-image models, we found that DALL-E 3 shows an improved capability to generate relational images relative to the previous generation of these models (DALL-
12

---

Published in Transactions on Machine Learning Research (04/2025)
E 2), but displays lack of robustness for images involving reversed (i.e., novel) relational configurations, and more complex scenes involving many objects and more than one relation. In our experiments with mul- timodal generative models, we found that all of the models we evaluated (including two multimodal variants of GPT-4, Claude 3.5 Sonnet, and two open-source models) displayed some capability for few-shot learning of relational visual concepts, for both real-world scenes and abstract visuospatial problems, but underper- formed relative to human participants, and did not show any capacity to learn relational concepts involving larger number of objects. Overall, these results present a complex picture, with the current generation of multimodal generative models displaying some capacity for compositional and relational processing, while still falling short of the robustness and generality exhibited by human visual processing.
These results raise the question of how to reconcile the complex pattern of successes and failures observed both in the present set of experiments and other recent work. A major theme throughout all of these studies is that the current generation of models seem to display especially poor performance in more complex multi- object scttings, Lewis et al. (2022) found that CLIP was capable of composing concepts at the singlo-object level, but could not do so reliably for multi-object scenes. This is echoed by our findings with lext-lo-image models, where performance is especially degraded for prompts involving large number of objects. In the case of multimodal generative models, both our results and the results of Yiu et al. (2024) indicate some capacity for processing of visual relations, whereas Mitchell et al.] (2023) find very poor performance on the ARC dataset. This discrepancy may also be explainable by a difficulty with multi-object processing: the SVRT and Bongard-HOO (both investigated in this work), and KiVA (investigated by Yiu et al. (2024) all involve a relatively small number of objects per image, whereas ARC problems typically involve a much larger number of objects. In the few cases that we investigated that did involve larger number of objects (for some SVRT problems), performance was very poor, with only one out of the five models that we evaluated showing performance better than chance.
Interestingly, similar results have also been observed for tasks that do not involve relations, such as count- ing Ranc ot al. 2024 Rahmanzadchgeryy et al., 2024), or multi-object scene description (Campbell et al. 2024). Indeed, Campbell et al. (2024) found that multimodal language models were able to solve visual analogy problems when they were decomposed into object-level representations, but could not do so reliably when the same problems were presented within single image. This pattern of results suggests that the poor performance observed in the current generation of multimodal models may be due primarily to more basic difficulty with parsing of multi-object scenes, rather than difficulty with relational processing per se. This would also explain the discrepancy between the relatively poor performance of multimodal models on visual analogy problems and the more robust capability of language models on text-based analogy problems (Webb el al. (202)), Musker el al. (2024). : Webb el al. (2024a); cf. Lewis & Mit.chel (2024)). )
Finally, it is worth considering how the compositional and multi-object capabilities of multimodal generative models might be further improved. [Campbell et al. (2024) argued that these difficulties are related to the
classic "binding problem' from cognitive science [Treisman & Gelade 1980 Greff et al. 2020 Frankland
ct al. 2021), which ariscs any time that a shared set of representational resources must be used to represent multiple objects, but without a mechanism for binding features at the level of objects. In recent years, object-
centric representation learning methods, most notably including slot-based approaches (Burgess et al. 2019
Locatello et al. 2020), have arisen as a solution to the binding problem, and have been shown to dramat tically improve performance in visual reasoning tasks that require multi-object compositionality (Ding et al.) 2021
Mondal et al. 2023 Webb et. al. 2024b Mondal et al. 2024). This suggests that a promising direction for future work may be to combine multimodal generative models with object-centric visual processing methods.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar Xxi::2303.08774, 2023.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Mcasuring abstract reasoning in neural networks. In International conference on machine learning, pp. 511 -200. PMI.R, 2018.
13

---

Published in Transactions on Machine Learning Research (04/2025)
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et. al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dalll-- pdf, 2(3):8, 2023.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Trina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene dccomposition and representation. arXie preprint 0rXiv:1901.11390 2019.
Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. arXiv preprint arxiv:2411 (00::2111.0000 1.00238, 2024.
Patrick Cavanagh. The language of vision. Perception, 50(3):195-213 2021.
François Chollct. On the measure of intelligence. ar preprint
2019.
Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint ar Xiv:2208. 00005, 2022.
Colin Conwell, Rupert Tawiah- Quaiah-Quashie, and Tomer Ullman. Relations, negations, and numbers: Looking for logic in generative text-to-image models. arXiv preprint orXi::2411.17666 2024.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances neural information processing systems, 34:9112 9124, 2021.
François Flcuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald Geman. Com- paring machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(4)):17611-17222 2011.
Steven M Frankland, Taylor Webb, and Jonathan D Cohen. No coincidence, george: Capacity- as the curse of compositionality. 2021.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethiee Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Aduances in Newral Information Processing Systems, 34:23885-23899, 2021.
Klaus Greff, Sjoerd Van Stecnkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. Xiv preprint arXic:2012.0523 052008 2020.
Alon Hafri and Chaz Firestone.
The perception of relations. Trends in Cognétive Sciences, 25(6):475-492
2021.
Will Douglas 2021.
could be the future of http://www.ccooollgooooorrrrrr/r common sense/ avocado- autchaar--uturr--ai--- ai-openai lepenni--dee--eee
Heaven.
This
avocado
armchair
AI,
URL
Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. anXiv preprint arXiv:2305.1322 13264, 2023.
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEFE/CVP conference on computer vision and pattern recognition, pp. 19056 19065, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and clementary visual reasoning. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2901 2910, 2017.
Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A Ivanova. Com- paring plausibility estimates in base and instruction-tuned large language models. arXiv preprint arXiv:2403.1443 14859, 2024.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Junkyung Kim, Matthew Ricci, and Thomas Serre. Not-so-clevr: learning name-different relations strains feedforward neural networks. Interface Socus, 844):20180011 2018.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Andrew Jun Lee, Hongjing Lu, and Keith Holyoak. Human relational concept learning the synthetic visual reasoning test. In Proceedings of the Annual Meeting of the Cognâtive Science Society, volume 45, 2023.
Martha Lewis and Melanie Mitchell. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv preprint arXiv:2402.08955 0885555 2024.
Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellic Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10597, 2022.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexcy Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525- 11538, 2020.
Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. arxio preprinl arXiv:2204.18807 2022.
Nicola. Messina, Giuseppe Amato, Fabio Carrara, Claudio Gennaro, and Fabrizio Falchi. Recurrent vision transformer for solving visual reasoning problems. arXiv preprint arXi:21111.14566 2021a.
Nicola Messina, Giuseppe Amato, Fabio Carrara, Claudio Gennaro, and Fabrizio Falchi. Solving the same- different task with convolutional neural networks. Pattern Recognition Letters, 143:75-80, 2021b.
Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning lasks. arXic preprint (tl::3311092243 2023.
Shanka Subhra Mondal, Taylor Webb, and Jonathan D Cohen. Learning to reason over visual objects. arXiv preprint Xrx::22003.022 022600 2023.
Shanka Subhra Mondal, Jonathan Cohen, and Taylor W Webb. Slot abstractors: Toward scalable abstract visual reasoning. arviv preprint Xiv:2440 03458, 2024.
Sam Musker, Alex Duchnowski, Raphaäl Millière, and Ellic Pavlick. Semantic structure-mapping in Ilm and human analogical reasoning. XrXiv preprint arXiv:2466. 13803, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Pooyan Rahmanzadehhervii, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint arXiv:2407.0658) 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.6112 1(2):3, 2022.
Sunayana Rane, Alexander Ku, Jason Baldridge, Ian Temney, Tom Griffiths, and Been Kim. Can generative multimodal models count to ten? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024.
Flizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89- -90,696,- 2007.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognâtive psychology. 12(1): 97-136, 1980.
Mohit Vaishnav and Thomas Serre. Gamr: A guided attention model for (visual) reasoning. arXiv preprint wrXi:2206.04928 2022.
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526 1541, 2023.
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Evidence from counterfactual tasks supports emergent analogical reasoning in large language models. arXiv preprint arXiv:2404.1307 2024a.
Taylor Webb, Shanka Subhra Mondal, and Jonathan D Cohen. Systematic visual reasoning through object- centric relational abstraction. Advances in Neural Information Processing Systems, 36, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-offthought prompting elicits reasoning large language models. Advances in neural in information processing systems, 35:24824 24837, 2022.
Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin. An early evaluation of gpt-4v (ision). arXiv preprint 211.:33310.1654,, 2023.
Eunice Yiu, Maan Qraitem, Charlie Wong, Anisa Noor Majhi, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Sacnko. Kiva: Kid-inspired visual analogies for testing large multimodal models. arXiv preprint ar axXiv:2407.17773, 2024.
16

---

Published in Transactions on Machine Learning Research (04/2025)
A Appendix
A.1 Code and Data Availability
All experimental materials, code, and data are available at:
compositionality VLMs
A.2 Prompts for Relational Image Generation Experiments
To test relational image generation text-to-image models, we used a set 75 relational prompts developed by Conwell Ullman (2022). Due to policy violations identified by DALL-E 3, we made the following modifications to these prompts:
kicking' + 'hugging'
'hitting' 'chasing'
knife' 'spoon'
la teacup near a cylinder' > 'a teacup near box'
'a monkey touching an iguana' + 'a monkey touching teacup'
a cylinder near bowl' -> 'a teacup near a bowl'
'a man hindering man' 'a man hindering robot'
'a woman pulling an iguana' + 'a monkey pulling an iguana'
'a child pulling a box' + 'a robot pulling a car'
a monkey hindering a monkey' > 'a monkey hindering an iguana'
aa man helping a man' + 'a man helping a monkey'
'a robot helping robot" + 'a robot helping man'
This list describes all modifications for both Experiments and 2. In Experiment 1, we sought to re-test the original list of 75 prompts but changed 25 due to policy violations. In Experiment 2, we modified 15 of these 75 (7 of the already 25 modified ones in Experiment and 8 new ones, resulting in 33 total prompts with at least one modification) to deal with reversed prompts that yielded policy violations, or prompts for which the reversed version was identical (e.g., 'a man helping a man'). From this set of 75, then selected 30 prompts and their corresponding reversed versions for Experiment 2. Specifically, agentic relations for which DALL-E' 3 displayed the highest level of agreement when generating images based on basic prompts. The selected physical relations were 'ncar', 'on', and 'in'. The selected agentic relations were 'touching', 'helping', and "kicking' Because prompls with the relation "kicking' frequently resulted in policy violations, we replaced 'kicking' with 'chasing' For each these relations, we used 5 basic prompts, and 5 reversed prompts in which the subject and object were swapped, resulting a total of 60 prompts (30 basic and 30 reversed). For each prompt, we used DALL-E to generate 10 images, yielding 600 images in total.
selected the three physical and
17

---

Published in Transactions on Machine Learning Research (04/2025)
A.3 Details for Human Behavioral Evaluation of Generated Images
We performed three human behavioral experiments to assess the extent to which the images generated by DALL-E 3 matched the prompts. To evaluate images generated from basic relational prompts (Section 2.1 we recruited 34 participants (26 female, average age = 20.18 years). Four of these participants were excluded from analysis because they indicated a lack seriousness in the post-experiment survey. To evaluate images generated from reversed relational prompts (Section 2.2), we recruited 43 participants (38 female, average age 19.6 years). One participant was excluded for taking minutes), and four of these participants were excluded because they indicated lack of seriousness in the post-experiment survey. To evaluate images generated from compositional relational prompts (Section 2.3 we recruited 42 participants (34 female, average age = 20.9 years). Five participants were excluded because they indicated a lack of seriousness in the post-experiment survey. All participants were recruited through the iredacted for anonymity] Psychology subject pool, and were compensated with course credit. All behavioral experiments were approved by the [redacted for anonymity] Institutional Review Board.
long complete the experiment (> 3000
All experiments were conducted remotely via online platform (Figure 12 On each trial, participants were presented with prompt, along with collection of 10 images generated by DALL-E 3, and asked to select all of the images that matched the prompt (Figure 12 Although there was no time limit, participants were encouraged to respond as quickly and accurately as possible. There were 75 trials for experiment 1 (basic relational prompts), 60 trials for experiment 2 (reversed prompts), and 60 trials for experiment 3 (compositional prompts). Trials were presented in a random order for each participant.
completed
Text Prompt: a blanket covering a bowl
any image larget contenoe, prees: Remembed It may be that all images match, no meges match or only some images match.
Figure 12: An example of the display presented to participants in the behavioral experiment.
18

---

Published in Transactions on Machine Learning Research (04/2025)
near
covering
touching
Relation
helping
pushing
hindering
kicking
hitting
hanging
pulling
tied
occluded
2
6
Rating
Figure 13: Results for basic relational prompts when participants responded using Likert scale from to instead of making a binary judgment. Relations are sorted along the Y axis based the rank order of agreement in the binary decision paradigm (in Figure 2). These results illustrate that the overall qualitative pattern (i.c., the rank ordering of the relations) is very similar between the two paradigms. Each point reflects the average rating for an individual image. Vertical lines indicate the mean rating for each relation, and boxes indicate 95% confidence intervals.
We also conducted an additional behavioral experiment in which participants assessed images using a Likert scale (from 1 to 7) rather than making a binary judgment. We recruited 45 participants for this study (37 female, average age 19.84 years). Three participants were excluded from the analysis because they indicated a lack of seriousness in the post-experiment survey. Each participant completed a total of 750 trials, each of which inchuded single image generated from basic relational prompt. The results of this experiment were qualitatively similar to the experiment that used binary decision (Figure 13), we used the binary decision paradigm for the experiments with reversed and compositional prompts.
19

---

Published in Transactions on Machine Learning Research (04/2025)
A.4 Additional Examples of Images Generated by DALL-E 3
A blanket covering a box
A box under a bowl
A child pulling box
A woman hindering a robot
Figure 14: Additional examples of images generated by DALL-E for basic relational prompts using the 'natural' style.
20

---

Published in Transactions on Machine Learning Research (04/2025)
A spoon on a bowl
A bowl on spoon
A tiger chasing a rabbit
A rabbit chasing a tiger
Figure 15: Additional examples of images generated by DALL-E 3 for basic relational prompts and their corresponding reversed prompts using the 'vivid' style.
21

---

Published in Transactions on Machine Learning Research (04/2025)
A bowl hanging over a teacup with the teacup hanging over a car
A child hugging a man with the man pulling a box
A spoon in a bowl with the bowl tied to a box
An iguana pushing a robot with the robot in a bowl
Figure 16: Additional examples of images generated by DALL-E 3 for compositional relational prompts using the 'vivid' style.
A.5 Analysis of Errors in Images Generated by DALL-E 3
We performed an analysis to better understand the types of errors found in the images generated by DALL-E 3. This analysis focused on the images generated for reversed relational prompts. We inspected each of the images with an average agreement of less than 0.2, and categorized them according to the following types of errors: wrong subject, wrong object, wrong relation, and relation binding error. For example, given the prompt 'a rabbit chasing a tiger', wrong subject error would indicate that the image does not contain rabbit, wrong subject error would indicate that the image does not contain tiger, wrong relation error would indicate that the image does not contain 'chasing' relation, and a relation binding error would indicate that the roles are reversed, i.e. the image depicts tiger chasing a rabbit. The results (Figure [7p showed that the overwhelming majority of errors involved either a wrong relation or a relation binding error.
A.6 Using Multimodal Language Models to Evaluate Images Generated by DALL-E 3
We performed an analysis to determine whether multimodal language models could be used automatically evaluate the images generated by DALL-E 3, focusing on the generated images for basic relational prompts. To do so, we prompted each model to rate whether a generated image matched the original prompt on a scale from 0 to 10. We then performed a correlation analysis between these scores and the average score for each image from the human behavioral experiment. As a baseline, we estimated the inter-subject reliability of the human ratings. This was accomplished by randomly splitting the human participants into two groups and computing the correlation between these groups. We performed this sinulation 100 times and computed the average of the correlation scores obtained in each run.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Number of errors
30
20
Wrong subject
Wrong object
Woog relation
Wrong relatton bindling
Error type
Figure 17: Error analysis for images generated by DALL-E
This analysis revealed that the human participant ratings were highly correlated with each other (human-to- human correlation: r == 0.91), whereas the models showed much lower correlations with the human ratings 0.51; Claude 3.5 Sonnet: 0.43; Swee22-VL72BB, 38B: 0.48). These results indicate that multimodal language models cannot reliably be employed to automate the assessment. of images generated by text-to-image models. This is consistent with the results of our evaluation of multimodal language models, which showed that they suffer from limited understanding of compositional images comparable to the limited ability of text-to-image models to generate those images.
(GPT-4v: 0.49; GPT-4o:
0.56; InternVL2.5
A.7 Comparison of Multimodal Language Models with Human Behavioral Data on SVRT
We compared a set of multimodal language models (including GPT-4o, CPT-4-visson--rreview, Claude 3.5 Sonnet, QWEN2-VL-72BB, and IntemnVL2.55388) with human performance on SVRT for each number of few-shot examples. In the behavioral experiment from Lee et al. (2023), episodes were terminated after participants made correct classifications in a row. As a result, in some episodes, participants received fewer than examples before the episode was terminated. For the purposes of comparison, we assume that participants in these episodes would have made correct classifications for subsequent examples. For instance, if a participant made correct classifications for the first. 7 examples (thus terminating the episode), we assume that they also would have made correct classification for an 8th and 9th example. This assumption was necessary to avoid biasing the distribution for 8 and 9 examples toward lower performing participants, since the best. performing participants were more likely to terminate episodes early by classifying all examples correctly. In Section A.8.] we present an experiment in which GPT-4 was evaluated using the same paradigm as human participants, allowing for closer comparison.
A.8 Evaluating GPT-4 with Chain-of- Thought Prompting
In the main texl, we evaluated GPT-4 on the SVRT and Bongard-HOI by presenting a set of labeled in- context examples followed by a target image for classification. Here, we also present results for experiments that used more extensive 'chain-of-thoughtt prompting strategics (Wei al. (202)). in which language model output is used to perform intermediate reasoning steps. We investigated four separate Chain-of Thouhht conditions, involving different amounts of intermediate computation:
23

---

Published in Transactions on Machine Learning Research (04/2025)
Chain-of-Thought O: No Chain-of-Thought is performed (this corresponds to the default results presented in the main text).
Chain-of- Thouught 1: In-context examples were presented one at a time, each coupled with a revised message instructing GPT-4 to describe the contents the image. These descriptions were then appended the context window. The final target/quers message was the same the Chain- of-Thought 0 condition.
Boain-of-Thoughie 2: Descriptions were not solicited for the in-context examples, but the tar- get/query prompt was expanded to encourage compositional thinking across all images. Specifically, GPT-4 was instructed to think carefully about the objects and relations in the series labeled examples, about the pattern of relations that may determine calegory membership, and apply the patterns to the target.
Chain-of-Thought 3: The combination of prompts for Chain-of-Thoouhh 1 and 2.
The results of these experiments are shown below. On Bongard-HOI, the Choug-of-Thought O condition performed as well or better than any of the other conditions (Figure 18 On SVRT, there were no discernible differences between the different Shain-of-Thougeee suggest that performance on these tasks cannot be improved via Chain-of-Thought prompting. This is consistent with the interpretation that performance in these models is constrained by basic difficulty with multi-object perception, rather than downstream reasoning processes.
conditions (Figure 19 Taken together, these results
Chain of Thought
Chain of Thought
Chain of Thought
Chain of Thought
1.0
Humans
0.9
Acccuracy
0.8
0.7
HorTrans
0.5
gpt-40
git-4-vision-preview
Figure 18:
Bongard-HOI Chain-of of----ooooooo Thought results. Dashed lines represent accuracies for humans and the HOTTrans baseline Jiing et al. 2022). Error bars reflect binomial standard errors.
A.8.1 Interactive Testing of GPT-4 on SVRT
We also performed an alternative evaluation of GPT-4 on the SVRT involving active learning paradigm, similar to the paradigm that is lypically used with human participants for this lask. In that paradigm, GPT- was prompted to categorize each in-context example one at a time, receiving immediate feedback after each classification. The model maintained context window of the previous classifications, which allowed it learn from past errors. On each trial, GPT-4 was presented with an image and asked to categorize it either positive negative. After each classification, the model received feedback ('Correct!' or 'Incorrect!') and
24

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
1.0
Classification Accuracy of Target Image
Classification Accuracy of Target Image
Chain of Thought
0.9
Chain of Thought
0.9
Chain of Thought
Chain Thought
0.8
0.7
0.5
0.4
0.4
6
Number of Labeled In-Context Examples
Number of Labeled In- n--Context Examples
(a) gpt-4o
(b) gpt visiio--preieee
Figure 19: SVRT Shain-of-Thoughe results for humans, (a) gpt-4o, and (b) gpt-t-vision-preview Points are averages across all SVRT concepts and attempts per concept, coupled with binomial standard error bars. Curves are separate logistic regression fits each chain-of-thought condition.
acknowledged the foedback before moving on to the next image. As with human participants, this process was continued until there were 7 correct classifications a row or the maximum of attempts was reached.
20.0
1.
1.0
Humans
Humans
gpt-4o
0.9
gpt-4o
gpt-4-vislon-preview
Probability Meeting Criterion
gpt-4-vision-previee
0.8
Trials to Criterion
12.5
Failure Rate
0.6
0.6
0.5
10.0
7.5
0.4
0.2
2.5
0.1
0.0
0.0
0.0
10
15
35
Agent
Agent
Trial
(a) ttials-to-criterion and failure rates
(b) Aalen-Johansen cumulative incidence functions
Figure 20: Active learning and survival analysis. (a) Trials-to-criters across all non-failed SVRT concepts and failure rates across all concepts with standard error of the mean and binomial standard error, respectively. (b) Probabilities of meeting criterion by trial, or cumulative incidence functions, estimated by Aalen-Johansen survival analysis, with binomial standard error bands.
Two dependent variables within the maximum 34 attempts) and trials-to-criterion (no data if concept failure; or total number of classification attempts before 7 correct in a row if criterion met). Figure 20 displays average trials-to- criterion over non-failed concepts and failure rates over all concepts.
measured per SVRT concept: concept failure (whether criterion met
We fit a lincar and logistic mixed-cffocts models to failure and trials-to-criterion data respectively, with participant type (human vs. gpt-4o vs. viso---issonppreviee) a fixed effect and problem ID as random effect. Pairwise contrasts with Tukey-adjusted p-values of estimates reveal that humans failed less fre-
25

---

Published in Transactions on Machine Learning Research (04/2025)
quently (gpt-4o: 2 p.14, < 0.0001: than than both variants (gpt-40: 1(625) = 3.96, p 0.0003; gpt-4-vision-preeiee:: ((624) = 3.31, 0.0028).
2 5.67, 0.0001) and met criterion faster
Because trials-to-criterion data are missing for concepts where criterion was not met (i.e., concept failure), average trials-to-criterion underestimate true rates learning. However, precise estimates of learning speeds can be established using survival analysis, method that takes into account censored data (e.g., the fre- quency of concept failures due to a ceiling of 34 attempts) to estimate trial-by-trial probabilities of meeting criterion. A standard approach to survival analysis is to use Kaplan-Meier estimator for such trial-by- trial probabilities. However, Kaplan-Meie assumes that censorship (i.e., concept failure) is independent of the probability of meeting criterion. Since our experiment violates this assumption by design, we used an Aalen-Johansen estimator with trials-to-criterion as the event of interest and concept failure (censorship) as a competing risk.
Figure 20p displays Aalen-Johansen probability curves. At least half of human participants meet criterion by the 9th trial, whereas gpt-4o and spi-4-vision-previer have 13% and 8.7% probability of meeting criterion by the same time. At trial 25, 6.5 people out of 10 meet criterion while neither variant reaches 50% of meeting criterion. To compare probability curves, we fit Fine- Gray competing risks regression with agent as fixed effect and SVRT concept ID as stratified predictor. The Fine-Gras regression with humans as the reference level reveals that humans reach criterion faster than either GPT-4 variant. (gpt-4o: SHR = 0.25, 95%C1 0.13 0.47, < 0.0001; gpt-4-vision vio---isioo--preieew: SHR 0.18, 95%CI 0.087 - 0.38, < 0.0001).
A.9 Testing with Swapped Labels on SVRT
1.00
Original class labels
0.95
Swapped class labels
0.90
Target Image Accuracy
0.85
0.80
0.75
0.70
0.65
0.60
0.55
0.50
0.45
0.40
Number Labeled In-Context Examples
Figure 21: SVRT few-shot classification accuracy for gpt-4-vision-preview with and without swapped class labels.
Finally, as with Bongard-HOL we tested whether GPT-4's performance can be explained by memorization of the SVRT dataset. Although all SVRT images were generated from scratch for our experiments, we confirmed that memorization could not explain our results by re-testing gpt-4-vision-preview with swapped class labels, There was no significant difference between performance with original vs. swapped labels (Figure 21 logistic regression: z = 0.87, = 0.38).
26


Reviews:
[{"review_id": "1SZbUPEdxX", "reviewer": "Reviewer_aDLR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper provides a set of well-designed experiments to evaluate modern multimodal models (DALL-E 3, GPT-4V, GPT-4o) in compositional vision scenarios.\n2. The authors have discussed the experiment results and concluded that while modern models do have some capability on compositional stuff, they still fall short of human performance.\n3. The authors talk about some potential directions that can improve the model's ability on the compositional ability, which can serve as a reference for people in the community who are interested.\n\n## Weaknesses\n1. The evaluation only involves 3 models  (DALL-E 3, GPT-4V, GPT-4o) from the same company (OpenAI). Not a single open-source multimodal model have been evaluated on these experiments. This can cause some the potential bias in the results.\n2. Most of the datasets used in the paper are simply re-purposed ones from previous works with little changes. There can be a lack of novelty since the results in this paper can already be reported by the authors of the original datasets. [1][2]  The authors need to clarify their differences with previous works the paper and highlight the unique contributions.\n3. The writing of the paper needs to be improved. On page 10, line 5, (We used the original codebase from https://fleuret.org/ ...) the website URLs are explicitly written in the main paragraph instead of appearing as a footnote, which can affect the reading experience. Also it's better to have subsections in the related work section to help readers better understand the context in different topics.\n4. The experiment number can sometimes not match the claim the authors have made. In second paragraph of section 2.2.2 Results, \"We found that LL was significantly correlated with the agreement (spearman correlation r(598) = 0.090, p = 0.027)\", the authors claim 0.09 as a high Spearman correlation, which is not the case. The authors did not explain where the magic number \"598\" comes from. \n\n\n**References:**\n\n[1] Conwell, Colin and Tomer David Ullman. “Testing Relational Understanding in Text-Guided Image Generation.” ArXiv abs/2208.00005 (2022): n. pag.\n[2] Jiang, Huaizu, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu and Anima Anandkumar. “Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 19034-19043.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigated how the multimodal generative and understanding model performs for the compositional instructions. For generative models, the authors investigated how DALL-E 2/3 performs for instructions combining the objects with 1 or more relationships. They found that the model can perform worse when the objects are revised across the relationship connecting them, or they meet more complex compositional instructions. Similarly, they investigated whether the multimodal understanding model can correctly do classification for images given a compositional text prompt. In conclusion, they found that while these current models do have some capacity in compositional problems, they still fall short of human performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add at least 2 more open-source models. For generative models, there are Mochi, CogVideoX. For understanding models, there are Qwen2-VL and Aria. And if with resources, Gemini and Claude can also be considered.\n2. Refer to the second weakness.\n3. Please improve the writing according to the thrid weakness. \n4. Please explain the Spearman correlation according to the fourth weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qowDN0WVmX", "reviewer": "Reviewer_CS2x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Main Concerns*\n1. Most of the statements tested were literal and descriptive; fewer abstract or metaphorical requests, such as \"happiness climbing a staircase of dreams,\" might see whether the model is able to deal with deeper or more symbolic themes.\n\n2. The paper doesn’t focus much to scenarios involving Counterfactual or other logical constructs, such as “an elephant reading a book while not standing on a surfboard.” Testing such prompts could offer more insight into DALL·E 3’s capacity for abstract reasoning.\n\n3.  Less obvious mistakes in very complex prompts are still difficult to find. There is high reliance on human evaluations; this raises scalability issues. Large-scale testing requires more automation.\n\n*Questions for Authors*\n1. In which way does the evaluation process take place? Are there any standard\nlabeling on which judges verify whether every element is aligned according to the text, or does it rely on a summary subjective opinion?\n2. How does this hold up against unusual prompts or highly specialized fields, like medical imaging or intricate diagrams?\n3. With the much more restrictive content filters of DALL·E\n3, does this restrict its capabilities in handling compositional instructions for borderline or sensitive prompts?\n4. Test prompts like \"an elephant reading a book while not standing on a surfboard\" require an understanding of negation. How would one go about effectively evaluating such cases, especially when training\ndata does not extensively see similar scenarios?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "*contributions*\n1. While improving compositional image generation is not new, the paper offers a systematic evaluation and pinpoints where models still struggle.\n2. Better compositional reasoning is a big deal for practical uses—everything from design workflows to storytelling can benefit. The analysis tackles real-world scenarios in generating images with multiple objects and relationships.\n\n*presentation*\n1. The paper clearly explains why compositional tasks matter and how the images are graded. The visual examples make it much easier to spot where things go right (or wrong).\n2. Each section flows nicely, clearly laying out how DALL·E 3 differs from earlier generative models.\n3. The side-by-side image comparisons and prompt examples are exceptionally well done, making it straightforward to judge whether the compositional instructions have been followed.\n\n*evaluation*\n1. The paper employs a combination of several clear-cut evaluation methods such as human judgments, the Likert scale, and binary compositional accuracy. Probably the most salient results include:\n1. The use of human judges ensures that the final scores align well with real-world understanding and preferences.\n2. Real-world data, from Bongard-HOI, but also synthetic data like SVRT featuring flipped and compositional\nprompts offer holistic views of model relational reasoning.\n3. Finally, the authors experiment with log-likelihood as a proxy to investigate how well the model's internal probability estimations reflect its actual performance, thus shedding light on possible biases in the training data.\n\n*main strength*\n1. This paper systematically tests a wide range of prompts covering spatial relations, object properties, and contextual interactions.\n2. The authors detail where the model stumbles—such as mixing up relationships when things get complicated, losing track of numbers, or drifting toward more common images in its training data.\n3. By pitting DALL·E 3 against DALL·E 2 and involving human evaluators, the study paints a clear picture of both the progress made and where improvements are still needed. The visual examples really drive these points home", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the weakness and question to authors\n\nMore hybrid metrics should be developed that would measure both accuracy and creativity.\n\nProvide fine-grained analyses of failure cases, especially involving spatial or relational mismatches.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1xhCQ4NuB", "reviewer": "Reviewer_Lrk1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- (S1) The work represents an interesting study of the capabilities of state-of-the-art models on complex scene understanding, incorporating multimodal perspectives through both text-to-image generation, and image-to-text annotation. \n- (S2) The work is thorough, extensive, and well grounded in qualitative studies with human participants, emphasising potential gaps in model performance and human performance on complex visual scene understanding tasks. These types of study are important for understanding current limitations in state-of-the-art multimodal models in common use today.  \n- (S3) Following S2, the results show gaps in model performance compared to human abilities in scene understanding. In particular, the gap between performance in Figures 6, 9, 11, were (in this reviewers opinion) quite striking. As such, the study serves as an informative read for modellers, and can serve to better illuminate potential gaps in multimodal learning signals. The authors highlight this in their discussion, and propose (at a very high level) more object-centric inductive biases or priors for representation learning paradigms. \n- (S4) The authors show thoroughness in the discussion of some of the experiments by pointing out that since it is not presently known what data was used to train the models, the results for Bongard-HOI could be a result of data leakage, i.e. the models could have, in fact, been trained on the dataset it is being evaluated on, which in turn, limits the significance of the results. Expanding the study to SVRT, and noting that this task results in a drop in performance, shows thoroughness and care in interpreting results and designing experiments. \n- (S5) The authors seem committed to reproducibility by stating their intent to release the code. \n- (S6) The appendix includes several additional results. The lack of improvement by chain-of-thought prompting, as well as the section on interactive testing both served as interesting reading.\n\n**Weaknesses**\n- (W1) While the study is comprehensive, the study focuses exclusively on closed-source models from OpenAI. While this is understandable given the ubiquitous use of DALL-E and GPT-4, one should consider that all models are trained by the same lab / company, and biases that may stem from standardised in-house modelling approaches may result in similar results and limitations observed in both text-to-image and image-to-text models. It is not unlikely (read; highly likely) that these models have been trained in conjunction with each other using a bootstrapping approach. Given that image-to-text and text-to-image models are essentially producing inputs for the other, this seems to be a natural distillation / tuning step. However we can only speculate as to what extent such a bootstrapping method was applied in the final product. Granted, the authors do not make expressive comparisons between the models from different modes, but this will likely affect the diversity of the results, and may lead to potential blind spots in the study.\n- (W2) Following the same line of argument, it is difficult to know exactly how significant parts of the study can reasonably be expected to be, particularly in the case for the image-to-text models. As the authors point out, it is difficult to use publicly available datasets for evaluation, as it is likely the data could have been sourced for training the model. For S4, I commend the authors in highlighting this issue. However, the generality of the study is somewhat inhibited by the choice to focus on closed-sourced OpenAI models. It also limits the practical applicability for the general community, since it is not clear that the results can be interpreted as general issues in multimodal modelling, or as a result of unknown confounding effects that could stem from particularities with OpenAI's training pipelines. While this could seemingly be considered tangential, broadening the study by showing similar characteristics (naturally, on a smaller scale) would alleviate any doubts that these are general phenomena, not inherent in strictly closed-source AI products available to the public as a product.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The work provides a thorough empirical study of complex relational scene understanding and composition for multimodal models, and compares this to a qualitative study using human participants. \n- The study evaluates the relational scene understanding of generative image models (DALL-E 2, DALL-E 3) with relational prompts, which are then reversed to generate novel scene configurations the model is less likely to have seen examples of. The study reveals improvement between model versions, but also reveals examples where ...\n- The study also evaluates GPT-4 variants on few-shot learning of relational scene understanding and composition on real world (Bongard-HOI) and synthetic scenes (SVRT). \n- The work reveals several cases where models score significantly lower than human participants for complex multi-object scenes.\n- From the results, the authors provide interesting observations and discussion around the implications of the study, suggesting that some of the observed limitations may stem from difficulties in parsing objects in complex multi-object scenes. \n- To alleviate the observed shortcomings, the authors propose integration of object-centric processing methods in representation learning setups, relating the issue to the \"binding problem\" in cognitive science.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the study is thorough and shows diligence, the issues raised by this reviewer is mostly focused on the question of generality; namely *are the observed results particularities in closed-sourced OpenAI models*, or *are these general limitations of multimodal AI models*? \n\nA set of requested changes are outlined below. We emphasise that **we do not expect a similar scope on any additional experiments as the main results**. \n\n- (C1) With regards to W1, given the non-zero likelihood of bootstrapped training between these models, it would be prudent to investigate the extent of any potential overlap between the evaluated models. In other words, **does poor performance in generating scenes with specific prompts correlate with weak annotations of such scene examples**? This could potentially also serve to show shared cross-modal limitations between the model representations. The paper’s discussion strongly emphasises that poor multi-object parsing might underpin the observed failures across tasks, including text-to-image generation and relational annotation. This aligns with our hypothesis; **there could be deeper dependencies across these modalities due to similar training biases**. Again, I want to point out that such a study does not necessarily need to be similar in scope as the original experimental results. A more limited test could help prospective readers in interpreting the results, and serves to illuminate potential blind spots in reading the paper.\n- (C2) Given the scope of the work, it seems unreasonable to ask for an equally broad comparison with open-source models. However, a study of more limited scope using open source alternatives seems advisable, if not completely necessary, to ensure the significance of the study is preserved. This reviewer would therefore recommend **a smaller scale study with open source alternatives** to ensure that the observed phenomena are general limitations of scene understanding in multimodal models. While this reviewer does not believe the results to be invalid, or non-transferrable to open-source models, showing that results extend beyond models trained by the same group / lab / company serves to emphasise the significance to the reader, and would strengthen the work.", "broader_impact_concerns": "As already mentioned, the lack of transparency in closed-source models represents an issue which readers may not necessarily be aware of, given the broadening of readership beyond academic circles. However, by adding additional experiments on open-source models, this can be discussed naturally in the text without requiring a broader impact statement. \n\nIn a broader context, the study is more an effort towards potentially mitigating relational and compositional biases in scene understanding. It is, in this reviewers opinion, therefore not necessary to extend the paper with a larger broader impact statement, as the paper highlights implicit limitations by the nature of the study itself.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the performance of multimodal generative and understanding models when handling under-compositional instructions. Reviewers agree that the study explores an important topic and features well-designed experiments. The authors have addressed concerns by incorporating additional results from various model families (e.g., Claude), open-source models (e.g., QWEN2-VL-72B), and providing necessary clarifications.

I encourage the authors to integrate these results and clarifications into the final version of the paper.
Decision: Accept (poster)