What Makes a Maze Look Like a Maze?
Authors: Joy Hsu
Jiayuan Mao
Joshua B. Tenenbaum
Noah Goodman
Jiajun Wu
Venue: ICLR 2025 Poster (2025)
Keywords: visual reasoning
abstract concepts
schemas
Primary area: datasets and benchmarks
TL;DR: A schema-based framework and benchmark for visual abstraction understanding.

Abstract: A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them. While off-the-shelf vision-language models excel at making literal interpretations of images (e.g., recognizing object categories such as tree branches), they still struggle to make sense of such visual abstractions (e.g., how an arrangement of tree branches may form the walls of a maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas—dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract components of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Benchmark, which consists of diverse, real-world images of abstract concepts and corresponding question-answer pairs labeled by humans. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models, and is a step toward human-aligned understanding of visual abstractions.

Published as a conference paper at ICLR 2025
WHAT MAKES A MAZE LOOK LIKE A MAZE?
Joy Hsu
Jiayuan Mao
Joshua B. Tenenbaum
Stanford University joyc:@stanford.eduu
MIT j jiayuarmemit.co edu
MIT
jbt @mit edu
Noah D. Goodman
Jiajun Wu
Stanford University
Stanford University j junwuêcs wtw.ssstttoo..ooooooo edu
ngoodman@staafooddeed
ABSTRACT
A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions reasoning about them. While off-the-stee fif-the-sheff lision-languag models excel at making literal interpretations of images (e.g.. recognizing object categories such tree branches), they still struggle to make sense such visual abstractions (c.g., how an arrange- ment of tree branches may form the walls of maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At graph descriptions of abstract concepts
the core of DSG are schemas-dependedr that decompose them into more primitive-leve symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract com- ponents of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Benchmark, which consists of diverse, real-world images of abstract concepts and corresponding question-ansee ansoer pairs labeled by humans. We show that DSG sig- nificantly improves the abstract visual reasoning performance of vision- langugge models, and a step toward human-ali aliigned understanding of visual abstractions.
INTRODUCTION
Humans possess the remarkable ability to flexibly acquire and apply abstract concepts when inter- preting the concrete world around us. Consider the concept "maze": our mental model can interpret mazes constructed with conventional materials (e.g., drawn lines) or unconventional ones (c.g., icing), and reason about mazes across a wide range of configurations and environments (c.g., in a cardboard box knitted square). Our goal is build systems that can make such flexible and broad generalizations as humans do. This necessitates a reconsideration of fundamental question: what makes maze look like maze? A maze is not defined by concrete visual features such as the specific material of walls or particular perpendicular intersections, but by lifted rules over symbols-a plausible model for maze includes its layout, the walls, and the designated entry and exit. Crucially, lifted model components can be realized by infinitely many real-world variations, opposed to grounded concepts which are tied specific visual inputs. For instance, walls constructed from candy canes, hay, or popsicle sticks, despite their diverse materials, are all identifiable as parts of a maze.
However, current lasion-language models (VLMs) often struggle to reason about visual abstractions at human level, frequently defaulting to literal interpretations of images, such as collection of object categories. These interpretations may be technically correct descriptions of the scene, but may not align with the abstract concept underlying the image. For example, when given the intended concept of "maze" and an image that realizes the concept with unconventional objects, VLMs such as GPT-4o (OpenAl 2024 and LLaVA (Liu et al. 2024) fail correctly recognize objects in the scene as components of the visual abstraction. is unclear whether these off the--hhhhhff models leverage knowledge of abstract concepts to make sense of images humans do.

---

Published as a conference paper at ICLR 2025
Abstract Concept: Maze
Deep Schema Grounding
LLM
Schema:
6920
walls
entry exit
V.
Q: "Imagine that the image represents maze. Are there clear entrances exits in the maze? Answer one the following answer choices: ycs, no."
Schema
(layout: grid-like walls: ice, extry- gates)
@
1
VLM
VLM answer: No
DSG answer: Yes /
Q: "Imagine the image represents maze. What is the player in this maze? Return none if there player. Answer with one the following answer choices: grass, moss, stick, none."
Schema
(layout: triangular,
walls: branches, entry-exit: corners)
V
@
VIM
VLM answer: Stick X
DSG answer: None /
Figure 1:
There exist abstract concepts, such as "maze", which are defined by lifted symbols and patterns, instead of concrete visual features. We propose Deep Schema Grounding (DSG), framework for visual reasoning over such abstract concepts, which uses schemas to structure models' interpretation of images. DSG hierarchically grounds conceptual schemas images and uses them to provide holistic context to VLMs, improving performance across diverse downstream queries.
To address the challenge of understanding abstract visual concepts, propose Deep Schema Grounding (DSG), a framework for models to interpret visual abstractions. At the core of DSG are schemas-a a dependency graph deseription of abstract concepts (see Figure I Schemas characterize common patterns that humans use to interpret the visual world, generalize efficiently from limited data, and reason across multiple levels of abstraction for flexible adaptation (Schank & Abelson 1975). A schema for "helping" allows us to understand relations between characters in a finger puppet scene, while schema for "tic-tac-toe" allows to play the game even when the grid is composed of hula hoops instead of drawn lines. A schema for "maze" makes maze look like maze.
DSG explicitly uses schemas generated by and grounded by large pretrained models reason about visual abstractions. Concretely, we model schemas as programs encoding directed acyclic graphs (DAGs), which decompose an abstract concept into a set of more concrete visual concepts as subcomponents, as illustrated in Figure The full framework is composed of three steps. First, we extract schema definitions of abstract concepts from large language model (LLM). Next, DSG hierarchically queries vision-language model (VLM), first grounding concrete symbols in the DAG (i.e. symbols that do not depend on the interpretation of other symbols), then using those symbols conditions ground more abstract symbols. Finally, grounding of all its components, as an additional input into a vision- lisiun-laggugge model to provide holistic context about the image, such that improve visual reasoning across diverse downstream
the resolved schema, including the
2

---

Published as a conference paper at ICLR 2025
queries about the abstract concept. Our method is general framework for abstract concepts without dependency on specific models; the LLMs and VLMs used are interchangeable.
To investigate the capabilities of models in understanding visual abstractions, introduce the Visual Abstractions Benchmark (VAB). VAB is a visual crestion-answering benchmark that consists of diverse, real-world images representing abstract concepts. The abstract concepts span different categories: strategic concepts that are characterized by rules and patterns (e.g., "tic-tac-toe"). scientific concepts of phenomena that cannot be visualized in their canonical forms (e.g., "cell"), social concepts that are defined by theory-of-ming relations (e.g., "deceiving") and domestic concepts of household objectives that cannot be directly defined by specific arrangements of objects (e.g, "table setting for two"). Each image is an instantiation of an abstract concept, and is paired with questions that probe understanding of the visual abstraction; for example, "Imagine that the image represents a maze. Are there clear entrances or exits in the maze?" The Visual Abstractions Benchmark comprises 540 of such examples, with answers labeled by 5 human annotators from Prolific. We found that off the-lff VLMs and integrated LLMs with APIs have much room for improvement this benchmark.
We evaluate Deep Schema Grounding on the Visual Abstractions Benchmark, and show that DSG consistently improves performance of vision-language models across question types, abstract concept categories, and base models. Notably, DSG improves GPT-4o by 6.6 percent points overall (1 9.9% relative improvement), and, in particular, demonstrates 10 percent point improvement (1 16.6% relative improvement) in questions that involve counting. While the challenge of visual abstraction understanding is still far from being fully solved, results show that DSG is promising solution that effectively uses schemas to structure the thinking process of visual reasoning systems.
In summary, our contributions are the following:
We propose Deep Schema Grounding (DSG), a hierarchical, decomposition baased framework that explicitly extracts and grounds schemas of concepts with large pretrained models.
We introduce the Visual Abstractions Benchmark (VAB), diverse, real-world visual question- answering benchmark that evaluates VLMs' understanding of visual abstractions.
We demonstrate DSG's improvement over prior state-of-the-ar works across variety of abstract concept categories, question types, response types, and metrics.
2
RELATED WORKS
Vision-language grounding and reasoning benchmarks. Visual-language benchmarks have traditionally tackled reasoning about content images, such as objects, relations, and actions (Chen et al. 2013| Yu et al., 2015), Antol et al.) 2015] Wu et al.] [016)] Zhu et al.] [016;) Krishna et al. Wang et al.) 2017). Recently, an increasing number of works have focused on evaluating visual-language tasks that require commonsense reasoning, usually by asking "what if" or "why" questions (Pirsiavash et af. 2014, Wagner et al. 2018 Zellers et al. 2019 Lu et al. (002),, or by generating holistic scene descriptions (Song et al. 2024). In contrast to these related works, emphasizes the visual grounding and reasoning problem of abstract concepts, such as uncommon visualizations of scientific terms unconventional constructions of strategic games. Such abstract concepts should be grounded on the relationship between lower-level concepts such as entities and patterns, therefore positing novel challenges vision-language models.
2017
paper
Concept representations in minds and machines. Our schema- based concept representations are inspired by the theory-theor of concepts in cognitive science and artificial intelligence (Schank
& Abelson 1975. Morton, 1980, Carey 1985 Gopnik 1988 Gopnik & Meltzoff 1997 Carey 2000). In short, these works treat concepts as organized within and around theories. Therefore, acquiring the concept involves learning its theory, and reasoning with a concept involves causal- explanatory reasoning of the theories. Such views have been largely leveraged in generating symbolic representations for concepts such as word meanings (Speer et al. (007)) and object shapes (Biederman 1987). However, most works have focused on using only symbolic features to describe scenes and have been primarily applied to simple object-level concrete concepts. In contrast, we propose using our schema representation
guide for grounding scene-level, abstract concepts in natural images.
Hierarchical grounding of visual concepts. Our approach is related to recent works on chain- of-thought reasoning in vision- Lanoung models (Lu et al., 2022 Chen et al. 2024), hierarchical image classification (Koh et al., 2020: Menon & Vondrick 2023), hierrccical relation detection (Li
ct al. 2024), and visual reasoning via programs (Hu et al. 2017 Mao et al., 2019 Hsu et al., (024). In contrast to these algorithms, which are primarily designed for object and relational reasoning,
3

---

Published as a conference paper at ICLR 2025
Extracting Schemas of Abstract Concepts
Schema:
"Maze"
(en(cencept=maze)
LLM
gen(layout concept-naze)
Prompt
gen(walls concept=maze) gen(entry-exit concept=maze, layout)
Hicrarchically Grounding Schemas on Images
(layout: rectangular, walls: coffee beans, entry-exit: coffee cups)
VLM
Q:
"Imagine that the image represents maze, what the layout?"
Schema:
Q: "Imagine that the image represents maze, what are walls?"
naze
layout
va walls
Q: 'Imagine that the image represents maze, and the layout (VLM layout prediction), what is the entry and exit?"
entry- exit
Visual Question Answering Augmented with Resolved Schemas
"Imagine that the image represents a maze, the layout rectangular, the walls coffee beans, and the entry and exit are coffce cups."
VLM
A: "None"
"What is the player in this maze?"
Figure 2: Deep Schema Grounding consists of three main stages: (1) extracting schemas of abstract concepts with large language models, (2) hierarchically grounding schemas on images with vision- language models, and (3) conducting visual
augmented with resolved schemas.
focus the grounding of scene-level concepts. Our framework also shares similar spirit to prior approaches (Menon Vondrick] 2023] Li et al. 2024) that leverage vision-language models generate text descriptions of class features for classification tasks. However, our paper proposes the generation of hierarchical schemas with LLMs to ground visual abstractions for reasoning. Additionally, our hierarchical schema representation is related hierarchical program representations used in text-only reasoning (Gao et al. 2023 Zelikman et al. 22033 Zhang et al. 2024); in contrast, we leverage program representations for understanding concepts natural images. Notably, DSG performs schema inference with a VLM; compared to fully symbolic inference processes, our framework is capable of leveraging all visual information in the image for interpreting abstractions.
DEEP SCHEMA GROUNDING
We study the task of reasoning about visual abstractions by concretely focusing on its visual question- answering (VQA) form, although our idea naturally generalizes other visual interpretation and reasoning tasks. A VQA instance a tuple of (v, q, a), where an image, q is natural language question, and is the answer to the question. Here, free-form responses in natural language. We assume that each image has an underlying abstract concept associated with it (e.g., "maze"), which is not associated with concrete visual features, but a set of lifted rules that define the concept. We also assume that g is question about the abstract concept, and the concept is revealed to all models in question q-
take the form of multiple choices or
The visual abstraction understanding task posits a series of challenges to off- the- VLMs. These questions centered around concepts with infinite possible instantiations in the real world. As
4

---

Published as a conference paper at ICLR 2025
Tic-tac-toe
Cell
tic-tac- toe
cell
syrbols
board
aeabranc
strotoay
organelle
Deceiving
Table Setting for Two
table- setting- for-tup
deceiving
chatrs
table
-doject
Figure 3: Examples of schemas for concepts and the visual features that they may be grounded to.
result, current VLMs struggle with this problem due to the potential novelty of the scene and the visual abstraction combinations encountered. However, crucial insight regarding these abstract concepts the consistency in composition patterns of lower-level concepts, which aggregate to construct the high-level concept; these underlying patterns remain the same across different variations.
Based on these insights, propose Deep Schema Grounding (DSG), framework that leverages schemas to interpret and reason about visual abstractions. DSG can be built on top most pretrained dision-language models (VLMs) to perform the visual question-answerne the DSG framework consists of three main steps: (1) extracting a schema the concept, (2) hierarchically grounding the schema to the visual input, and (3) leveraging the resolved schema as input to the base VLM. We describe the schema definition and each stage in detail below.
task. Illustrated in Figure 2
3.1 VISUAL ABSTRACTION SCHEMA
A visual abstraction schema is a concise program that defines a directed acyclic graphical (DAG) representation of partieular concept. As illustrated in Figure each node in the schema corresponds to subcomponent concept of the higher-lewe abstract concept. For example, the formation of maze can be decomposed into three components: the layout, the construction the walls, and the positioning of the entry and exit of the maze. The dependencies among individual components yield DAG configuration-in this case, the placement of the entry and the exit of the maze depends on layout of the maze. From inference perspective, the dependency graph describes the order in which the model should interpret each component, and how the interpretation of a particular component should be conditioned on the interpretation of
previously interpreted components.
The most important feature of a visual abstraction schema is that the definition of such concepts is universal, in the sense that the program describes universal characterization of mazes made of various components, rather than being restricted to specific visual instantiations (e.g., mazes formed by drawn lines). Hence, schemas enable machines better generalize novel scenes.
3.2 EXTRACTING SCHEMAS OF ABSTRACT CONCEPTS
DSG utilizes large language models (LLMs) as a repository of knowledge from which schemas can be acquired. Given that LLMSs are trained on extensive corpora of human language data, we hypothesize that they contain muman-aligned schemas for wide range of abstract concepts. Compared to prior works that use LLMs to translate natural language utterances into programs, DSG uses LLMs to extract explicitly structured knowledge of abstract concepts based on the concept's 's name.
In particular, goal is to extract universal schema for abstract concepts that is not instance-specifie but is applicable across diverse visual stimuli. Notably. our prompt to the LLMs is generic and simple, with only example of schema for the abstract concept "academia" (which consists of key components: faculty, students, etc.), and a short instruction of the schema generation task. We find that L.LMs demonstrate remarkable ability to extrapolate across wide array of concepts, and
5

---

Published as a conference paper at ICLR 2025
produce faithful decompositions of the visual abstractions. We detail the prompt and LLM- -eneerttd schemas for each concept in Appendix C
3.3 HIERARCHICALLY GROUNDING SCHEMAS ON IMAGES
Instead of directly answering questions given the image and the question inputs, we first hierarchically ground individual components in the schema to visual entities in the image. In the maze example in Figure 2 this corresponds to inferring the materials of the walls, the layout of the maze,
Formally, given a schema definition of the con- cept. DSG hierarchically grounds each component in the concept DAG. The outcome of this process represented as short text descriptions, such as "wall: coffee beans".. We implement this procedure by leveraging pretrained VL.Ms to predict the most likely grounding of component given the image and text query (see Figure Since our grounding procedure is hierarchical, for components that are conditioned on other concepts (e.g.. the interpreta- tion of entry-exit is conditioned on the layout), the VLM also takes into account the grounding results of these prerequisite components. This ensures that each step of the grounding process is informed by the previously established context, facilitating more coherent and accurate grounding.
gen concept=maze)
Q: "Imagine that the image represents a maze, what the layout?" A: Rectangular.
gen gen(walls concept-naze)
Q: "Imagine that the image represents a maze, what are the walls?" A: Coffee beans.
gen (enttry-eext | concept=maze, layout)
Q: "Imagine that the image represents a maze, and the layout is rectangular, what is the entry and exit?" A: Coffee cups.
Figure 4: The schema grounding process.
Grounding components following the hierarchical schema structure is crucial for abstract concepts with many possible instantiations. For example, for the "maze" concept, DSG first resolves compo- nents that are more coarse and concrete (e-g., layout), then uses those components as conditions to resolve more difficult grounding problems of fine-grained and abstract components (e.g., entry-exit). Figure 3 shows examples schemas and how they may be grounded onto images. More examples and failure modes be found in Appendix
3.4 VISUAL QUESTION-ANSWERYY AUGMENTED WITH GROUNDED SCHEMMS
Finally, DSG leverages the grounding of individual components the image to answer questions. For example, given the concept "maze" and the grounded mapping of individual components: {layout: rectangular, walls: coffee beans, entry-exit: coffee cups), we augment the final question- answering step by providing the VLM with the component mapping. as well as the full conversation history of the grounding process. In particular, the groundings of the components are included in text prompt such as: "Imagine that the image represents a maze, and the layout rectangular, and the walls are coffee beans, and the entry and exit are coffee cups." Compared with baselines that only have access the abstract concept itself (e.g., "Imagine that the image represents maze." "), our concept grounding system shows significant performance improvement with the resolved schema as holistic context for the image.
Notably, our DSG framework is inference-only: it leverages the strong generalization capabilities of LLMs (for proposing schemas) and VLMs (for grounding schemas and reasoning). to better understand visual abstractions. Programmatic schemas are used the intermediate representation to bridge the explicit commonsense knowledge in LLMs and the visual reasoning capability of VLMs.
VISUAL ABSTRACTIONS BENCHMARK
In order to evaluate models on visual abstraction reasoning, we propose a new benchmark, the Visual Abstractions Benchmark (VAB), illustrated with examples in Figure It consists of 180 images and 3 questions per image, with a total of 540 test examples. VAB is comprised of 12 different abstract concepts spanning categories, where each concept is associated with 15 examples of real-world images. The questions bridge multiple question types: binary-choice, counting, and open-ended questions. We detail the set of abstract concepts and the image and text components of VAB below.

---

Published as a conference paper at ICLR 2025
Strategic Concepts
Scientifie Concepts
Tic-tac-toe
Maze
Treasure map
Solar system
Atom
Cell
Social Concepts
Domestic Concepts
Helping
Deceiving
Negotiating
Table setting for two
Guest bedruoms
Door decoration
Figure 5: The Visual Abstractions Benchmark comprises diverse, real-world images that represent 12 different abstract concepts across 4 categories.
Abstract concepts. We choose 12 different visual abstractions as concepts. All of them can be grounded different objects and configurations of objects in images. We focus concepts that are not tied to distinet visual features (e.g., unlike canonical colors and shapes) but instead require higher-level, relational, and lifted patterns. These visual abstractions cover wide range of visual scenarios and possible linguistic queries, and they can be grouped into four broader categories.
The first category consists of strategic concepts, which games characterized by rules and patterns, including "tic-tac-toe", "maze", and "treasure map". The exact object instantiations and visual features may vary, but humans can easily identify the state of the game when given the concept. The second category consists of scientific concepts with no possible visualization in the physical world, including "solar system", "atom", and "cell". While we cannot directly view these concepts, we can create analogies of them based on everyday objects. The third category consists of social concepts, which are intentional actions that involve the deduction of roles and relations, including "helping". "deceiving". and "negotiating". Examples of these theory-of-mind concepts in the real world can be seen in many videos and movies; for a single image frame, we must make deductions to assign possible roles entities in the scene. The fourth category consists of domestic concepts, inspired by everyday household tasks (Li et al.) (023). These concepts are goals in household environments that cannot be easily defined by specific arrangements of objects, including "table setting for two". "guest bedrooms", and "door decoration".. While humans easily recognize when such objectives are completed, we do not have simple definition for what exact configuration should be generated.
Images and question-answers. We curate 15 images for each abstract concept from the Internet. Our criteria for the images are to be real, natural, and captured in the physical world. Our final set of images is diverse in terms of subjects, environments, and domains. See Figure for examples.
7

---

Published as a conference paper at ICLR 2025
Table I: DSG outperforms prior works across all questions and different question types in the Visual
Abstractions Benchmark. Results are averaged over 5 runs.
All
Counting
Binary
Open
ViperGPT (Surís et al. 2023)
0.260
0.191
0.426
0.141
VisProg Gupta & Kembhavi
2023
0.304
0.249
0.385
0.255
LLaVA (Liu et al. 2024)
0.415
0.344
0.552
0.321
InstructBLIP (Da et 2024
0.437
0.322
0.538
0.392
GPT-4o (OpenAl 2024
0.664
0.604
0.655
0.693
DSG (ours)
0.730
0.704
0.690
0.776
Human
0.846
0.816
0.833
0.868
Table 2: DSG improves the performance of GPT-4o across categories of abstract concepts.
Strategic
Scientific
Social
Domestic
GPT-4o (OpenAl DSG (ours)
2024)
0.653
+0.052
0.646 +0.074
0.647 +0.163
0.708 #0.082
0.751 +0.088
0.693 10.126
0.723 +0.027
0.754 10.104
We create 3 types of natural language questions for each abstract concept in VAB. These questions ask about different aspects of the images and take various forms, including binary-choice questions, counting questions, and general open-ended questions. In total, there are 6 types of counting questions, 14 types of binary-choice questions, and 16 types open-ended questions. An example of a binary-choice question is "Imagine that the image represents a table setting for two. What is the configuration of the seats? Describe the answer as across or adjacent."; an example of counting question is "Imagine that the image represents cell. Excluding the nucleus, how many types of organelles are there?": an example of an open-ended question is "Imagine that the image represents a treasure map. Based on the image, find the starting location; what is the main color of the map region that is physically closest to the starting location?" The full Visual Abstractions Benchmark comprises 540 of such questions.
Answers to all image-question pairs are labeled by human annotators from Prolific, we take their modal response as the answer, to be used for exact-match text accuracy. While the Visual Abstractions Benchmark focuses on multiple-choice questions and answers, we also provide a free-response variant of VAB. We present examples for each type of question along with corresponding images and answers in Appendix B and release our benchmark here
5 RESULTS
We validate Deep Schema Grounding in comparison to baseline VLMs and integrated LLMs with APIs on the Visual Abstractions Benchmark. We first compare our work with ViperGPT Suris et al. 2023). VisProg (Gupta & Kembhavi] (203)), LLaVA (Liu et al.) (2024), InstructBLIP (Dai et al. 2024]. and GPT-4o (I 2024). Then, we ablate different components of the DSG framework. Finally, we show that DSG is model-agnostic and can benefit both closed-source and open-source models. Bascline models are given the abstract concept itself in the prompt along with the question (c.g., "Imagine that the image represents a maze.") "), but without DSG's resolved schema of the concept. All results are averaged over 5 runs. Here, systematically evaluate multiple-choice answers, where accuracy is calculated as exact-match accuracy. In Appendix A we provide additional comparisons with graded accuracy metric based the degree of alignment to human judgments, and results free-form answers using BERTScores (Zhang el al.) 2020) and LLMs (Kamalloo et al. 2023)
VLMs benefit from decomposing abstract concepts into primitive-level components. Table ] summarizes the multiple-choice accuracy of DSG with GPT-4o as the base VLM, in comparison to that of prior works across 5 runs. DSG significantly outperforms VLMs and integrated LLMs with APIs across all questions and all types of questions. In particular, our method demonstrates a boost in accuracy of 6.6 percent points overall (t 9.9% relative improvement) compared to GPT-4o. Across different question types, DSG shows the largest performance improvement of 10 percent points (t 16.6% relative improvement) for counting- questions, such "How many entities
8

---

Published as a conference paper at ICLR 2025
Table 3: Summarization of DSG ablations; the full DSG framework shows strongest performance.
schema
grounding
hierarchy
context
All
Counting
Binary
Open
schema
X
X
0,681
0.629
0.672
0.709
+ grounding
X
0.693
0.727
0.660
0.710
hierarchy
/
/
0.701
0.713
0.671
0.723
DSG (ours)
/
0.730
0.704
0.690
0.776
helping?" We hypothesize that the significant improvement in accuracy for these questions is due to more accurate labels for objects with DSG, as the schema grounding step will concretely associate concepts such as "the entity who is helping" with particular parties in the image (e.g., pandas). These questions usually require a more sophisticated reasoning process to answer correctly. In comparison, for binary-choice questions, there may exist more superficial correlations that off-the-shelf VLMs can rely on. Importantly, we also report random human annotator's accuracy of adhering to the modal human response to highlight the difficulty of problems in VAB.
In Table 2 we more closely analyze the performance improvement of DSG on GPT-4o across concept categories. We report the average performance and the standard deviation for 5 runs. We see that DSG outperforms GPT-4o across the four categories. In particular, our method demonstrates a strong boost in accuracy of 9.8 percent points (t 15% relative improvement) for strategic concepts, and 7.6 percent points (f 11.7% relative improvement) for social concepts. In Appendix A we include qualitative examples of cases where DSG succeeds and fails in correctly grounding schemas. Overall, DSG's decomposition based framework improves performance of VLMs across abstract concepts.
Notably, we also analyze the performance of GPT-4v OpenAl (2023) and DSG with GPT-4v as the base model, compared to that of GPT-4o. We summarize results follows: GPT-4v yields overall accuracy 0.653, GPT-4o yields 0.664, DSG with GPT-4v yields 0.707, and DSG with GPT-4o yields 0.730. We see that DSG significantly improves the performance of both base models. Importantly, DSG with GPT-4v as the base model outperforms GPT-4o by 4.3 percent points, and showcases the potential for DSG improve the visual reasoning performance of weaker models.
Explicit hierarchical grounding of schemas better than implicit and sequential grounding. We present ablation studies of different modules in DSG in Table 3 In particular, evaluate (1) variant of DSG that is given only the LLM-generated schema for chain-of- chain-of-thought reasoning within a single query, without using explicit grounding steps (+ schema), (2) variant where DSG performs decomposed grounding sequentially instead of hierarchically (+ grounding), and (3) hierarchical variant (+ hierarchical) where in the final question-annswrring step, we provide the model with only the grounding result of each schema component, instead of a full history of its grounding process.
With the first variant, we test whether GPT-4o can use the schema appropriately without a structured thinking process, We explore the second variant to test whether decomposing the visual abstraction into set of key elements sufficient for improving the understanding of abstract concepts, and evaluate the role of hierarchical dependencies in lower-level concepts for grounding. Finally, we present results on the third variant test whether the full grounding trace of DSG is helpful during the final dnestion-answering step. The results show that all these design choices help: a schema only does not lend itself to much improved performance, and hierarchical grounding from concrete to abstract symbols, well as the inclusion of the full context. improves performance in many concepts.
DSG improves other (open-source) VLMs, too. Tables Fand summarrize the results of DSG with open-source LLaVA as the base VLM. DSG yields consistent improvement overall (1 7.0% relative improvement). However, in contrast to GPT-4o, both LLaVA and DSG yield poor results on counting questions, as the base VLM struggles at enumeration regardless of its abstraction understanding.
Table 4: DSG improves the performance of the open-source LLaVA model.
All
Counting
Binary
Open
LLaVA (Liu et al. 2024)
0.415
0.344
0.552
0.321
DSG (ours)
0.444
0.344
0.590
0.354

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of LLaVA and DSG across categories; DSG outperforms the base VLM.
Strategic
Scientific
Social
Domestic
LLaVA (Liu et al.
2024)
0.407 #0.038
0.333 =0.101
0.378 10.126
0.541 #0.091
DSG (ours)
0.430 +0.046
0.378 00.119
0.393 #0.154
0.578 +0.096
DISCUSSION
Our framework demonstrates that by explicitly grounding conceptual schemas through large pretrained vision and language models, we can achieve better visual abstraction understanding in diverse real- world instantiations. We find that LLMs generally possess outputting comprehensive schemas that are accurately decomposed into more concrete symbols and their dependencies. The extracted schemas effectively capture the concise essence of concept, such as the answer to our initial question of what makes maze look like a maze. However, as DSG relies on pre-trained LLMs generate schemas and does not restrict the schemas set of specified symbols, it is possible that the schemas may contain harmful biases based on the concept given, or may include symbols that are difficult for VLMs to interpret.
knowledge of concepts,
In particular, we observe that current VLMs struggle to ground challenging schema components involving spatial constraints. DSG does not explicitly improve this capability or specify how to parameterize spatial configurations. While DSG can generate schema components that contain spatial relationships, and try to ground them by describing them in language, it remains limited by the VLMs' abilities to conduct such spatial understanding. For example, let us consider extracted schema for the concept maze includes a symbol representing "an unobstructed path from entry to exit". That such path exists in the maze typically only one and often one that is difficult to find-is a defining feature of mazes, yet is challenging for VLMs to correctly identify. VLMs' lack of spatial understanding weakens DSG's ability interpret complex and spatially grounded components. Scaling DSG to better understand concepts that require precise layouts of schema components is difficult but crucial for future work.
where the
A key aspect of DSG is that our framework does not limit the expressiveness of schemas to specific set of symbols or the grounding process to a specific set of answers. The only prior we inject into DSG is the hierarchical process of resolving the schema-th the interface between the LLM and VLMs unconstrained language. In theory, DSG can propose and ground any concept and any individual component, allowing the LLM and VLM to determine what defines visual abstraction without any restrictions. But such flexibility may also render the schema ambiguous and not universal enough to improve downstream tasks. However, we see that the empirical evidence in our paper supports DSG, and we believe that DSG serves as a proof of concept for the kind of inductive biases that should be incorporated into visual reasoning systems a structured thinking process guided by flexible but explicit knowledge can augment visual abstraction interpretation.
While we have made progress understanding what makes maze look like maze, in proposing DSG as a framework for leveraging explicit schemas backed by powerful LLMS and VLMs, we are far from solving all reasoning across abstract concepts as humans do. There may be ways to improve the grounding of individual components, as well as more efficient priors that can be incorporated into the system. We leave such exploration to future work.
CONCLUSION
We propose Deep Schema Grounding as a promising approach to understanding visual abstractions. DSG leverages schemas of concepts to decompose abstract concepts into subcomponents and model their dependencies. Our framework extracts explicitly structured schemas from large language models and hierarchically grounds them to images with vision-language models. On the Visual A Abstractions benchmark composed of real-world images with diverse
Benchmark, a visual underlying abstract concepts, DSG demonstrates significant improvements compared to base VLMs. DSG is a step toward interpreting visual abstractions as humans do; more remains to be done the challenge of visual abstraction reasoning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Michael Li and Weiyu Liu for providing valuable feedback. This work is in part supported by the Stanford Institute for -ennan-eeeteeeddd Artificial Intelligence (HAI), ONR N00014-23- 1-001423--1-2555 1-2355, ONR YIP 400014-24-1-2117, NSF RI #221 1258, Air Force Office of Scientific Research (AFOSR) YIP FA9550-225 1-99550-23-1-0111 23-1-0127, the Center for Brain, Minds, and Machines (CBMM), the MIT Quest for Intelligence, MIT-IBM Watson AT Lab, and Analog Devices. JH also supported by the Knight Hennessy Scholarship and the NSF Graduate Research Fellowship.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015.
Irving Biederman. Recognition-by-Comppee Componetts: A Theory of Human Image Understanding. Psycho- logical Review, 1987.
Susan Carey. Conceptual Change in Childhood. MIT Press, 1985.
Susan Carey. The Origin of Concepts. Journal of Cognition and Development, 2000.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv:1504.000 00325, 2015.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and Improving Chain-of-thought Reasoning in Vision-Language Models. NAACL, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung. and Steven Hoi. InstructBLIP: Towards General-purpose Vision- language Models with Instruction Tuning. NeurIPS, 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aide aided Language Models. In ICML, 2023.
Alison Gopnik. Conceptual and Semantic Development as Theory Change: The Case of Object Permanence. Mind & Language, 1988.
Alison Gopnik and Andrew N Meltzoff. Words, Thoughts, and Theories. MIT Press, 1997.
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional Visual Reasoning without Training. In CVPR, 2023.
Joy Hsu, Jiayuan Mao, Josh Tenenbaum, and Jiajun Wu. What's Left? Concept Grounding with Logic-Ente Logic-Enhanced Foundation Models. NeurIPS, 2024.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to Reason: End-to-End Module Networks for Visual Question Answering. ICCV, 2017.
Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. Evaluating Open-domain Question Answering in the Era of Large Language Models. ACI., 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept Bottleneck Models. In ICML, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting Language and Vision using Crowdsourced Dense Image Annotations. IJCV, 2017.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- 1k: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In CoRL, 2023.
11

---

Published as a conference paper at ICLR 2025
Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models. NeurIPS, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, NeurIPS, 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. NeurlPS, 2022.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision. In ICLR, 2019.
Sachit Menon and Carl Vondrick. Visual Classification via Description from Large Language Models. ICLR, 2023.
Adam Morton. Frames of Mind: Constraints on the Common-sense Conception of the Mental. Oxford University Press, 1980.
OpenAl. ChatGPT Can Now See, Hear, and Speak.
https///openai.com/bbloe
chatgpt-can canott-can-aee now-see-eeeeoeooeeeee hear- and-speal: 2023.
OpenAl. Hello GPT-4o. httppr//pppaa..ccom/mmm////lblbp gpt- 2024.
Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Inferring the Why in Images. 1rXiv:1406.5472. 2, 2014.
Roger Schank and Robert P Abelson. Scripts, Plans, and Knowledge. In ICCAI, 1975.
Xiujie Song, Mengyue Wu, Kenny Q Zhu, Chunhao Zhang, and Yanyi Chen. A Cognitive Eval- uation Benchmark of Image Reasoning and Description for Large Vision Language Models. arXiv:2402. 18409, 2024.
5.5: An Open Multilingual Graph of Robyn Speer. Joshua Chin, and Catherine Havasi. ConceptNet 5.5: General Knowledge. In AAAI, 2017.
Didac Surís. Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In ICCV, 2023.
Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski. Mario Fritz, and Ales Leonardis. Answering Visual What-if Questions: From Actions to Predicted Scene Descriptions. In ECCV Workshops, 2018.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. FVQA: Fact-based Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of oh-thutght Prompting Elicits Reasoning in Large Language Models. NeurIPS. 2022.
Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources. CVPR, 2016.
Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara L Berg. Visual Madlibs: Fill in the Blank Image Generation and Question Answering. In ICCV, 2015.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A Unified Natural Language Framework for Algorithmic Reasoning. NeurIPS, 2023.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From Recognition to Cognition: Visual Commonsense Reasoning. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2025
Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Xixin Wu, Yoon Kim, Helen Meng, and James Glass. Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning. NAACL, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi BERTScore: Evaluating Text Generation with BERT. ICLR, 2020.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded Question Answering in Images. CVPR, 2016.
13

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY FOR: WHAT MAKES A MAZE LOOK LIKE A MAZE?
The appendix is organized as the following. In Appendix A we present comparisons of Deep Schema Grounding prior works with graded accuracy metric, comparisons on free response generation, results on single- v.s. multi multi-round reasoning, results with language-on baselines, ablations on schema complexity, qualitative examples of DSG, and examples of hierarchical grounding. In Appendix B we show more examples of the Visual Abstractions Benchmark, explain our benchmark construction procedure, describe our annotation collection process, and release our dataset. In Appendix ( we provide prompt to the LLM and all extracted schemas, as well as results from a human study evaluating the quality of the generated schemas.
A ADDITIONAL RESULTS
A.1 GRADED ACCURACY COMPARISON
Due to variation in human responses the Visual Abstractions Benchmark-arising from natural errors in annotation or potential ambiguity in questions-we compare Deep Schema Grounding to prior works with graded accuracy metric, based on degree of alignment to human judgements. Predicted answers each receive K points, where K is the number of human annotators out of 5 that agree with the prediction. In Table we report results with the graded accuracy metric over 5 runs. DSG consistently outperforms previous works, showing an overall accuracy increase of 6.2 percent points (f 10% relative improvement), and an accuracy boost in counting questions of 10.8 percent points (f 20.8% relative improvement). We observe a similar trend in performance as with modal response accuracy reported the main text-more improvement is seen counting questions.
Table 6: Comparison of DSG to prior works with a graded accuracy metric based on degree of alignment human judgements.
All
Counting
Binary
Open
ViperGPT (Surís al. 2023)
0.254
0.200
0.413
0.136
VisProg Gupta & Kembhavi
2023
0.293
0.227
0.380
0.242
LLaVA (Luu al. 2024
0.400
0.327
0.548
0.298
InstructBLIP (Dai et al.
2024
0.420
0.307
0.528
0.368
GPT-4o (OpenAl 2024
0.622
0.519
0.655
0.632
DSG (ours)
0.684
0.627
0.681
0.708
Human
0.789
0.728
0.784
0.816
A.2 COMPARISON ON FREE RESPONSE GENERATION
We present comparisons between DSG and GPT-4o on free response generation in Table 7 with accuracy first evaluated by BERTScores Zhang et al. (2020). We see that DSG outperforms GPT-4o across all questions and question types.
Table 7: Performance on free response generation; DSG outperforms the base VLM.
All
Counting
Binary
Open
GPT-44 (OpenAl 2024)
0.900
0.917
0.942
0.858
DSG (ours)
0.908
0.949
0.946
0.859
Additionally, we report free-response results by incorporating an L..M-based strategy for evaluating free-response answers (Kamalloo et al. 2023). In particular, we use zero-shot evaluation via LLM prompting, which includes the ground truth and predicted answer in the prompt, to verify the correctness of the prediction. We use GPT-4 as the base LLM, and compare GPT-4o with DSG. In T we see that DSG outperforms GPT-4o overall, with particularly strong results on counting and open-ended questions, and shows stronger signal than with BERTScores. However, performance binary questions is lower, which we hypothesize is due to some of the components being potentially misleading ambiguous in binary contexts.
14

---

Published as a conference paper at ICLR 2025
Table 8: Performance on free response generation with Kamalloo al. 2023) 's GPT-4-based metric for evaluating open-domain QA models.
All
Counting
Binary
Open
GPT-4o
36.67
45.56
55. 55.24
14.58
DSG (ours)
37.78
53.33
49.52
21.25
A.3 RESULTS ON SINGLE-ROUND Vs. MULTI-ROUND REASONING
The key idea behind DSG lies in schema decomposition (e.g., in the form of extracting components: layout, walls, and entry and exit) to guide the VLM to interpret the scene. DSG introduces an explicit guidance to the VLM system, by first generating decomposition of the target concept, parsing the image, then answering the question. The specific method of processing these questions-whethe in single query across multiple queries is not fundamentally important. While in theory, these steps could be merged into a single CoT-style prompt (Wei et al.) (2022), our following experiments show that multi-round execution with explicit grounding steps significantly improves performance in practice, at least for the current vision-language models.
Here, we compare results of (1) GPT-4o, (2) single-round CoT baseline given schema components (e.g. the VLM is given components for reasoning in a single query, without using the explicit grounding steps), (3) single-round CoT baseline given the full hierarchical schema (also in Table 3 and (4) our full multi-round DSG framework. All results are averaged over 5 runs. In Table D we see that DSG's multi-rount execution outperforms single-round. CoT-style baselines, likely due to its explicit guidance and grounding.
Table 9: Comparison of DSG with single- and multi-round grounding of the schema, with results averaged over 5 runs.
All
Counting
Binary
Open
GPT-4o
0.664
0.604
0.655
0.693
Single-round with schema components
0.667
0.607
0.661
0.696
Single-round with hierarchical schema
0.681
0.629
0.672
0.709
Multi-round with hierarchical schema (DSG)
0.730
0.704
0.690
0.776
A.4 LANGUAGE-ONLY BASELINES
We additionally report results on language baselines. The first is language-only baseline, where GPT-4o only has access to the question itself without input image or schema. The second is a language-with-schet
bascline, where GPT-4o has access to the question and the DSG grounded schema (the schema is extracted from the image and provides context), but the final query itself does not have access the image. We report results over 5 runs, and present results in Table TO
Table 10: Comparison with language-only baseline and baseline with DSG's grounded schema, with results averaged over 5 runs.
All
Counting
Binary
Open
Language-only Language w/ DSG schema
0.397
0.300
0.510
0.335
0.522
0.440
0.530
0.546
Notably, the language-only baseline performs worse than open-source and closed-source VLMss showing that the visual input is essential for achieving higher accuracy on the benchmark. We note that the language-only baseline outperforms integrated LLMs with API baselines, as these methods tend to fail in execution. With DSG's grounded schema, the language model performs significantly better, highlighting the importance of the holistic context provided by the grounded schema. We that in open-ended questions, the grounded schema significantly improves performance of the language-onie baseline. We hypothesize that this is because the schema helps rule out implausible answer choices by leveraging the grounded components.
15

---

Published as a conference paper at ICLR 2025
A.5 SCHEMA COMPLEXITY ABLATION
We highlight that there's 's tradeoff between encouraging more detailed components in a schema and avoiding both components that are less broadly applicable across image instantiations as well potential failures of current VL.Ms in handling such detailed concepts, especially more spatially-focused ones. capabilities of VL.s increase, we helieve DSG's performance will scale accordingly, and can increase the detail of the schema without making this tradcoff. In addition, the complexity of the generated schema currently influenced by the number of components the example prompt. As LLMs become less dependent the form of the initial prompt, they can decide the schema without being influenced by the bias the example.
To explore the impact of schema complexity and better understand the capabilitiies of current VLMs, we report ablations results varying the numbers of components in the example prompt: 3, 5, 7, and 9. The example prompts based on the abstract concept academia expanded to different levels of detail.
With 3 components::
gen -aaadepptaa
lfsculty concept-acacdmiaa)
gen Istudents concept
(reseerch-outgus concaptsacademia, faculty, studants)
With 5 components:
(concept-academiie gen (faculty concept)academia)
Istudanta conceptszcademial
gen Icourses concept-academla, faculty, students:
gen (cesearch output: concept-acedemia, faculty, students, courses)
(fund! ng
concept =academla a, research-output)
With 7 companents:
gen (eoncepttacadent gen (faculty concept=academia)
gen (students concept
gen idepariments concept=acadenis, faculty, students)
(courses concegt=academia, departments, faculty, students)
gen (research-outppe concoptsacademia,, departmente, faculty, students)
(funding concept-academta, research-oitput, department.s) çea lenctaauuuuuu concept aaademia,,,, funding, depaztments, courses]
With 9 components:
gon (concopt gen ty aacdept=) gen tatudents concept=scademia
gen Idepartments concept=acadenie, faculty, students)
icourses concept-academia, departments, faculty, students)
gen iresearch-ostpur- concapt=academla departments, faculty, studentss (Eunding conceptracademlaa, research-ojtputt, departments)
gen iinfrostructure concopt-acodemto, funding, departments, courses)
conceps=academia, infraatructure, departments, faculty, studentst gen toommunitt engagement concept-acedemia research output, administration, studeats, faculty)
gen ladminist
Table 11: Ablation with varying number of components in the example prompt.
All
Counting
Binary
Open
DSG w/ 3 comp.
0.730
0.689
0.700
0.771
DSG w/ comp.
0.691
0.622
0.671
0.733
DSG w/ 7 comp.
0.685
0.667
0.662
0.713
DSG w/ 9 comp.
0.676
0.700
0.657
0.683
In Table II the results demonstrate that accuracy decreases as the number of components in the example prompt increases, likely due to current VLMs struggling with grounding more specific components. We note that all DSG variants still outperform the GPT-4o baseline. Interestingly, our findings also highlight tendency for LLMs to generate shorter schemas, detailed example prompts. In Table [2 we show the number components in the prompts and the mean number of components in the resulting schemas, averaged over all abstract concepts.
when exposed to more
16

---

Published as a conference paper at ICLR 2025
Table 12: Number of components in schema and the resulting average number of components in the generated schemas.
Generated comp.
comp. 3.00
comp. 4.66
comp.
3.75
comp.
7.00
Notably, although the number of components in the example prompt does affect the average number of components in resulting schemas, this effect is less prominent and the resulting average plateaus. This phenomenon indicates that schemas may potentially converge to shorter lengths. And while current VLMs' limitations constrain schema complexity, these capabilities improve with development, expect DSG's performance to scale, enabling our framework to adopt more detailed schemas without sacrificing effectiveness.
A.6 QUALITATIVE EXAMPLES
We present examples of DSG's s resolved schemas in Figure 6 in particular, we highlight cases where DSG's schema grounding process errors in red.
repeesents the
imagine that repeesents map, rup is cookic, marka the spot is chocolate and the path leing dashed lines.
represents maze, layout deuble-spiral the walls are sand ridges. and the and extt onter ends spirats.
tape on and the sycahols are plates and cups, and the strategy is black
Imagine that the image reprevents solor sysem, and the sun orange slice. and planets fruits the yogurt
Imagine image represents stom, and the mucless is cluster of candles, and the electrons onengy are icing trails.
Imagine that the image repeceents well, and the membrane is an orange string, and the muclous organelles
and
materials arranged board.
Imagine that the image helper helping, and recipient actopus tay, and the tadi celebration setup.
Imsgine that image deceiving and and the vierim lege figure with helmet, and die deceptive object spear.
linagine that the negociating, perticipants eotting is home office, and objects are dolls.
Intgine image repeesents secting up sable for two, and the thlle a pienic table, and cushlons, and sotting pienic setup.
Imagine duat image repeesenes tilying up guest room, and the bed is neatty made bed, and stecage dresser nighistand, cleantiness is immoculate.
Imagine that the image repeesents purting decorasions on door, and the door classroom
TELE
type valentine's decerutions, and the sools are adiheshve cape tucks.
Figure 6: Examples of DSG's S resolved schemas across different abstract concepts; in red we highlight failure cases where inference of individual components errors.
A.7 HIERARCHICAL GROUNDING EXAMPLE
Figure we highlight an example of how DSG's hierarchical inference improves schema grounding accuracy compared to sequential version of DSG.
A.8 COMPUTING RESOURCES
We use OpenAl's API for GPT-4o (OpenAl 2024). ViperGPT (Suris et al. 2023). and VisProg (Gupta
& Kembhavi 2023). Open-sourced models, LLaVA (Liu al. 2024) and InstructBIIIP DDai et al.
17

---

Published as a conference paper at ICLR 2025
Hicrarchical Grounding
Sequential Grounding
Concept Atom
gen(nucleus (oocept=aatae)
gen(nucTeus concapt-atarr)
Q: "Imagine that the imuge represents an alom, what is the nucleus?"
Q: "IImagine that the image represents an alomm, what is the nucleus?"
Grapes
Grapes
gen(elec trons I conceptsatoo))
gen(elec trons I conceptiatoo)
Q: "Imagine the image represents an atom, the electrons?"
Q: "Imagine the image represents an atom, electrons?"
A: Blueberries
A: Blueberries
(eeeeenyreeeeellee concept-atos, nuclevs, electrons) Q: "Imagine that the image represents an atom, and the nueleus is grapea, and the clectrons are blueberiiss. what energy levels?"
gen(energy levels concept-ator)
Q: "Imagine that the image represents an atom, what are the energy levels?" A: Orbitals
A: Plate edges
goncccecppt)a)o))
gen(cancept-atte)
Resolved schema: Imagine image represents atom, and the nucleus is grapes, and the electrons are blueberries, and the energy levels are plate edges.
Resolved schema: Imagine image represents atom, and the nucleus is grapes, and the electrons are blueberries, and the energy levels are orbitals.
In comparison to DSG, which hierarchically grounds each component, sequential variant errors when grounding a more fine-grained symbol, energy-levels, without conditioning on prior predictions of more concrete symbols, nucleus and electrons.
Figure 7:
2024). and the API calls the aforementioned integrated LLMs with APIs, were run inference-only with A40 internal cluster.
18

---

Published as a conference paper at ICLR 2025
B VISUAL ABSTRACTIONS BENCHMARK
B.1 BENCHMARK EXAMPLES
In Figure g we provide visual question-answering examples of VAB across concept types and question types.
B.2 BENCHMARK CONSTRUCTION
We constructed the Visual Abstractions Benchmark inspired by areas where abstract concepts would occur in the real-world, including strategic games which can be played across instantiations, scientific concepts where many classroom science projects exist, social concepts with theory-of-mine examples, and domestic concepts with tasks that do not have predefined and exact configurations of end states. Images were manually curated from the Internet, primarily using Google Image Search. We searched for diverse real-world representations of each concept (e.g.. tic-tac-toe on the beach). Images were chosen such that each image is (a) faithful to the given concept, (b) diverse compared to other chosen images, (c) real-world, and (d) high-quality; this process is difficult to procedurally replicate. We then wrote general questions for each concept, aiming for broad applicability and alignment with the images. These questions focus on reasoning grounded in the abstract concept rather than specific visual details of individual images. Once images and questions were curated, they were sent to human annotators via Prolific for labeling.
B.3 ANNOTATION
We collect annotations for the Visual Abstractions Benchmark through Prolific 5 human annotators labeled each question-image pair. Figure shows our annotation interface, built from jsPsych. Each annotator was compensated the rate $12 USD per hour, and no potential risk was incurred by the annotators.
yes
nn
Imagine that the image represents solar system. Is there an item that represents the sun?
Figure Example of our annotation interface shown to Prolific annotators.
B.4 RELEASE
We release the Visual Abstractions Benchmark here, We group all images by their corresponding abstract concepts. Each image is carefully curated and manually audited be safe for release.
19

---

Published as a conference paper at ICLR 2025
imagies But reprcsess te ti the What an the eomiii the the toueu Answer unc fillowing sener dasies cheellee chocollto and cendlasiioee moderr arakknnl one.
inagges fur repesenta tie ti see What kann has rease symius playdd es the knawer fellowing clokes excender, krace, tersto.
lragitz thr
resserra tictic What van
leadin the pasat sonver xit falloning
boop,
yellow fisker, uncataan
Chocolac and salnnkeeedhe
Bazato
A. Yellow fialez
(A lmnainn Bua the image roncactt rasc ure the followizs chiloo. liko, sumre stit.
@ lnagin tua the iraae rencectt reac. lae thee playe Aurrac with enc the folewing sniver ciitscs bull heat, king, auncc
@ hragine last the imaac reasectle mase,
Aupocr ancul the killwingg zain chulsoo you the
Ves
@ lanainn fins the image rarcucto lomsere
lrogles fort repeosets mocoue map Based irage. the surting the
Ot hragine last the issae remosstl traase mag that roll Aenn wils of (ollering snovac chiccte you
oncef the fallaity snoee dascc back line oed lias aikkes, betilla marking
plysinally clossd roth crring bncriiee Reum nen if thas so regises Atarne wilhoae the kelowingg ang Mac, buck white eil Red
O hragine tua the isage eepocetts sysies fiom
heagine tua the innge reproctts spsien
@ hragiie that le innge
reprossto ssake seikes
(elloiing the wil system hnsver with oea ofthe inloviin snover
Oo wain of the lken plysically deset to Eatt and also repcoena a planti Aaawar choicca nod yallow. parple brows, and. frows
Arewe with anz the following snsver cholcoo you,
cholaor you
@ hragin Oua the isage rensoeate shon, Whel sysibul with the following sniver disico biue leal, coall, goll beal plae clanes pupple
hragin that the iange repsoctl sn skan. reareel Ansvet with of be (elloving snavece chiccee grea bell cenage venrge ball.
@ langiis that lee inmae reprovate un skens, syshel
sooll?
Anae will uncel the folloning enever chockos akanisss Gell. ceeageveraal oandyy crigs,
poee
GiMd head
Orange ball and you ball Incagina
4: Rice crigy
Imogies
call Whrt cymbel experoane the maderat knnwar with ane the dillniing aunner panball lonrice watemader
call lattr the oxcor of the call menhharss Ancrer with one the Silkwinng sellow, lie tonoueras.
rysoorear Eschading excinas how many ty organel are thene If following choics fieee, for, five, lin. conecces..
O
Aoocudo
Pik
d Fire
Imagas tue the image sane bemn achirvdd the achion will is proprou Ansver with one the fsllentta snover scloeed progres.
Incanne tha image help with doing sometingg help with aopping soewthing bven hamperang! Acaver with onc the fillowdng chocces doing copplng
( leapine thar the tmuge
eotitic halpaag theme roons rstarn zamecoas Aarwer with cfitts foliening ansver cameress.
proptoe
@t lmagee tua the iinge renvactt dectiving, What
(t lmnggin that the inaa: rezsoaeta decaiving Dous the
@i Iinauin that the tmags reprovates deadiving ing. eanry
with onz the follwning sziver disicol has caok, lacil, poke: drgy, encortie
have doulea about the doccooes? Asavvrr with cnc the fillowius assoer choices you, on, recomain.
Aaane with of tac follondia
ansovn choices ent, ron.
A: Namomes
lmagan that the image
langne that the inma: repsoutb nsperniigg Doo as caily scas lase axpet land le fulbnnng dhokkne you so.
hangin that dee inmas
reaecacte repoiiatina Whe being ixpeiiaed of
reprovato isactaing, b bedanaa adde regotaiio with reato lilkning snanic clackse uxectaas.
scpaniaed Auancr wich-me the
following snover cheices: feed. lesse. shacp, senc.
Shop
No
At Yos
lmagin that the inage repocccto adiin up abii fer tea
lragine that innae reasocce octiin up able tea. Bacol imago you placol for penie?? Anevec with one the fillowing sniver chuic end two, fox, fire,
lanpii that the inma:
repsovato sctting linngn aill for two. sonai
Basolles imnas cus you
die configaation of the
for cad persed Atswer wilkus of felloiing ceiver chickes sea two, these.
folkering snnvec claccee
edpoctt.
Tiee
Tio
Acroes
Iragee
beagee
ioags
tidling
liyiin
repsoveta rdying
Blaod os ta image you
Arc the linas (sdec? A Aarmer vit onz the tnloncin ansverr
Acc there stll pencruil iscratte Uke tocronn? Acavccr with
prodec bow vaary ailooware
you,
the folowing ansver disica ons two, devc. dox, fire, die, sncerois
Tive
No
A: Yos
Iraglee decar
Iragine image thes cocoeniine lis contrr of the deet de diussly vanwanding the doat? snmeer chiiko samanding Sermunding
TAKE
O leagine image
pusting
dess Wastin the thane the docorniant Anwe with one kilowitg savis dasicds
the daseniim
with compence spaafie the doer? Asovet with ous vi
pou
unuatais.
Figure 9; Examples of visual question-answering pairs in the Visual Abstractions Benchmark.
20

---

Published as a conference paper at ICLR 2025
C
LLM-GENERATED SCHEMAS
C.I
PROMPT
Below, we provide the prompt used as input GPT-4 extract schemas for visual abstractions; substitute labstract-conceppl for a given concept.
you give progrem representing
concept? example,
gen cconceptsacademis
gen (Eacuity | concept-academia)
istudents conpeptrecademial
gen iresearch-autpur concopt= faculty, studanta))
(concept-lasstratt connoeett--
format without explanation.
Keep the program simple with four or less components. Use enly the most necessazy parts that napped objects image.
C.2 ALL SCHEMAS
Here, we present all LLM-generated schemas for the Visual Abstractions Benchmark.
Strategic concepts.
gen (concept -tie tac Loe)
(board concept-tic tac--ttc--ee ace)
gen (syniolss
concept
rategy
concope-tti-ttoo-to,, symbols)
gen (concept-maze)
gen (layout concept-maze)
gen (walls concept-maze]
lentry-exxit concept layout)
gen Imap concept=treseure map) gen
spot sencept=treasuze map)
(path corcop::trraauur-nnp map]
Scientific concepts.
gen (eoncept-eollr--yyttell
concept-solar system)
gen (planets concept
torbits conoopt-oolar-ssyatmm sun, planets)
gen (concepL-aLom!
Insclaus | concept
gen felectrons concept-atoom
|
wewleeerr-leeeeeeee concept=atom, mucleus, electrens)
(concept=cell)
gen Imenorane concept=cell)
gen concept-cell)
torganelleee concept=cell)
Social concepts.
gen (concept hhencept=eelllniie pig)
thelper concept-helping)
gen trecipient concep:-helping)
ttaak poniep:=heiping helper, recipient)
gen (concegt Idece! ver concept= gen Ivictim concept=deceiving
(deceptiie object cencept-deceiving, deceiver, victim)
(concapt-neggoiittiig
gen lparticipants concept=negotiattie (atin)) gen (setting into participants)
jebts
21

---

Published as a conference paper at ICLR 2025
Domestic concepts.
gen (concepl-aettingg up Lable for Lwol
gen (table
tLwo chairs concepi-seiLing up Lable for Lwo)
gen seateeg concege-setting-ug sepettiinn-----eeeee up-table-for-tx0 teble)
gen (ngpapptttdd/l/g-----geeeee
gen concegt-Lidying quest-room.
gealstorage concept=:idying guest room)
gemicleanliness sancepp:tadyynnn-u--eeeee
bed, sterage)
gen ldacoration-typa http:ppppppppttoo--eeoooooooooo.
gen 1door-type
peputting-updeeeee lons-on-door
gen concept=puiting up decoralions door, decoration Lype)
C.3 HUMAN STUDY OF SCHEMA QUALITY
To evaluate the correctness and quality of the generated schemas, we conduct a human study using Prolific, with 20 participants. Each participant was asked 3 questions about each schema. The questions are as follow for an example concept maze:
Hlere you are given a concept, maze, and the coze components underlying it: Layout, walls, well these components represent this concept scale of 1-77 Hare you ara givan a concept, maze, and the core companenta undarlying it: layout, walls, 1isted accuratelly represent part this
entry-exit.
entry-exit, Hоw many components concept?
Hare you are given concept, maze, and care comporenta undarlying layout, wa 18, mutzy-exity Bow more components, if any, would you add to acouralely represent this concept?
We collect 3 responses for each of 12 concepts with 20 responses each, and yield a total of 720 human annotated answers. Table 13 see that (a) participants rated schemas as reasonably representative, (b) 86.3% of the components were considered accurate by participants, and (c) participants suggested adding 1.6 components on average per schema. The concept that ranks highest for the first question is the concept cell with rating of 6.15, and lowest as the concept negotiating with 4.50. The concept solar-system had the highest number of components to keep at 95%, while the concept negotiating is the lowest again at 61.7%. The concept tic-tac-tor was the most complete, with average of concept that participants might add, while the concept solar-systen had an average of 2.65. Though there is still room for improvement, the schemas generated by the LLM are largely accurate and aligned with human expectations. Importantly, none of the schemas were judged as fully invalid, showing the robustness of DSG's approach.
Table 13: Prolific study on quality of generated schemas, with results averaged over 20 participants.
Representatiivenss (1-7) # comp. to keep
# comp. to add
Participant responses
5.74
2.59
1.59
22


Reviews:
[{"review_id": "yPHTPoY1Va", "reviewer": "Reviewer_yKhN", "summary": "The paper introduces a framework called Deep Schema Grounding (DSG) and a VQA benchmark for evaluating abstract visual reasoning of VLMs. For a given image and a question about the concept in the image, DSG first generates a schema of the concept using LLM. Then it grounds the components of the schema onto images using VLMs. Once the schema is grounded, the grounding information is given as context in text from to the VML to answer the question about the image. They show that giving grounded schema context to the VLMs improves their ability to correctly answer questions about abstract concepts in the image.", "questions": "- Many recent works (https://arxiv.org/pdf/2305.10355, https://arxiv.org/pdf/2401.06209, others) have shown that VLMs  are ‘blind’ and don’t actually look at the visual information in images. VLMs mostly rely on language and questions in VQA benchmarks can be answered by having a good language prior. I was wondering how much of the questions in the proposed benchmark can be correctly answered by “language” only? It would be great to show a “language-only” baseline for the proposed VQA benchmark. Especially after adding grounded schema as context- it can show the impact of providing grounded schema too.\n\n- A qualitative analysis of how/why is the grounded schema is improving VQA capabilities of the VLM would be great. Especially for the cases where the question does not ask anything about the concepts of the schema (like the example of “What is the player in the maze?” mentioned in the weakness section)\n\n- Are the generated schemas verified by some expert? How are we sure that the schemas provided by the LLMs are correct for the provided concepts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Innovative use of pre-trained LLMs and VLMs to give additional context (in terms of grounded schemas) for VQA. In principle, this additional information should help answer pre-trained VLMs to better answer the questions related to the abstract concepts in the image. (However I think that the generated schemas shown in the appendix of the paper are not detailed enough to achieve this - please check weakness section)\n\n- Hierarchical Grounding: I like how they have used a hierarchical method of grounding components of the schemas. Grounding independent concepts first and then the dependent concepts of the schema should be the correct way for grounding which is used in the paper.", "weaknesses": "- Schemas are not detailed enough to give information about the concept. For example, tic-tac-toe schema include {board, symbols, strategy}. Although a tic-tac-toe game has  {board, symbols, strategy} it is not complete. Many board games have {board, symbols, strategy}. This schema does not tell that the board is a 3x3 grid. Similarly, for “negotiating” the schema is {participants, setting, object}. This schema could be of many different settings. (p.s. I do understand that the capabilities of current VLMs are not enough to handle detailed concepts)\n\n- It is shown that giving grounded schema as additional context to VLM to answer VQA question is improving the performance. But it is not shown or discussed why?/how? is the additional context improving the performance. For instance, in the example given in figure 1 of the paper where the question asks “What is the player in the maze?”, how is giving information about the layout, walls and entry-exit helping a VLM answer question about the ‘player’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6kCF9zDAta", "reviewer": "Reviewer_P1iE", "summary": "The paper proposes Deep Schema Grounding (DSG), a method to break down a visual concept into smaller concepts with dependencies on other concepts. The authors show the capability of DSG in solving complex visual question answering task. A visual abstraction benchmark is proposed with 12 abstract concepts and 180 images.", "questions": "Could you try something like chain of thought prompting as I mentioned above?\n\nIs there any way to generalize the idea to create a more general benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides a good solution for VLMs to better understand abstract concepts in an image. The presentation of the paper is clear with many figures for demonstration. The experiments are comprehensive, making the main message convincing. The benchmark created is novel and interesting.", "weaknesses": "In terms of the idea behind DSG, it seems to be close to chain of thought with specific instructions. For example, the maze example in the paper can be integrated with just one prompt: \"Imagine that the image represents a maze. <the question> Think step by step by recognizing the layout of the maze, the walls of the maze, then the entry and exit of the maze one by one.\" Maybe gpt-4o can automatically do this even without the instructions. This is probably why in Table 7 if the generation is free form, DSG does not outperform gpt-4o much despite using more API calls.\n\nTherefore, I am a bit concerned whether these types of schemas are necessary. After all, the dependency graph is generated by gpt-4, so at least gpt-4o should know how to do it if prompted well. But I agree that for smaller models, DSG can be very useful.\n\nAnother concern I have is about the diversity of the benchmark because the number of categories is quite limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lDB0DZW8s", "reviewer": "Reviewer_nf1E", "summary": "This paper introduces a VQA benchmark and method for visual abstract reasoning.\n\nThe \"Visual Abstractions Benchmark\" consists of 180 images, annotated with 3 questions each (and each question has 5 answer annotations from Prolific workers). These questions ask about abstract concepts in the image: for example, \"Q: Imagine that the image represents a maze. Are there clear entrances or exits in the maze? [Yes or No]\" (Figure 1; an image of an ice skating rink partitioned like a maze).\n\nThe method, \"Deep Schema Grounding (DSG)\", is a prompting framework that (1) uses an LLM to decompose an abstract concept into a schema, (2) uses a VLM to ground the \"primitive\" concepts from the schema to concrete concepts in the image, and (3) uses a VLM to answer the question given the schema and grounding.\n1. The schema is a DAG: a \"maze\" would decompose into \"layout\" and \"walls\", and \"layout\" further decomposes into \"entry-exit\".\n2. A VQA question is used for grounding: e.g. given the image, \"..., what is the layout?\". This graph is hierarchically grounded: e.g. \"entry-exit\" is grounded using the question \"..., the layout is [], what is the entry and exit?\". The ice rink example above entails the following grounding: \"{layout: grid-like, walls: ice, entry-exit: gates}\".\n3. The image and grounded schema are joined as a VQA question for the VLM to produce a final answer.\n\nApplying this framework improves performance of GPT-4o by 10% (relative) on the benchmark.", "questions": "The word \"lifted\" is used in several places (e.g. \"lifted symbols\" or \"lifted rules\"). I think this terminology isn't entirely obvious. If this has a specific meaning, could the authors please elaborate in text (or just use a different word if not)?\n\nIt could be informative to clarify the distribution of schemas based on their graph depth in the text. E.g. I see that \"cell\" has depth 2 and \"atom\" has depth 3 (L875-881). The single in-context prompt for schema generation is \"academia\" (depth=3; Sec. C.1). Could abstract concepts exist with depths larger than 3? Would the LLM limit their decompositions to depth=3 (at most) because that is the only prompt? I think the authors could elaborate further on their prompt engineering and exploration, especially since their method is a prompting framework.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is extremely clear and straightforward.\n\nIn general, I believe the comparisons are fair and sufficient ablations are provided, so it is methodologically sound. For example, they show a 10% relative improvement when applying their method with GPT-4o and 7% with (the open-source and weaker) LLaVA model. They also compare against other methods (ViperGPT and VisProg) that also use GPT models (GPT-3) for visual question decomposition, but don't perform nearly as well on this benchmark. Finally, they provide sufficient ablations of their own method (separating the schema/grounding/hierarchy/context components and showing incremental gains of each).\n\nI believe this is a creative benchmark and that existing vision–language benchmarks under-explore abstract visual concepts. The method is task-specific, but I think that is fine, because it is prompting-based. I don't think the approach is entirely original but I think it sufficiently rounds out the paper: they show how explicit structure improves performance of existing models on their benchmark.\n\nAltogether: this paper presents a benchmark, shows that existing vision–language models underperform, and introduces an explicit prompting method that can be applied with any LLM/VLM to improve performance. The best performance with this method is still just 73% (with the strongest, closed models) or 44% (with open-source models), leaving sufficient challenge for future work.", "weaknesses": "I believe the construction of the dataset is under-specified (and I also checked the appendix). I understand that answer annotations were provided by crowd workers. However, how were the questions written? How were the images selected (and where do they come from: L376 just says \"the Internet\")? Likewise for the 12 abstract concepts and their 4 categories. Right now, I am imagining that the authors curated these themselves. That could be fine, but I would like to know more details, so that we can determine whether there may be any biases: e.g. does the introduced method perform better on this particular selection of abstract concepts?\n\nI notice that answers in the open-ended setting are evaluated using BERTScore. However, (Kamalloo 2023) says\n\n> unsupervised semantic similarity-based evaluation metrics such as BERTScore (Zhang et al., 2020) that rely on token-level matching of contextualized representations exhibit poor correlation with human judgment in QA evaluation (Chen et al., 2019) and lack the ability to capture attributability (Maynez et al., 2020)\n\nI also find this to be true in general. Could you show examples, so we can see if the metric is indeed suitable for this distribution of data? Otherwise, could you consider using a method that is better aligned with human judgments for this evaluation: possibly (Kamalloo 2023)?\n\nReference: [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://aclanthology.org/2023.acl-long.307) (Kamalloo 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9MNsy3tQW3", "reviewer": "Reviewer_XbBJ", "summary": "The paper proposes the Deep Schema Grounding (DSG) framework, aiming to enhance VLM’s capability in understanding abstract visual concepts like “the maze”. The framework integrates schema representations with hierarchical grounding to guarantee the VLM’s to better capture human-aligned visual abstractions. DSG operates by extracting schemas of abstract concepts, then grounding schemas on images following a hierarchical manner, then augmenting the VQA problem with resolved schemas, resulting in improved visual reasoning. The authors also introduce the Visual Abstractions Benchmark (VAB) to systematically assess abstract concept reasoning. Experimental results indicate that DSG consistently outperforms baseline models across diverse abstract concept categories and question types.", "questions": "1) Is the proposed DSG method applicable to other general visual reasoning benchmarks, such as RAVEN, Bongard, CVR, and SVRT?\n2) Can DSG apply to the scenario where there are multiple abstract concepts in an image? I expect the authors to provide some insight for this, but I don't expect the authors to scale up to more experiments in the rebuttal though.\n3) In table 3, the performance of schema plus grounding surpluses the performance of DSG in counting problem. Why would this happen?\n4) I also have some questions regarding the length control of the schema. In the prompt provided in the appendix, it is limited to 'Keep the program simple with four or fewer components.' Why is this limitation imposed? Can the authors provide experimental results for schemas with different numbers of components? Or should the depth of the schema increase for more abstract concepts, while simpler schemas can suffice for more general concepts?\n5) How robust is the method in general to failures: How often does the LLMs fail to provide a valid schema and how often does VLMs fail to provide a correct grounding? Does an invalid schema or grounding result in the failure when generate the answer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-written, it effectively conveys its proposed framwork and the motivation behinds it with clarity.\n2) The experiments are well-made and the experimental analysis is comprehensive. Extensive experiments validate the effectiveness and efficiency of the DSG, which serves as a plug-and-play trick to enhance the performance of existing VLMs.\n3) The pipeline that utilize schema representations with hierarchical grounding for visual concept understanding seems reasonable to the reviewer, as it emulates the human cognitive process of understanding abstract concepts.\n4) The proposed method is simple yet effective, and it is easy to follow.", "weaknesses": "1) The gerealibility is somehow resricted. The method has been tested on only 12 types of abstract concepts and it has not been tested on the problems which invovles multiple abstract concept, which raises questions about its generalizability to a broader range of abstractions.\n2) The techinical contribution of DSG is limited. The main technical contribution of this paper lies in the combination of neural-symbolic and VLM by the development of specifically designed prompts. (Prompt for LLM to generate schema, prompt for VLM to generate concept grounding). The improvements made to the prompts are relatively straightforward, and the utilization of CoT is a standard practice.\n3) The risk of the hallucination of LLM/VLM. During schema construction and hierarchical prompting, the LLM may produce hallucinated errors. If errors occur in hierarchical content, individual instances may fail; however, if schema generation is flawed, it could lead to failures across an entire category of problems. The paper lacks a thorough analysis of failure cases. The authors provide some failure cases demonstration in Appendix A.3. But it seems that these results are not enough. See question 5) for further suggestion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6AWjImkMO", "reviewer": "Reviewer_8h5R", "summary": "This paper introduces Deep Schema Grounding (DSG), a framework that extracts structured representations from LLMs to grounding and reasoning over visual abstractions. DSG leverages the inner knowledge of LLMs to generate schema descripting the abstraction concept. The content within abstract schema will be later grounding into corresponding compoents in the image by Large Vision Model via a hierarchically process. To evaluate the effectiveness of DSG, a new benchmark, Visual Abstraction Benchmark (VAB), is purposed in paper to evaluate model's understanding in visual abstractions.  VAB consists of 540 test examples consists of 180 images presenting 12 abstraction concepts spanning 4 categoris (15 examples each).  \n\nThe paper do comprehenisve experiment comparing DSG with 5 current VLMs. The result shows DSG can boost 6.6 percent accuracy oveall compared to GPT-4o while current VLMs shows a big gap between human performance. Further abalation study pinpoint that each modules in DSG is necessary and contribute to its success. DSG can boost model performance on both open-source and close-source model but authors notes the challenge still exist in visual abstraction field.", "questions": "Overall the limited size of benchmark is my primary concern. I like how paper design the framework and benchmark. The benchmark provide a  good practice on handle and evaluate abstract concept, which is helpful in current literature.\n\nQuestions: \n1. In llnes 399 to 400, it seems that there are multitypes of question and also each question type has different baseline accuracy (0.5 for binary in ideal). Do each categorie has a same distribution of different questions types? Otherwise the comparison over each category might be misleading. \n2. Is it possible the schema generated from LLMs doesn't fit well with image? For example, the first maze image (candy maze) in Figure 5, it is unclear for human what is the layout and where is the entry point.  How does the framework handle this scenario where the generated schema bring additional concept and may cause hallucinations.\n\n\n\nSuggestion: \n1. To make reader understanding the significance of abstraction, author can enrich the introduction with some practical application of abstraction elaborate more on how human use abstraction, such abstraction enable human mind to learning from small, obtain different levels of abstraction which can later be used to trim the learning space for new concepts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Schema-based framework: The paper start from cognitive science and theory-theory to purpose a schema-based concept representation framework. The process of representing high-level abstract concept and later grounding in actual image bring boost in model's ability of abstract understanding. The pre-generetad abstract concept schema seems helpful on mitigate halluciations (16.6% relative improvement in counting-based questions) and is beneficial for both open and closed source models.\n2. Curately designed Visual Abstraction Benchmark (VAB): The author provide detail explanation the reason behind VAR design choice. Detailed description of data collection and curation process is provided. The dataset is released for public use. As abstraction can play important role in understanding concept and ease the learning process, I believe this benchmark is a good step for solving the abstraction problem.\n3. A comprehensive experiment: the experiment extensively test current LVMs and point out their weakness in visual abstraction understanding.  Further experiment demonstrate the effectiveness of DSG and the ablation studies prove the design choice of each module. \n4. Writing. The paper is well-written overall, with clear and well-designed figures and tables. The comprehensive appendix offers additional details beyond those covered in the main text. I like the way authors start their paper with an illustration on ''what makes a maze look like a maze'', which provide a easy-understanding approach for readers, especially for the concept of visual abstraction.", "weaknesses": "1. Size of Benchmark: one concern I have for the benchmark is relatively small size of the data. VAB only comprised 12 abstract concepts acroos 4 categoires, I know the data collection can be challenging but the limited size can decrease the significance of the benchmark evaluation. The author mentions the benchmark is divertse in terms of subject, environments and domains (Line 377). A dataset statistic over these metadata can be really helpful to support the argument. \n2. Error grounding: Perception is also a challenge issue in the whole framework. It is possible that VLM have the understanding of the abstraction concept but fails to percept the image well and ground the concept within actual images (especially for images containing multiple subjects, image of cell in Figure 3).  That may also be the reasoning the schema-based framework become most helpful on couting problem. A further experiment showing VLMs can percept the image correctly is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)