What Makes a Maze Look Like a Maze?

## Abstract

Abstract A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them. While off-the-shelf vision-language models excel at making literal interpretations of images (e.g., recognizing object categories such as tree branches), they still struggle to make sense of such visual abstractions (e.g., how an arrangement of tree branches may form the walls of a maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas —dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract components of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Dataset, which consists of diverse, real-world images of abstract concepts and corresponding question-answer pairs labeled by humans. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models, and is a step toward human-aligned understanding of visual abstractions.

## 1 Introduction

Figure: Figure 1: There exist abstract concepts, such as “maze”, which are defined by lifted symbols and patterns, instead of concrete visual features. We propose Deep Schema Grounding (DSG), a framework for visual reasoning over such abstract concepts, which uses schemas to structure models’ interpretation of images. DSG hierarchically grounds conceptual schemas on images and uses them to provide holistic context to VLMs, improving performance across diverse downstream queries.
Refer to caption: x1.png

Humans possess the remarkable ability to flexibly acquire and apply abstract concepts when interpreting the concrete world around us. Consider the concept “maze”: our mental model can interpret mazes constructed with conventional materials (e.g., drawn lines) or unconventional ones (e.g., icing), and reason about mazes across a wide range of configurations and environments (e.g., in a cardboard box or on a knitted square). Our goal is to build systems that can make such flexible and broad generalizations as humans do. This necessitates a reconsideration of a fundamental question: what makes a maze look like a maze? A maze is not defined by concrete visual features such as the specific material of walls or its perpendicular intersections, but by lifted rules over symbols—a plausible model for a maze includes its layout, the materials composing the walls, and the designated entry and exit. Crucially, these model components can be realized by infinitely many real-world variations. For instance, walls constructed from candy canes, hay, or popsicle sticks, despite their diverse materials, are all identifiable as parts of a maze.

However, current vision-language models (VLMs) often struggle to reason about visual abstractions at a human level, frequently defaulting to literal interpretations of images, such as a collection of object categories. These interpretations may be technically correct descriptions of the scene, but may not align with the abstract concept underlying the image. For example, when given the intended concept of “maze” and an image that realizes the concept with unconventional objects, VLMs such as GPT-4V (OpenAI, ) and LLaVA (Liu et al., ) fail to correctly recognize objects in the scene as components of the visual abstraction. It is unclear whether these off-the-shelf models leverage knowledge of abstract concepts to make sense of images as humans do.

To address the challenge of understanding abstract visual concepts, we propose Deep Schema Grounding (DSG), a framework for models to interpret visual abstractions. At the core of DSG are schemas—a dependency graph description of abstract concepts (see Figure 1). Schemas characterize common patterns that humans use to interpret and reason about the visual world (Schank & Abelson, ). A schema for “helping” allows us to understand relations between characters in a finger puppet scene, while a schema for “tic-tac-toe” allows us to play the game even when the grid is composed of hula hoops instead of drawn lines. A schema for “maze” is what makes a maze look like a maze.

DSG explicitly uses schemas generated by and grounded by large pretrained models to reason about visual abstractions. Concretely, we model schemas as programs encoding directed acyclic graphs (DAGs), which decompose an abstract concept into a set of more concrete visual concepts as subcomponents, as illustrated in Figure 1. The full framework is composed of three steps. First, we extract schema definitions of abstract concepts from a large language model (LLM). Next, DSG hierarchically queries a vision-language model (VLM), first grounding concrete symbols in the DAG (i.e. symbols that do not depend on the interpretation of other symbols), then using those symbols as conditions to ground more abstract symbols. Finally, we use the resolved schema, including the grounding of all its components, as an additional input into a vision-language model to provide holistic context about the image, such that we can improve visual reasoning across diverse downstream queries about the abstract concept.
Our method is a general framework for abstract concepts that does not depend on specific models; the LLMs and VLMs used are interchangeable.

To investigate the capabilities of models in understanding visual abstractions, we introduce the Visual Abstractions Dataset (VAD). VAD is a visual question-answering dataset that consists of diverse, real-world images representing abstract concepts. The abstract concepts span $4$ different categories: strategic concepts that are characterized by rules and patterns (e.g., “tic-tac-toe”), scientific concepts of phenomena that cannot be visualized in their canonical forms (e.g., “cell”), social concepts that are defined by theory-of-mind relations (e.g., “deceiving”), and domestic concepts of household objectives that cannot be directly defined by specific arrangements of objects (e.g, “table setting for two”). Each image is an instantiation of an abstract concept, and is paired with questions that probe understanding of the visual abstraction; for example, “Imagine that the image represents a maze. Are there clear entrances or exits in the maze?” The Visual Abstractions Dataset comprises $540$ of such examples, with answers labeled by $5$ human annotators from Prolific. We found that off-the-shelf VLMs and integrated LLMs with APIs have significant room for improvement on this benchmark.

We evaluate Deep Schema Grounding on the Visual Abstractions Dataset, and show that DSG consistently improves performance of vision-language models across question types, abstract concept categories, and base models. Notably, DSG improves GPT-4V by $5.4$ percent point overall ($\uparrow 8.3\%$ relative improvement), and, in particular, demonstrates a $6.7$ percent point improvement ($\uparrow 11.0\%$ relative improvement) in questions that involve counting. While the challenge of visual abstraction understanding is still far from being fully solved, our results show that DSG is a promising solution that effectively schemas to structure the thinking process of visual reasoning systems.

In summary, our contributions are the following:

- •
We propose Deep Schema Grounding (DSG), a hierarchical, decomposition-based framework that explicitly extracts and grounds schemas of concepts with large pretrained models.
- •
We introduce the Visual Abstractions Dataset (VAD), a diverse, real-world visual question-answering benchmark that evaluates VLMs’ understanding of visual abstractions.
- •
We demonstrate DSG’s improvement over prior state-of-the-art works across a variety of abstract concept categories, question types, response types, and metrics.

## 2 Related Works

##### Vision-language grounding and reasoning benchmarks.

Visual-language benchmarks have traditionally tackled reasoning about content in images, such as objects, relations, and actions (Chen et al., ; Yu et al., ; Antol et al., ; Wu et al., ; Zhu et al., ; Krishna et al., ; Wang et al., ). Recently, an increasing number of works have focused on evaluating visual-language tasks that require commonsense reasoning, usually by asking “what if” or “why” questions (Pirsiavash et al., ; Wagner et al., ; Zellers et al., ; Lu et al., ), or by generating holistic scene descriptions (Song et al., ). In contrast to these related works, our paper emphasizes the visual grounding and reasoning problem of abstract concepts, such as uncommon visualizations of scientific terms or unconventional constructions of strategic games. Such abstract concepts should be grounded on the relationship between lower-level concepts such as entities and patterns, therefore positing novel challenges to vision-language models.

##### Concept representations in minds and machines.

Our schema-based concept representations are inspired by the theory-theory of concepts in cognitive science and artificial intelligence (Schank & Abelson, ; Morton, ; Carey, ; Gopnik, ; Gopnik & Meltzoff, ; Carey, ). In short, these works treat concepts as organized within and around theories. Therefore, acquiring the concept involves learning its theory, and reasoning with a concept involves causal-explanatory reasoning of the theories. Such views have been largely leveraged in generating symbolic representations for concepts such as word meanings (Speer et al., ) and object shapes (Biederman, ). However, most works have focused on using only symbolic features to describe scenes and have been primarily applied to simple object-level concrete concepts. In contrast, we propose using our schema representation as a guide for grounding scene-level, abstract concepts in natural images.

##### Hierarchical grounding of visual concepts.

Our approach is related to recent works on chain-of-thought reasoning in vision-language models (Lu et al., ; Chen et al., ), hierarchical image classification (Koh et al., ; Menon & Vondrick, ), hierarchical relation detection (Li et al., ), and visual reasoning via programs (Hu et al., ; Mao et al., ; Hsu et al., ). In contrast to these algorithms, which are primarily designed for object and relational reasoning, we focus on the grounding of scene-level concepts. Our framework also shares a similar spirit to prior approaches (Menon & Vondrick, ; Li et al., ) that leverage vision-language models to generate text descriptions of class features for classification tasks. However, our paper proposes the generation of hierarchical schemas with LLMs to ground visual abstractions for reasoning. Additionally, our hierarchical schema representation is related to hierarchical program representations used in text-only reasoning (Gao et al., ; Zelikman et al., ; Zhang et al., ); in contrast, we leverage program representations for understanding concepts in natural images. Notably, DSG performs schema inference with a VLM; compared to fully symbolic inference processes, our framework is capable of leveraging all visual information in the image for interpreting abstractions.

## 3 Deep Schema Grounding

Figure: Figure 2: Deep Schema Grounding consists of three main stages: (1) extracting schemas of abstract concepts with large language models, (2) hierarchically grounding schemas on images with vision-language models, and (3) conducting visual question-answering augmented with resolved schemas.
Refer to caption: x2.png

We study the task of reasoning about visual abstractions by concretely focusing on its visual question-answering (VQA) form, although our idea naturally generalizes to other visual interpretation and reasoning tasks. A VQA instance is a tuple of $\langle v,q,a\rangle$, where $v$ is an image, $q$ is a natural language question, and $a$ is the answer to the question. Here, $a$ can take the form of multiple choices or free-form responses in natural language. We assume that each image $v$ has an underlying abstract concept associated with it (e.g., “maze”), which is not associated with concrete visual features, but a set of lifted rules that define the concept. We also assume that $q$ is a question about the abstract concept, and the concept is revealed to all models in question $q$.

The visual abstraction understanding task posits a series of challenges to off-the-shelf VLMs. These questions are centered around concepts with infinite possible instantiations in the real world. As a result, current VLMs struggle with this problem due to the potential novelty of the scene and the visual abstraction combinations encountered. However, a crucial insight regarding these abstract concepts is the consistency in composition patterns of lower-level concepts, which aggregate to construct the high-level concept; these underlying patterns remain the same across different variations.

Based on these insights, we propose Deep Schema Grounding (DSG), a framework that leverages schemas to interpret and reason about visual abstractions. DSG can be built on top of most pretrained vision-language models (VLMs) to perform the visual question-answering task. Illustrated in Figure 2, the DSG framework consists of three main steps: (1) extracting a schema of the concept, (2) hierarchically grounding the schema to the visual input, and (3) leveraging the resolved schema as input to the base VLM. We describe the schema definition and each stage in detail below.

### 3.1 Visual Abstraction Schema

A visual abstraction schema is a concise program that defines a directed acyclic graphical (DAG) representation of a particular concept. As illustrated in Figure 2, each node in the schema corresponds to a subcomponent concept of the higher-level abstract concept. For example, the formation of a maze can be decomposed into three components: the layout, the construction of the walls, and the positioning of the entry and exit of the maze. The dependencies among individual components yield a DAG configuration—in this case, the placement of the entry and the exit of the maze depends on the layout of the maze. From an inference perspective, the dependency graph describes the order in which the model should interpret each component, and how the interpretation of a particular component should be conditioned on the interpretation of one or more previously interpreted components.

The most important feature of a visual abstraction schema is that the definition of such concepts is universal, in the sense that the program describes a universal characterization of mazes made of various components, rather than being restricted to specific visual instantiations (e.g., mazes formed by drawn lines). Schemas enable machines to better generalize to novel scenes.

Figure: Figure 3: Examples of schemas for concepts and the visual features that they may be grounded to.
Refer to caption: x3.png

### 3.2 Extracting Schemas of Abstract Concepts

DSG utilizes large language models (LLMs) as a repository of knowledge from which schemas can be acquired. Given that LLMs are trained on extensive corpora of human language data, we hypothesize that they contain human-aligned schemas for a wide range of abstract concepts. Compared to prior works that use LLMs to translate natural language utterances into programs, DSG uses LLMs to extract explicitly structured knowledge of abstract concepts based on the concept’s name.

In particular, our goal is to extract a universal schema for abstract concepts that is not instance-specific, but is applicable across diverse visual stimuli. Notably, our prompt to the LLMs is generic and simple, with only one example of a schema for the abstract concept “academia” (which consists of key components: faculty, students, etc.), and a short instruction of the schema generation task. We find that LLMs demonstrate a remarkable ability to extrapolate across a wide array of concepts, and produce faithful decompositions of the visual abstractions. We detail the prompt and LLM-generated schemas for each concept in the Appendix.

### 3.3 Hierarchically Grounding Schemas on Images

Instead of directly answering questions given the image and the question inputs, we first hierarchically ground individual components in the schema to visual entities in the image. In the maze example in Figure 2, this corresponds to inferring the materials of the walls, the layout of the maze, etc.

Figure: Figure 4: The schema grounding process.
Refer to caption: x4.png

Formally, given a schema definition of the concept, DSG hierarchically grounds each component in the concept DAG. The outcome of this process is represented as short text descriptions, such as “wall: coffee beans”. We implement this procedure by leveraging pretrained VLMs to predict the most likely grounding of a component given the image and a text query (see Figure 4). Since our grounding procedure is hierarchical, for components that are conditioned on other concepts (e.g., the interpretation of entry-exit is conditioned on the layout), the VLM also takes into account the grounding results of these prerequisite components. This ensures that each step of the grounding process is informed by the previously established context, facilitating more coherent and accurate grounding.

Grounding components following the hierarchical schema structure is crucial for abstract concepts with many possible instantiations. For example, for the “maze” concept, DSG first resolves components that are more coarse and concrete (e.g., layout), then uses those components as conditions to resolve more difficult grounding problems of fine-grained and abstract components (e.g., entry-exit). Figure 3 shows examples of schemas and how they may be grounded onto images. More examples and failure modes can be found in the Appendix.

### 3.4 Visual Question-Answering Augmented with Grounded Schemas

Finally, DSG leverages the grounding of individual components in the image to answer questions. For example, given the concept “maze” and the grounded mapping of individual components: {layout: rectangular, walls: coffee beans, entry-exit: coffee cups}, we augment the final question-answering step by providing the VLM with the component mapping, as well as the full conversation history of the grounding process. In particular, the groundings of the components are included in a text prompt such as: “Imagine that the image represents a maze, and the layout is rectangular, and the walls are coffee beans, and the entry and exit are coffee cups.” Compared with baselines that only have access to the abstract concept itself (e.g., “Imagine that the image represents a maze.”), our concept grounding system shows significant performance improvement with the resolved schema as holistic context for the image.

Notably, our DSG framework is inference-only: it leverages the strong generalization capabilities of LLMs (for proposing schemas) and VLMs (for grounding schemas and reasoning), to better understand visual abstractions. Programmatic schemas are used as the intermediate representation to bridge the explicit commonsense knowledge in LLMs and the visual reasoning capability of VLMs.

## 4 Visual Abstractions Dataset

In order to evaluate models on visual abstraction reasoning, we propose a new dataset, the Visual Abstractions Dataset (VAD), illustrated with examples in Figure 5. It consists of $180$ images and $3$ questions per image, with a total of $540$ test examples. VAD is comprised of $12$ different abstract concepts spanning $4$ categories, where each concept is associated with $15$ examples of real-world images. The questions bridge multiple question types: binary-choice, counting, and open-ended questions. We detail the set of abstract concepts and the image and text components of VAD below.

Figure: Figure 5: The Visual Abstractions Dataset comprises diverse, real-world images that represent $12$ different abstract concepts across $4$ categories.
Refer to caption: x5.png

##### Abstract concepts.

We choose $12$ different visual abstractions as concepts. All of them can be grounded on different objects and configurations of objects in images. We focus on concepts that are not tied to distinct visual features (e.g., unlike canonical colors and shapes) but instead require higher-level, relational, and lifted patterns.
These visual abstractions cover a wide range of visual scenarios and possible linguistic queries, and they can be grouped into four broader categories.

The first category consists of strategic concepts, which are games characterized by rules and patterns, including “tic-tac-toe”, “maze”, and “treasure map”. The exact object instantiations and visual features may vary, but humans can easily identify the state of the game when given the concept. The second category consists of scientific concepts with no possible visualization in the physical world, including “solar system”, “atom”, and “cell”. While we cannot directly view these concepts, we can create analogies of them based on everyday objects. The third category consists of social concepts, which are intentional actions that involve the deduction of roles and relations, including “helping”, “deceiving”, and “negotiating”. Examples of these theory-of-mind concepts in the real world can be seen in many videos and movies; for a single image frame, we must make deductions to assign possible roles to entities in the scene. The fourth category consists of domestic concepts, inspired by everyday household tasks (Li et al., ). These concepts are goals in household environments that cannot be easily defined by specific arrangements of objects, including “table setting for two”, “guest bedrooms”, and “door decoration”.
While humans easily recognize when such objectives are completed, we do not have a simple definition for what exact configuration should be generated.

##### Images and question-answers.

We curate $15$ images for each abstract concept from the Internet. Our criteria for the images are to be real, natural, and captured in the physical world. Our final set of images is diverse in terms of subjects, environments, and domains. See Figure 5 for examples.

We create $3$ types of natural language questions for each abstract concept in VAD. These questions ask about different aspects of the images and take various forms, including binary-choice questions, counting questions, and general open-ended questions. In total, there are $6$ types of counting questions, $14$ types of binary-choice questions, and $16$ types of open-ended questions. An example of a binary-choice question is “Imagine that the image represents table setting for two. What is the configuration of the seats? Describe the answer as across or adjacent.”; an example of a counting question is “Imagine that the image represents a cell. Excluding the nucleus, how many types of organelles are there?”; an example of an open-ended question is “Imagine that the image represents a treasure map. Based on the image, find the starting location; what is the main color of the map region that is physically closest to the starting location?” The overall dataset comprises $540$ of such questions.

Answers to all image-question pairs are labeled by $5$ human annotators from [Prolific](www.prolific.com); we take their modal response as the answer, to be used for exact-match text accuracy. While the Visual Abstractions Dataset focuses on multiple-choice questions and answers, we also provide a free-response variant of VAD. We present examples for each type of question along with corresponding images and answers in the Appendix, and release our dataset [here](https://downloads.cs.stanford.edu/viscam/VisualAbstractionsDataset/VAD.zip).

## 5 Results

**Table 1: DSG outperforms prior works across all questions and different question types in the Visual Abstractions Dataset. Results are averaged over $5$ runs.**
|  | All | Counting | Binary | Open |
| --- | --- | --- | --- | --- |
| ViperGPT (Surís et al., ) | $0.260$ | $0.191$ | $0.426$ | $0.141$ |
| VisProg (Gupta & Kembhavi, ) | $0.304$ | $0.249$ | $0.385$ | $0.255$ |
| LLaVA (Liu et al., ) | $0.415$ | $0.344$ | $0.552$ | $0.321$ |
| InstructBLIP (Dai et al., ) | $0.437$ | $0.322$ | $0.538$ | $0.392$ |
| GPT-4V (OpenAI, ) | $0.653$ | $0.611$ | $0.659$ | $0.663$ |
| DSG (ours) | $\mathbf{0.707}$ | $\mathbf{0.678}$ | $\mathbf{0.707}$ | $\mathbf{0.719}$ |
| Human | $0.846$ | $0.816$ | $0.833$ | $0.868$ |

**Table 2: DSG improves the performance of GPT-4V across categories of abstract concepts.**
|  | Strategic | Scientific | Social | Domestic |
| --- | --- | --- | --- | --- |
| GPT-4V (OpenAI, ) | $0.644$ $\pm 0.033$ | $0.572$ $\pm 0.098$ | $0.679$ $\pm 0.098$ | $0.717$ $\pm 0.053$ |
| DSG (ours) | $\mathbf{0.721}$ $\pm 0.057$ | $\mathbf{0.631}$ $\pm 0.134$ | $\mathbf{0.724}$ $\pm 0.071$ | $\mathbf{0.753}$ $\pm 0.058$ |

We validate Deep Schema Grounding in comparison to baseline VLMs and integrated LLMs with APIs on the Visual Abstractions Dataset. We first compare our work with ViperGPT (Surís et al., ), VisProg (Gupta & Kembhavi, ), LLaVA (Liu et al., ), InstructBLIP (Dai et al., ), and GPT-4V (OpenAI, ). Then, we ablate different components of the DSG framework. Finally, we show that DSG is model-agnostic and can benefit both closed-source and open-source models. Baseline models are given the abstract concept itself in the prompt along with the question (e.g., “Imagine that the image represents a maze.”), but without DSG’s resolved schema of the concept. All results are averaged over $5$ runs. Here, we focus on a systematic evaluation of multiple-choice answers, where accuracy is calculated as exact-match accuracy. In the Appendix, we provide additional comparisons with a graded accuracy metric based on the degree of alignment to human judgments, and report results on free-form answers evaluated by BERTScores (Zhang et al., ).

##### VLMs benefit from decomposing abstract concepts into primitive-level components.

Table 1 summarizes the multiple-choice accuracy of DSG with GPT-4V as the base VLM in comparison to that of prior works across $5$ runs. DSG significantly outperforms VLMs and integrated LLMs with APIs across all questions and all types of questions. In particular, our method demonstrates a boost in accuracy of $5.4$ percent point overall ($\uparrow 8.3\%$ relative improvement) compared to GPT-4V. Across different question types, DSG shows the largest performance improvement of $6.7$ percent point ($\uparrow 11.0\%$ relative improvement) for counting-based questions, such as “How many entities are helping?” We hypothesize that the significant improvement in accuracy for these questions is due to more accurate labels for objects with DSG, as the schema grounding step will concretely associate concepts such as “the entity who is helping” with particular parties in the image (e.g., pandas). We see that these questions usually require a more sophisticated reasoning process to answer correctly. In comparison, for binary-choice questions, there may exist more superficial correlations that off-the-shelf VLMs can rely on. Importantly, we also report a random human annotator’s accuracy of adhering to the modal human response, to highlight the difficulty of problems in VAD.

In Table 2, we more closely analyze the performance improvement of DSG on GPT-4V across concept categories. We report the average performance and the standard deviation for $5$ runs. We see that DSG outperforms GPT-4V across the four categories. In particular, our method demonstrates a strong boost in accuracy of $7.7$ percent point ($\uparrow 12.0\%$ relative improvement) for strategic concepts, and $5.9$ percent point ($\uparrow 10.3\%$ relative improvement) for scientific concepts. In the Appendix, we include qualitative examples of cases where DSG succeeds and fails in correctly grounding schemas. Overall, we see that DSG’s decomposition-based framework improves performance of VLMs.

**Table 3: Summarization of DSG ablations; the full DSG framework shows strongest performance.**
|  | schema | grounding | hierarchy | context | All | Counting | Binary | Open |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + schema | ✓ | ✗ | ✗ | ✗ | $0.647$ | $0.613$ | $0.647$ | $0.659$ |
| + grounding | ✓ | ✓ | ✗ | ✗ | $0.671$ | $0.653$ | $0.674$ | $0.676$ |
| + hierarchy | ✓ | ✓ | ✓ | ✗ | $0.683$ | $0.651$ | $0.690$ | $0.688$ |
| DSG (ours) | ✓ | ✓ | ✓ | ✓ | $\mathbf{0.707}$ | $\mathbf{0.678}$ | $\mathbf{0.707}$ | $\mathbf{0.719}$ |

##### Explicit hierarchical grounding of schemas is better than implicit and sequential grounding.

We present ablation studies of different modules in DSG in Table 3. In particular, we evaluate (1) a variant of DSG that is given only the LLM-generated schema (+ schema), without explicit grounding of each symbol, (2) a variant where DSG performs decomposed grounding sequentially instead of hierarchically (+ grounding), and (3) a hierarchical variant (+ hierarchical) where in the final question-answering step, we provide the model with only the grounding result of each schema component, instead of a full history of its grounding process.

With the first variant, we test whether GPT-4V can use the schema appropriately without a structured thinking process. We explore the second variant to test whether decomposing the visual abstraction into a set of key elements is sufficient for improving the understanding of abstract concepts, and evaluate the role of hierarchical dependencies in lower-level concepts for grounding. Finally, we present results on the third variant to test whether the full grounding trace of DSG is helpful during the final question-answering step. The results show that all these design choices help: a schema only does not lend itself to improved performance, and hierarchical grounding from concrete to abstract symbols, as well as the inclusion of the full context, improves performance in many concepts.

##### DSG improves other (open-source) VLMs too.

Tables 4 and 5 summarize the results of DSG with open-source LLaVA as the base VLM. DSG yields consistent improvement overall ($\uparrow 7.0\%$ relative improvement). However, in contrast to GPT-4V, both LLaVA and DSG yield poor results on counting questions, as the base VLM struggles at enumeration regardless of its abstraction understanding.

**Table 4: DSG improves the performance of the open-source LLaVA model.**
|  | All | Counting | Binary | Open |
| --- | --- | --- | --- | --- |
| LLaVA (Liu et al., ) | $0.415$ | $\mathbf{0.344}$ | $0.552$ | $0.321$ |
| DSG (ours) | $\mathbf{0.444}$ | $\mathbf{0.344}$ | $\mathbf{0.590}$ | $\mathbf{0.354}$ |

**Table 5: Comparison of LLaVA and DSG across categories; DSG outperforms the base VLM.**
|  | Strategic | Scientific | Social | Domestic |
| --- | --- | --- | --- | --- |
| LLaVA (Liu et al., ) | $0.407$ $\pm 0.038$ | $0.333$ $\pm 0.101$ | $0.378$ $\pm 0.126$ | $0.541$ $\pm 0.091$ |
| DSG (ours) | $\mathbf{0.430}$ $\pm 0.046$ | $\mathbf{0.378}$ $\pm 0.119$ | $\mathbf{0.393}$ $\pm 0.154$ | $\mathbf{0.578}$ $\pm 0.096$ |

## 6 Discussion

Our framework demonstrates that by explicitly grounding conceptual schemas through large pretrained vision and language models, we can achieve better visual abstraction understanding in diverse real world instantiations. We find that LLMs generally possess well-structured knowledge of concepts, outputting comprehensive schemas that are accurately decomposed into more concrete symbols and their dependencies. The extracted schemas effective capture the concise essence of a concept, such as the answer to our initial question of what makes a maze look like a maze. However, as DSG relies on pre-trained LLMs to generate schemas and does not restrict the schemas to a set of specified symbols, it is possible that the schemas may contain harmful biases based on the concept given, or may include symbols that are difficult for VLMs to interpret.

In particular, we observe that current VLMs struggle to ground challenging schema components involving spatial constraints. DSG does not explicitly improve this capability or specify how to parameterize spatial configurations. While DSG can generate schema components that contain spatial relationships, and try to ground them by describing them in language, it remains limited by the VLMs’ abilities to conduct such spatial understanding. For example, let us consider a case where the extracted schema for the concept maze includes a symbol representing “an unobstructed path from entry to exit”. That such a path exists in the maze—typically only one and often one that is difficult to find—is a defining feature of mazes, yet is challenging for VLMs to correctly identify. VLMs’ lack of spatial understanding weakens DSG ability to interpret complex and spatially grounded components. Scaling DSG to better understand concepts that require precise layouts of schema components is difficult but crucial for future work.

A key aspect of DSG is that our framework does not limit the expressiveness of schemas to a specific set of symbols or the grounding process to a specific set of answers. The only prior we inject into DSG is the hierarchical process of resolving the schema—the interface between the LLM and VLMs is unconstrained language. In theory, DSG can propose and ground any concept and any individual component, allowing the LLM and VLM to determine what defines a visual abstraction without any restrictions. But such flexibility may also render the schema ambiguous and not universal enough to improve downstream tasks. However, we see that the empirical evidence in our paper supports DSG, and we believe that DSG serves as a proof of concept for the kind of inductive biases that should be incorporated into visual reasoning systems—a structured thinking process guided by flexible but explicit knowledge can augment visual abstraction interpretation.

While we have made progress on understanding what makes a maze look like a maze, in proposing DSG as a framework for leveraging explicit schemas backed by powerful LLMs and VLMs, we are far from solving all reasoning across abstract concepts as humans do. There may be ways to improve grounding of individual components, as well as more efficient priors that can be incorporated into the system. We leave such exploration to future work.

## 7 Conclusion

We propose Deep Schema Grounding as a promising approach to understanding visual abstractions. DSG leverages schemas of concepts to decompose abstract concepts into subcomponents and model their dependencies. Our framework extracts explicitly structured schemas from large language models and hierarchically grounds them to images with vision-language models. On the Visual Abstractions Dataset, a visual question-answering benchmark composed of real-world images with diverse underlying abstract concepts, DSG demonstrates significant improvements compared to base VLMs. DSG is a step toward interpreting visual abstractions as humans do; more remains to be done in the challenge of visual abstraction reasoning.

#### Acknowledgments

We thank Michael Li and Weiyu Liu for providing valuable feedback. This work is in part supported by the Stanford Institute for Human-Centered Artificial Intelligence (HAI), ONR N00014-23-1-2355, ONR YIP N00014-24-1-2117, NSF RI #2211258, Air Force Office of Scientific Research (AFOSR) YIP FA9550-23-1-0127, the Center for Brain, Minds, and Machines (CBMM), the MIT Quest for Intelligence, MIT–IBM Watson AI Lab, and Analog Devices. JH is also supported by the Knight Hennessy Scholarship and the NSF Graduate Research Fellowship.

## Supplementary for: What Makes a Maze Look Like a Maze?

The appendix is organized as the following. In Appendix A, we present comparisons of Deep Schema Grounding to prior works with a graded accuracy metric, comparisons on free response generation, qualitative examples of DSG, and examples of hierarchical grounding. In Appendix B, we show examples of the Visual Abstractions Dataset, describe our annotation collection process, and release our dataset. In Appendix C, we provide our prompt to the LLM and all extracted schemas.

## Appendix A Additional Results

### A.1 Graded Accuracy Comparison

Due to variation in human responses on the Visual Abstractions Dataset—arising from natural errors in annotation or potential ambiguity in questions—we compare Deep Schema Grounding to prior works with a graded accuracy metric, based on degree of alignment to human judgements. Predicted answers each receive $\frac{K}{5}$ points, where $K$ is the number of human annotators out of $5$ that agree with the prediction. In Table 6, we report results with the graded accuracy metric over $5$ runs. DSG consistently outperforms previous works, showing an overall accuracy increase of $5.2$ percent point ($\uparrow 8.5\%$ relative improvement), and an accuracy boost in counting questions of $7.3$ percent point ($\uparrow 13.4\%$ relative improvement). We observe a similar trend in performance as with modal response accuracy reported in the main text—more improvement is seen in counting questions.

**Table 6: Comparison of DSG to prior works with a graded accuracy metric based on degree of alignment to human judgements.**
|  | All | Counting | Binary | Open |
| --- | --- | --- | --- | --- |
| ViperGPT (Surís et al., ) | $0.254$ | $0.200$ | $0.413$ | $0.136$ |
| VisProg (Gupta & Kembhavi, ) | $0.293$ | $0.227$ | $0.380$ | $0.242$ |
| LLaVA (Liu et al., ) | $0.400$ | $0.327$ | $0.548$ | $0.298$ |
| InstructBLIP (Dai et al., ) | $0.420$ | $0.307$ | $0.528$ | $0.368$ |
| GPT-4V (OpenAI, ) | $0.611$ | $0.546$ | $0.648$ | $0.603$ |
| DSG (ours) | $\mathbf{0.663}$ | $\mathbf{0.619}$ | $\mathbf{0.680}$ | $\mathbf{0.663}$ |
| Human | $0.789$ | $0.728$ | $0.784$ | $0.816$ |

### A.2 Comparison on Free Response Generation

We present comparisons between DSG and GPT-4V on free response generation in Table 7 and Table 8, with accuracy evaluated by BERTScores Zhang et al. (). We see that DSG outperforms GPT-4V across all questions and question types, and most concept categories.

**Table 7: Performance on free response generation; DSG outperforms the base VLM.**
|  | All | Counting | Binary | Open |
| --- | --- | --- | --- | --- |
| GPT-4V (OpenAI, ) | $0.855$ | $0.816$ | $0.940$ | $0.795$ |
| DSG (ours) | $\mathbf{0.865}$ | $\mathbf{0.829}$ | $\mathbf{0.952}$ | $\mathbf{0.802}$ |

**Table 8: DSG improves the performance of GPT-4V in free response generation across most categories of abstract concepts.**
|  | Strategic | Scientific | Social | Domestic |
| --- | --- | --- | --- | --- |
| GPT-4V (OpenAI, ) | $0.813$ $\pm 0.044$ | $0.833$ $\pm 0.080$ | $0.903$ $\pm 0.056$ | $\mathbf{0.871}$ $\pm 0.063$ |
| DSG (ours) | $\mathbf{0.828}$ $\pm 0.055$ | $\mathbf{0.845}$ $\pm 0.066$ | $\mathbf{0.917}$ $\pm 0.052$ | $0.870$ $\pm 0.063$ |

### A.3 Qualitative Examples

We present examples of DSG’s resolved schemas in Figure 6; in particular, we highlight cases where DSG’s schema grounding process errors in red.

Figure: Figure 6: Examples of DSG’s resolved schemas across different abstract concepts; in red we highlight failure cases where inference of individual components errors.
Refer to caption: x6.png

### A.4 Hierarchical Grounding Example

In Figure 7, we highlight an example of how DSG’s hierarchical inference improves schema grounding accuracy compared to a sequential version of DSG.

Figure: Figure 7: In comparison to DSG, which hierarchically grounds each component, a sequential variant errors when grounding a more fine-grained symbol, energy-levels, without conditioning on prior predictions of more concrete symbols, nucleus and electrons.
Refer to caption: x7.png

### A.5 Compute Resources

We use OpenAI’s API for GPT-4V (OpenAI, ), ViperGPT (Surís et al., ), and VisProg (Gupta & Kembhavi, ). Open-sourced models, LLaVA (Liu et al., ) and InstructBLIP (Dai et al., ), and the API calls of the aforementioned integrated LLMs with APIs, were run inference-only with $1$ A40 on an internal cluster.

## Appendix B Visual Abstractions Dataset

### B.1 Dataset Examples

In Figure 9, we provide visual question-answering examples of VAD across concept types and question types.

### B.2 Annotation

We collect annotations for the Visual Abstractions Dataset through [Prolific](www.prolific.com); $5$ human annotators labeled each question-image pair. Figure 8 shows our annotation interface, built from jsPsych. Each annotator was compensated at the rate of $\$12$ USD per hour, and no potential risk was incurred by the annotators.

Figure: Figure 8: Example of our annotation interface shown to Prolific annotators.
Refer to caption: extracted/5849968/figures/annotation_example.png

### B.3 Release

We release the Visual Abstractions Dataset [here](https://downloads.cs.stanford.edu/viscam/VisualAbstractionsDataset/VAD.zip). We group all images by their corresponding abstract concepts. Each image is carefully curated and manually audited to be safe for release.

Figure: Figure 9: Examples of visual question-answering pairs in the Visual Abstractions Dataset.
Refer to caption: x8.png

## Appendix C LLM-Generated Schemas

### C.1 Prompt

Below, we provide the prompt used as input to GPT-4 to extract schemas for visual abstractions; we substitute [abstract-concept] for a given concept.

### C.2 All Schemas

Here, we present all LLM-generated schemas for the Visual Abstractions Dataset.

Strategic concepts.

Scientific concepts.

Social concepts.

Domestic concepts.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Many recent works (https://arxiv.org/pdf/2305.10355, https://arxiv.org/pdf/2401.06209, others) have shown that VLMs  are ‘blind’ and don’t actually look at the visual information in images. VLMs mostly rely on language and questions in VQA benchmarks can be answered by having a good language prior. I was wondering how much of the questions in the proposed benchmark can be correctly answered by “language” only? It would be great to show a “language-only” baseline for the proposed VQA benchmark. Especially after adding grounded schema as context- it can show the impact of providing grounded schema too.

- A qualitative analysis of how/why is the grounded schema is improving VQA capabilities of the VLM would be great. Especially for the cases where the question does not ask anything about the concepts of the schema (like the example of “What is the player in the maze?” mentioned in the weakness section)

- Are the generated schemas verified by some expert? How are we sure that the schemas provided by the LLMs are correct for the provided concepts?
soundness: 3
strengths: - Innovative use of pre-trained LLMs and VLMs to give additional context (in terms of grounded schemas) for VQA. In principle, this additional information should help answer pre-trained VLMs to better answer the questions related to the abstract concepts in the image. (However I think that the generated schemas shown in the appendix of the paper are not detailed enough to achieve this - please check weakness section)

- Hierarchical Grounding: I like how they have used a hierarchical method of grounding components of the schemas. Grounding independent concepts first and then the dependent concepts of the schema should be the correct way for grounding which is used in the paper.
summary: The paper introduces a framework called Deep Schema Grounding (DSG) and a VQA benchmark for evaluating abstract visual reasoning of VLMs. For a given image and a question about the concept in the image, DSG first generates a schema of the concept using LLM. Then it grounds the components of the schema onto images using VLMs. Once the schema is grounded, the grounding information is given as context in text from to the VML to answer the question about the image. They show that giving grounded schema context to the VLMs improves their ability to correctly answer questions about abstract concepts in the image.
weaknesses: - Schemas are not detailed enough to give information about the concept. For example, tic-tac-toe schema include {board, symbols, strategy}. Although a tic-tac-toe game has  {board, symbols, strategy} it is not complete. Many board games have {board, symbols, strategy}. This schema does not tell that the board is a 3x3 grid. Similarly, for “negotiating” the schema is {participants, setting, object}. This schema could be of many different settings. (p.s. I do understand that the capabilities of current VLMs are not enough to handle detailed concepts)

- It is shown that giving grounded schema as additional context to VLM to answer VQA question is improving the performance. But it is not shown or discussed why?/how? is the additional context improving the performance. For instance, in the example given in figure 1 of the paper where the question asks “What is the player in the maze?”, how is giving information about the layout, walls and entry-exit helping a VLM answer question about the ‘player’.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Could you try something like chain of thought prompting as I mentioned above?

Is there any way to generalize the idea to create a more general benchmark?
soundness: 3
strengths: The paper provides a good solution for VLMs to better understand abstract concepts in an image. The presentation of the paper is clear with many figures for demonstration. The experiments are comprehensive, making the main message convincing. The benchmark created is novel and interesting.
summary: The paper proposes Deep Schema Grounding (DSG), a method to break down a visual concept into smaller concepts with dependencies on other concepts. The authors show the capability of DSG in solving complex visual question answering task. A visual abstraction benchmark is proposed with 12 abstract concepts and 180 images.
weaknesses: In terms of the idea behind DSG, it seems to be close to chain of thought with specific instructions. For example, the maze example in the paper can be integrated with just one prompt: "Imagine that the image represents a maze. <the question> Think step by step by recognizing the layout of the maze, the walls of the maze, then the entry and exit of the maze one by one." Maybe gpt-4o can automatically do this even without the instructions. This is probably why in Table 7 if the generation is free form, DSG does not outperform gpt-4o much despite using more API calls.

Therefore, I am a bit concerned whether these types of schemas are necessary. After all, the dependency graph is generated by gpt-4, so at least gpt-4o should know how to do it if prompted well. But I agree that for smaller models, DSG can be very useful.

Another concern I have is about the diversity of the benchmark because the number of categories is quite limited.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: The word "lifted" is used in several places (e.g. "lifted symbols" or "lifted rules"). I think this terminology isn't entirely obvious. If this has a specific meaning, could the authors please elaborate in text (or just use a different word if not)?

It could be informative to clarify the distribution of schemas based on their graph depth in the text. E.g. I see that "cell" has depth 2 and "atom" has depth 3 (L875-881). The single in-context prompt for schema generation is "academia" (depth=3; Sec. C.1). Could abstract concepts exist with depths larger than 3? Would the LLM limit their decompositions to depth=3 (at most) because that is the only prompt? I think the authors could elaborate further on their prompt engineering and exploration, especially since their method is a prompting framework.
soundness: 4
strengths: This paper is extremely clear and straightforward.

In general, I believe the comparisons are fair and sufficient ablations are provided, so it is methodologically sound. For example, they show a 10% relative improvement when applying their method with GPT-4o and 7% with (the open-source and weaker) LLaVA model. They also compare against other methods (ViperGPT and VisProg) that also use GPT models (GPT-3) for visual question decomposition, but don't perform nearly as well on this benchmark. Finally, they provide sufficient ablations of their own method (separating the schema/grounding/hierarchy/context components and showing incremental gains of each).

I believe this is a creative benchmark and that existing vision–language benchmarks under-explore abstract visual concepts. The method is task-specific, but I think that is fine, because it is prompting-based. I don't think the approach is entirely original but I think it sufficiently rounds out the paper: they show how explicit structure improves performance of existing models on their benchmark.

Altogether: this paper presents a benchmark, shows that existing vision–language models underperform, and introduces an explicit prompting method that can be applied with any LLM/VLM to improve performance. The best performance with this method is still just 73% (with the strongest, closed models) or 44% (with open-source models), leaving sufficient challenge for future work.
summary: This paper introduces a VQA benchmark and method for visual abstract reasoning.

The "Visual Abstractions Benchmark" consists of 180 images, annotated with 3 questions each (and each question has 5 answer annotations from Prolific workers). These questions ask about abstract concepts in the image: for example, "Q: Imagine that the image represents a maze. Are there clear entrances or exits in the maze? [Yes or No]" (Figure 1; an image of an ice skating rink partitioned like a maze).

The method, "Deep Schema Grounding (DSG)", is a prompting framework that (1) uses an LLM to decompose an abstract concept into a schema, (2) uses a VLM to ground the "primitive" concepts from the schema to concrete concepts in the image, and (3) uses a VLM to answer the question given the schema and grounding.
1. The schema is a DAG: a "maze" would decompose into "layout" and "walls", and "layout" further decomposes into "entry-exit".
2. A VQA question is used for grounding: e.g. given the image, "..., what is the layout?". This graph is hierarchically grounded: e.g. "entry-exit" is grounded using the question "..., the layout is [], what is the entry and exit?". The ice rink example above entails the following grounding: "{layout: grid-like, walls: ice, entry-exit: gates}".
3. The image and grounded schema are joined as a VQA question for the VLM to produce a final answer.

Applying this framework improves performance of GPT-4o by 10% (relative) on the benchmark.
weaknesses: I believe the construction of the dataset is under-specified (and I also checked the appendix). I understand that answer annotations were provided by crowd workers. However, how were the questions written? How were the images selected (and where do they come from: L376 just says "the Internet")? Likewise for the 12 abstract concepts and their 4 categories. Right now, I am imagining that the authors curated these themselves. That could be fine, but I would like to know more details, so that we can determine whether there may be any biases: e.g. does the introduced method perform better on this particular selection of abstract concepts?

I notice that answers in the open-ended setting are evaluated using BERTScore. However, (Kamalloo 2023) says

> unsupervised semantic similarity-based evaluation metrics such as BERTScore (Zhang et al., 2020) that rely on token-level matching of contextualized representations exhibit poor correlation with human judgment in QA evaluation (Chen et al., 2019) and lack the ability to capture attributability (Maynez et al., 2020)

I also find this to be true in general. Could you show examples, so we can see if the metric is indeed suitable for this distribution of data? Otherwise, could you consider using a method that is better aligned with human judgments for this evaluation: possibly (Kamalloo 2023)?

Reference: [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://aclanthology.org/2023.acl-long.307) (Kamalloo 2023)

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) Is the proposed DSG method applicable to other general visual reasoning benchmarks, such as RAVEN, Bongard, CVR, and SVRT?
2) Can DSG apply to the scenario where there are multiple abstract concepts in an image? I expect the authors to provide some insight for this, but I don't expect the authors to scale up to more experiments in the rebuttal though.
3) In table 3, the performance of schema plus grounding surpluses the performance of DSG in counting problem. Why would this happen?
4) I also have some questions regarding the length control of the schema. In the prompt provided in the appendix, it is limited to 'Keep the program simple with four or fewer components.' Why is this limitation imposed? Can the authors provide experimental results for schemas with different numbers of components? Or should the depth of the schema increase for more abstract concepts, while simpler schemas can suffice for more general concepts?
5) How robust is the method in general to failures: How often does the LLMs fail to provide a valid schema and how often does VLMs fail to provide a correct grounding? Does an invalid schema or grounding result in the failure when generate the answer?
soundness: 3
strengths: 1) The paper is well-written, it effectively conveys its proposed framwork and the motivation behinds it with clarity.
2) The experiments are well-made and the experimental analysis is comprehensive. Extensive experiments validate the effectiveness and efficiency of the DSG, which serves as a plug-and-play trick to enhance the performance of existing VLMs.
3) The pipeline that utilize schema representations with hierarchical grounding for visual concept understanding seems reasonable to the reviewer, as it emulates the human cognitive process of understanding abstract concepts.
4) The proposed method is simple yet effective, and it is easy to follow.
summary: The paper proposes the Deep Schema Grounding (DSG) framework, aiming to enhance VLM’s capability in understanding abstract visual concepts like “the maze”. The framework integrates schema representations with hierarchical grounding to guarantee the VLM’s to better capture human-aligned visual abstractions. DSG operates by extracting schemas of abstract concepts, then grounding schemas on images following a hierarchical manner, then augmenting the VQA problem with resolved schemas, resulting in improved visual reasoning. The authors also introduce the Visual Abstractions Benchmark (VAB) to systematically assess abstract concept reasoning. Experimental results indicate that DSG consistently outperforms baseline models across diverse abstract concept categories and question types.
weaknesses: 1) The gerealibility is somehow resricted. The method has been tested on only 12 types of abstract concepts and it has not been tested on the problems which invovles multiple abstract concept, which raises questions about its generalizability to a broader range of abstractions.
2) The techinical contribution of DSG is limited. The main technical contribution of this paper lies in the combination of neural-symbolic and VLM by the development of specifically designed prompts. (Prompt for LLM to generate schema, prompt for VLM to generate concept grounding). The improvements made to the prompts are relatively straightforward, and the utilization of CoT is a standard practice.
3) The risk of the hallucination of LLM/VLM. During schema construction and hierarchical prompting, the LLM may produce hallucinated errors. If errors occur in hierarchical content, individual instances may fail; however, if schema generation is flawed, it could lead to failures across an entire category of problems. The paper lacks a thorough analysis of failure cases. The authors provide some failure cases demonstration in Appendix A.3. But it seems that these results are not enough. See question 5) for further suggestion.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Overall the limited size of benchmark is my primary concern. I like how paper design the framework and benchmark. The benchmark provide a  good practice on handle and evaluate abstract concept, which is helpful in current literature.

Questions: 
1. In llnes 399 to 400, it seems that there are multitypes of question and also each question type has different baseline accuracy (0.5 for binary in ideal). Do each categorie has a same distribution of different questions types? Otherwise the comparison over each category might be misleading. 
2. Is it possible the schema generated from LLMs doesn't fit well with image? For example, the first maze image (candy maze) in Figure 5, it is unclear for human what is the layout and where is the entry point.  How does the framework handle this scenario where the generated schema bring additional concept and may cause hallucinations.



Suggestion: 
1. To make reader understanding the significance of abstraction, author can enrich the introduction with some practical application of abstraction elaborate more on how human use abstraction, such abstraction enable human mind to learning from small, obtain different levels of abstraction which can later be used to trim the learning space for new concepts.
soundness: 3
strengths: 1.  Schema-based framework: The paper start from cognitive science and theory-theory to purpose a schema-based concept representation framework. The process of representing high-level abstract concept and later grounding in actual image bring boost in model's ability of abstract understanding. The pre-generetad abstract concept schema seems helpful on mitigate halluciations (16.6% relative improvement in counting-based questions) and is beneficial for both open and closed source models.
2. Curately designed Visual Abstraction Benchmark (VAB): The author provide detail explanation the reason behind VAR design choice. Detailed description of data collection and curation process is provided. The dataset is released for public use. As abstraction can play important role in understanding concept and ease the learning process, I believe this benchmark is a good step for solving the abstraction problem.
3. A comprehensive experiment: the experiment extensively test current LVMs and point out their weakness in visual abstraction understanding.  Further experiment demonstrate the effectiveness of DSG and the ablation studies prove the design choice of each module. 
4. Writing. The paper is well-written overall, with clear and well-designed figures and tables. The comprehensive appendix offers additional details beyond those covered in the main text. I like the way authors start their paper with an illustration on ''what makes a maze look like a maze'', which provide a easy-understanding approach for readers, especially for the concept of visual abstraction.
summary: This paper introduces Deep Schema Grounding (DSG), a framework that extracts structured representations from LLMs to grounding and reasoning over visual abstractions. DSG leverages the inner knowledge of LLMs to generate schema descripting the abstraction concept. The content within abstract schema will be later grounding into corresponding compoents in the image by Large Vision Model via a hierarchically process. To evaluate the effectiveness of DSG, a new benchmark, Visual Abstraction Benchmark (VAB), is purposed in paper to evaluate model's understanding in visual abstractions.  VAB consists of 540 test examples consists of 180 images presenting 12 abstraction concepts spanning 4 categoris (15 examples each).  

The paper do comprehenisve experiment comparing DSG with 5 current VLMs. The result shows DSG can boost 6.6 percent accuracy oveall compared to GPT-4o while current VLMs shows a big gap between human performance. Further abalation study pinpoint that each modules in DSG is necessary and contribute to its success. DSG can boost model performance on both open-source and close-source model but authors notes the challenge still exist in visual abstraction field.
weaknesses: 1. Size of Benchmark: one concern I have for the benchmark is relatively small size of the data. VAB only comprised 12 abstract concepts acroos 4 categoires, I know the data collection can be challenging but the limited size can decrease the significance of the benchmark evaluation. The author mentions the benchmark is divertse in terms of subject, environments and domains (Line 377). A dataset statistic over these metadata can be really helpful to support the argument. 
2. Error grounding: Perception is also a challenge issue in the whole framework. It is possible that VLM have the understanding of the abstraction concept but fails to percept the image well and ground the concept within actual images (especially for images containing multiple subjects, image of cell in Figure 3).  That may also be the reasoning the schema-based framework become most helpful on couting problem. A further experiment showing VLMs can percept the image correctly is necessary.

Meta-review: This paper introduces Deep Schema Grounding (DSG), which enhances VLMs in abstract visual reasoning by using an LLM to generate schemas that decompose abstract concepts into smaller components, which are then grounded onto images for answering related questions. Experimental results show that DSG improves GPT-4o’s performance by 10% on the newly constructed Visual Abstractions Benchmark (VAB), outperforming other VLMs and demonstrating the contribution of each DSG component through ablation studies.

There are five reviewers for this paper. The reviewers highlight the key strengths of the paper, including its simplicity and effectiveness, along with its contributions to both the benchmark and methodology, supported by well-structured experiments and comprehensive ablation studies.

Meanwhile, the main concerns from the reviewers are 1) the dataset's limited size, along with its lack of diversity in categories and ability to handle multiple abstract concepts in a single image, may not offer enough breadth to effectively evaluate generalizability. 2) the DSG approach seems similar to chain-of-thought prompting with specific instructions. Some reviewers (Reviewer P1iE) question whether schemas are necessary, suggesting that GPT-4 could handle the task with a well-designed prompt. 3) Some schemas, such as those for tic-tac-toe and negotiating, are too generic and fail to capture key details, limiting the information provided to the VLMs. 4) There is a lack of explanation as to why or how the additional context provided by the grounded schema improves performance in VQA. 5) The DSG method has not been tested on other benchmarks, and its robustness to errors or failures in schema generation or concept grounding is unclear. 

During the author-reviewer discussion phase, the authors' clarifications and additional experimental results effectively addressed the reviewers' concerns. All reviewers rated the paper positively and recommended acceptance. ACs support the reviewers’ recommendation. Authors should address the key points raised in the reviews, particularly the requests for additional analyses and experiments, when preparing the final version.

Decision: Accept (Poster)