Do Vision-Language Models Really Understand Visual Language?

## Abstract

Abstract Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicting complex concepts and their relationships in the form of an image. The symbolic nature of diagrams presents significant challenges for building models capable of understanding them.
Yet, recent studies seem to suggest that Large Vision-Language Models (LVLMs) can even tackle complex reasoning tasks involving diagrams.
In this paper, we investigate this phenomenon by developing a comprehensive test suite to evaluate the diagram comprehension capability of LVLMs.
Our test suite uses a variety of questions focused on concept entities and their relationships over a set of synthetic as well as real diagrams across several domains to evaluate the recognition and reasoning abilities of models. Our evaluation of three LVLMs (GPT-4V, GPT-4o, and Gemini) shows that while these models can accurately identify and reason about entities, their ability to understand relationships is notably limited.
Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowledge as shortcuts to identify and reason about the relational information.
Thus, we conclude that LVLMs have a limited capability for genuine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.

## 1 Introduction

Symbolic signals such as language serve as powerful tools in communication by abstracting and interpreting information. Visual language is a form of communication that uses symbols, shapes, and spatial arrangements to convey complex ideas (Greenspan & Shanker, ; Li, ).
Diagrams, which encapsulate symbolic information in the visual stream, are a prime example of visual language (Zdebik, ; Anderson et al., )
that are extensively used in practice across various domains, e.g., mathematics (Seo et al., ), science (Lu et al., ), education (Kembhavi et al., ; ), and illustrations (Hiippala & Orekhova, ; Lu et al., ).
Developing models capable of understanding symbolic information, e.g. in diagrams, is a critical milestone in advancing machine intelligence (Bauer & Johnson-Laird, ; de Rijke, ; Cromley et al., ).
Even though recent Large Vision-Language Models (LVLMs, OpenAI, ; Anil et al., ) have demonstrated some success on diagram-based visual reasoning tasks (Lu et al., ; Zhang et al., ; Chen et al., ), it remains unclear whether the performance on these tasks truly reflects the models’ ability to comprehensively understand the symbolic information in diagrams.

For this purpose, we design a comprehensive test suite that investigates the ability of LVLMs to understand diagrams.
As defined by Foucault () and Deleuze (), diagrams are abstract tools that organize visual entities using relational information.
Drawing inspiration from this, our test suite focuses on evaluating diagram understanding by assessing how well models can understand entities and relations in typical diagrams ([§ 2.1](#S2.SS1)).
We evaluate diagram understanding by defining two types of tasks pertaining to fast recognition of entities and relations and slow multi-step reasoning (Kahneman, ) over the relations
([§ 2.2](#S2.SS2)).
While we cannot cover every diagram type for practical reasons, we still cover diagrams across six domains. To ensure that our evaluation is both controlled and generalizable, our test suite includes both clean synthetic diagrams and $1,001$ annotated real diagrams carefully selected from existing datasets Krishnamurthy et al. (); Kembhavi et al. () ([§ 2.3](#S2.SS3)).
Using our test suite, we conduct a detailed analysis of the model’s strengths and weaknesses in understanding diagrams and explore the following questions:

Figure: Figure 1: The responses of GPT-4o to two diagram-related questions reveal a notable pattern. The model struggles to correctly answer the relation question in the simple synthetic diagram, yet it successfully understands the relationship in a complex real diagram. We demonstrate that this pattern occurs consistently ([Tabs. 4](#S3.T4) and [5](#S3.T5)).
Refer to caption: /html/2410.00193/assets/x1.png

##### Q1: Can LVLMs understand diagrams?

To assess the basic capabilities of LVLMs, we first test them using clean synthetic diagrams, followed by evaluations in real-world scenarios for comparison. Our findings from these evaluations lead to three key observations:

- •
LVLMs can identify entities and reason about them.
By generating synthetic diagrams to evaluate models from multiple perspectives, we observe that models consistently perform well on entity-related questions. They can accurately identify and reason about entities in synthetic diagrams, regardless of whether the entity is represented textually or visually ([§ 3.1](#S3.SS1)).
- •
LVLMs struggle with identifying and reasoning about relations ([Fig. 1](#S1.F1)a).
In synthetic scenarios, the models exhibit significant difficulty in identifying relationships between depicted concepts and in performing reasoning tasks based on those relationships ([§ 3.2](#S3.SS2)). This challenge persists across various diagram settings and prompting templates ([§§ E.2.2](#A5.SS2.SSS2) and [E.2.3](#A5.SS2.SSS3)).
- •
For real diagrams, LVLMs still understand entities and cannot reason about relations. But they can identify relations ([Fig. 1](#S1.F1)b).
We annotate real diagrams with questions from multiple aspects and evaluate models on them.
Unexpectedly, we find that the models perform significantly better on relation recognition questions in real diagrams compared to those in synthetic diagrams ([§ 3.3](#S3.SS3)).

##### Q2: If LVLMs cannot identify relations in simple synthetic diagrams, how do they manage to answer complex questions in practice? ( Fig. 1 c)

One potential hypothesis is that the models leverage their background knowledge as a shortcut to answer these questions. To test it, we explore the impact of knowledge on question-answering (QA) and draw the following observations:

- •
Knowledge enhances model performance in relation recognition.
We construct synthetic diagrams that incorporate semantic knowledge and observe a notable improvement in relation recognition questions, suggesting that models perform better on knowledge-grounded diagrams ([§ 4.1](#S4.SS1)).
Additionally, in real diagrams, we categorize questions based on whether they require background knowledge (e.g., commonsense) or can be answered independently of it. The results show that LVLMs excel at answering questions that draw upon background knowledge ([§ 4.2](#S4.SS2)).
- •
LVLMs only answer relation questions correctly for simple real diagrams.
We classify diagrams by complexity that is based on the number of entities, and analyze QA performance for simple and complex diagrams. Results reveal that while accuracy on entity questions remains consistent, that on relation questions drops significantly with the increase of complexity.
This indicates that the models’ seemingly good performance on relation questions is primarily driven by handling simpler diagrams, rather than by genuine relation understanding ([§ 5.1](#S5.SS1)).
- •
LVLMs rely on learned knowledge to hallucinate relations.
In the case study, we demonstrate that even when no relations are provided, LVLMs infer relations based on their learned knowledge. Furthermore, if provided relations contradict the models’ learned knowledge, they tend to disregard them and instead rely on their background knowledge to answer the questions ([§ 5.2](#S5.SS2)).

##### Findings.

We summarize our research findings as follows: While LVLMs can recognize and reason about entities, they struggle with relations. The models do not engage in genuine diagram parsing or reasoning; rather, their seemingly strong performance on various diagram benchmarks is an illusion created by their reliance on knowledge shortcuts. Specifically, these models identify the entities depicted in the diagrams and simply retrieve relevant pre-learned knowledge.

## 2 Test Suite Design

In this section, we provide a definition of a diagram and provide a desiderata for our evaluation suite.

### 2.1 Diagrams As Graphs

Diagrams work as an abstract tool to describe concepts and relationships (Foucault, ; Deleuze, ).
In practice, we choose this representation as it is quite flexible and a broad set of diagrams can be represented in a format as shown in [Fig. 1](#S1.F1).
For example, logical diagrams such as water cycles illustrate the process transitions (relations) among cycle stages (entities). Schematic diagrams such as circuits demonstrate the connections (relations) among electronic components (entities).
Previous work (Song et al., ; Hiippala & Orekhova, ) also chose to annotate and model diagrams as concepts and their connections.(^1^11Given the vast number of diagram types, we would leave certain complex cases that are challenging to be represented in this way for future work.)

Following this representation, we define a diagram as a graph $\scalebox{0.8}{\mbox{$\displaystyle G$}}=\{\scalebox{0.8}{\mbox{$\displaystyle\mathcal{V}$}},\scalebox{0.8}{\mbox{$\displaystyle\mathcal{E}$}}\}$. Here,
$\displaystyle\mathcal{V}$
is the set of entities (e.g., “Squid” in the example diagram in [Fig. 1](#S1.F1)b). Each entity ${\scalebox{0.8}{\mbox{$\displaystyle V$}}}\in\scalebox{0.8}{\mbox{$\displaystyle\mathcal{V}$}}$ could be represented in multiple ways, e.g., text and visuals in the diagram. Each relation ${\scalebox{0.8}{\mbox{$\displaystyle E$}}}=({\scalebox{0.8}{\mbox{$\displaystyle V$}}},{\scalebox{0.8}{\mbox{$\displaystyle V$}}}^{\prime})\in\scalebox{0.8}{\mbox{$\displaystyle\mathcal{E}$}}$ connects two entities. Relations are either explicitly represented by arrows or via implicit relationships (e.g., relative positioning of entities).

### 2.2 How Do We Evaluate The Model’s Diagram Understanding Ability?

In designing our test suite, we draw inspiration from  (Kahneman, ) who argues that
the thinking process can be naturally divided into two modes: System 1, which handles automatic, quick, and intuitive thinking (e.g., pattern recognition and everyday decisions), and System 2, which is responsible for deliberate, slow, and logical thinking (e.g., logical reasoning and critical analysis). We evaluate both the recognition and reasoning abilities of models via question-answering (QA). We carefully design a set of
questions posed in a multiple-choice QA format with each question having one correct answer and three incorrect options and simply use the model’s accuracy in answering the questions as an evaluation of the model’s ability on that skill. Overall, we denote the set of questions as
$\displaystyle Q$
. We denote the set of
questions related to understanding entities as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}})$, while those related to relations as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}})$.
Additionally, we use subscripts to distinguish between different types of questions: questions on synthetic diagrams are denoted as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}$, whereas questions for real diagrams are denoted as
$\displaystyle Q$
.
Our test suite questions are categorized as follows:

##### Recognition vs. Reasoning Questions.

To measure the two key abilities of models in recognition of entities and relations vs. reasoning, we design two types of questions: Name Recognition $({\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ and Number Counting (
NC
). Specifically, the
NR
questions measure the recognition ability of models by verifying the existence of specific entities or relations. In contrast,
NC
questions measure reasoning ability by asking for the number of certain types of entities or relations.
We formally denote these question sets as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}(\cdot|{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ and ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}(\cdot|{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$, respectively.

##### Knowledge-Required vs. Knowledge-Free.

Next, we test if LVLMs use any knowledge shortcuts (Ye & Kovashka, ; Tang et al., ) to answer our questions without true diagram understanding.
Diagrams often encode some background real-world knowledge, and the models may use their background knowledge as a shortcut to answer the questions. To further tease out the models’ true understanding of diagrams, we design questions that both do and do not require background knowledge, allowing us to test the models’ capabilities in each scenario.
If a question requires the model to use background knowledge (e.g., semantic or commonsense knowledge), we call it a knowledge-required (
KR
) question, which is denoted as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}(\cdot|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}})$. On the other hand, questions that do not rely on such external knowledge are termed knowledge-free (
KF
) questions, denoted as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}(\cdot|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}})$. This distinction helps us clearly separate the model’s capacity for pure visual processing from its ability to incorporate and utilize prior knowledge when answering questions.

**Table 1: The template and example of question annotations. The underlined entity varies across diagrams. There are no KR questions for synthetic diagrams since they do not have background knowledge. Questions for real diagrams correspond to [Fig. 1](#S1.F1)b. In terms of relations, “Explicit Relation” refers to relations that are clearly depicted through arrows or segments, while “Implicit Relation” refers to those that are conveyed indirectly, such as through relative position relationships.**
| Synthetic Diagram | Question | Template |
| --- | --- | --- |
| Entity | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which one of the entities exists in the diagram? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many text labels are there in the diagram? |  |
| Implicit Relation | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which one of the text labels is placed on the left of the entity cow? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many text labels are placed on the left of the entity cow? |  |
| Explicit Relation | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which one of the pairs is connected in the diagram? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many entities are connected to cow? |  |
| Real Diagram | Question | Example |
| Entity | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | How many entities are there in the diagram? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | Which entity is in the diagram? |  |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which producer is in the diagram? |  |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many consumers are in the diagram? |  |
| Relation | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which entity is connected to Fish? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many arrows are linked to Fish in the diagram? |  |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which is not the predator of Krill? |  |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many types of prey are consumed by Fish in the foodweb? |  |

Templates and examples of each question type are presented in [Tab. 1](#S2.T1). Each question type targets a specific diagram component or model ability within the evaluation. For real diagrams, the question templates are tailored to the specific context of each domain. Details on the question design in each domain are in [§ D.1](#A4.SS1). From these questions, we can derive the following intuition:

###### Intuition 1 .

KR
questions in real diagrams are generally more challenging than
KF
questions in synthetic diagrams. The reasons are: 1) Real diagrams are inherently more complex, containing a wider range of information compared to synthetic diagrams; 2) Beyond assessing basic abilities, answering
KR
questions also requires the integration of additional background knowledge.

###### Intuition 1 .

### 2.3 Which Diagrams Do We Consider?

Diagrams are extensively utilized in various domains, appearing in different forms and encompassing a wide range of information types. While evaluating models in a synthetic setting helps to reduce the impact of confounding factors, the resulting conclusions may not fully extend to real-world cases. Conversely, evaluating models on real diagrams allows for broader coverage of diagram types, though it may introduce biases due to the additional information or knowledge embedded in these diagrams. To address this challenge, our test suite incorporates both synthetic and real diagrams, providing a balanced and comprehensive evaluation.

##### Synthetic diagram set.

We generate synthetic diagrams in two steps. First, we randomly create between $2$ to $9$ entities represented by images or text. Then, we randomly establish between $1$ to maximum relations among them using directed arrows.(^2^22The probability of random strategy distributes uniformly across entity number and relation number.)
To ensure clarity in the synthetic diagrams, we carefully avoid situations when arrows cross over certain entities.
In practice, we construct our entity set
$\displaystyle\mathcal{V}$
by sampling from a pre-defined set containing $377$ distinct entities provided by Lu et al. (). By default, we generate $1,000$ diagrams for each experiment.

**Table 2: Details of the real diagram set. “Entity Rep.” and “Relation Rep.” refer to the way that entities and relations are represented. “Topics” introduces the typical types of diagrams in that domain.**
| Domains | Ecology | Biology | Physics | Astronomy | Chemistry | Geology |
| --- | --- | --- | --- | --- | --- | --- |
| Num. | 462 | 205 | 77 | 101 | 54 | 102 |
| Entity Rep. | Text & Visual | Text & Visual | Visual | Text & Visual | Text & Visual | Text & Visual |
| Relation Rep. | Explicit | Explicit | Explicit | Implicit | Explicit | Implicit |
| Topics | Food Chain | Life Cycle | Circuit | Solar System | Water Cycle | Planet Structure |
| Food Web | Satellite System | Carbon Cycle | Star Structure |  |  |  |

##### Real diagram set.

We carefully filtered and curated a selection of $1,001$ real-world diagrams from Krishnamurthy et al. (); Kembhavi et al. () to include in our test suite. These diagrams span a diverse range of domains, including ecology, biology, physics, astronomy, chemistry, and geology. This selection ensures that our test suite covers a broad spectrum of scientific disciplines, providing a comprehensive evaluation of the models’ capabilities. Detailed statistical information about these diagrams can be found in [Tab. 2](#S2.T2). Example questions are given in [Tab. 1](#S2.T1).

## 3 Do LVLMs Understand Diagrams?

In this section, we investigate whether LVLMs can understand entities ([§ 3.1](#S3.SS1)) and relations ([§ 3.2](#S3.SS2)) in synthetic diagrams. Additionally, we present the evaluation results on real diagrams ([§ 3.3](#S3.SS3)).

### 3.1 Can LVLMs Understand Entities?

We begin by evaluating whether LVLMs can identify and reason about entities represented by text boxes (i.e., text entities) or visual icons (i.e., visual entities). Additionally, we examine the models’ ability to correctly identify the spatial information (i.e., locations) of these entities in [§ E.2.1](#A5.SS2.SSS1).

##### Preparation.

We evaluate three LVLMs: GPT-4Vision (i.e., GPT-4V, OpenAI, )), GPT-4o (OpenAI, ), and Gemini 1.5 Pro (i.e., Gemini, Anil et al., )), where the evaluation takes place from June to September 2024. Details about model configurations are in [§ E.1](#A5.SS1).
Prompting templates and demonstration examples for various models are given in [Figs. 10](#A6.F10), [11](#A6.F11), [12](#A6.F12) and [13](#A6.F13) in [§ F.1.1](#A6.SS1.SSS1).

**Table 3: Performance of LVLMs on QA in terms of entities in text boxes or visual icons. LVLMs can always identify entities correctly, and can also reason about them effectively with CoT prompting.**
| Accuracy (%) | Text Entity | Visual Entity |  |  |
| --- | --- | --- | --- | --- |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ |  |
| GPT-4V (ZS / CoT) | 97.41 / 97.81 | 50.60 / 99.60 | 83.40 / 85.74 | 32.38 / 93.65 |
| GPT-4o (ZS / CoT) | 91.63 / 99.20 | 64.06 / 100.0 | 87.50 / 92.58 | 46.72 / 94.88 |
| Gemini (ZS / CoT) | 86.85 / 88.05 | 71.89 / 95.78 | 90.23 / 87.70 | 67.21 / 86.48 |
| Average (ZS / CoT) | 91.97 / 95.02 | 62.18 / 98.46 | 87.04 / 88.67 | 48.77 / 91.67 |

##### Results.

We evaluate LVLMs under two settings: zero-shot prompting (ZS) and the Chain-of-Thought prompting (CoT, Wei et al., ) as in [Tab. 3](#S3.T3). The results demonstrate that all LVLMs can easily recognize entities in both text boxes ($>95\%$ accuracy) and visual icons ($>88\%$ accuracy). The accuracies on
NR
questions, which assess entity recognition, remain consistently high. For the
NC
questions, which evaluate reasoning ability, we find that LVLMs can answer almost perfectly with CoT prompting, achieving $98.46\%$ accuracy for text entities and $91.67\%$ for visual entities.
Our findings on entity recognition and reasoning can be summarized as follows:

###### Observation 1 (Ability to understand entity) .

LVLMs can nearly perfectly identify entities in diagrams and demonstrate strong reasoning abilities regarding these entities.

This ability is fundamental to various vision tasks. Our observation aligns with existing research, confirming that models possess the basic capability to perform simple object detection and count objects to some extent. Next, we turn our focus to complex relations to determine whether LVLMs can comprehend the intricate interactions between entities.

###### Observation 1 (Ability to understand entity) .

### 3.2 Can LVLMs Understand Relations?

We categorize relations into two types for our research: implicit relations (e.g., relative positions of entities) and explicit relations (e.g., arrows or segments).

##### Preparation.

We generate synthetic diagrams following previous settings ([§ 3.1](#S3.SS1)).
To reduce errors contributed by entity understanding, here we represent entities by text, which yields the best performance on corresponding
NR
and
NC
questions ([Tab. 3](#S3.T3)).
Example questions are in [Tab. 1](#S2.T1). The prompting templates and demonstration examples are in [Figs. 16](#A6.F16), [17](#A6.F17), [18](#A6.F18) and [19](#A6.F19) in [§ F.1.2](#A6.SS1.SSS2).

**Table 4: Performance of LVLMs on QA for relations. LVLMs struggle to identify both implicit and explicit relations and are unable to reason about them effectively.**
| Accuracy (%) | Implicit Relation | Explicit Relation |  |  |
| --- | --- | --- | --- | --- |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ |  |
| GPT-4V (ZS / CoT) | 75.89 / 72.33 | 30.36 / 34.41 | 57.59 / 61.60 | 49.62 / 59.51 |
| GPT-4o (ZS / CoT) | 72.53 / 77.27 | 37.04 / 55.26 | 61.81 / 76.58 | 57.60 / 70.15 |
| Gemini (ZS / CoT) | 58.50 / 60.87 | 30.36 / 31.78 | 60.97 / 68.52 | 69.58 / 70.15 |
| Average (ZS / CoT) | 68.97 / 70.16 | 32.59 / 40.49 | 60.13 / 68.00 | 58.94 / 66.60 |

##### Results.

[Tab. 4](#S3.T4) presents the accuracies for relation questions.
The results indicate that all models generally struggle with relation recognition (
NR
questions), even when using CoT prompting, which leads to an average accuracy of around $70\%$. Similarly, the models also have difficulty reasoning about relations (
NC
questions), with the average accuracy ranging from $40\%$ to $66\%$.
Notably, GPT-4V’s performance on counting implicit spatial relations (i.e., relative positions) is nearly equivalent to random guessing ($34.41\%$), and it shows significant difficulty in recognizing or reasoning about explicit relations, even under the CoT prompting setting.

##### Consistency Verification.

Before proceeding, we validate our findings on relations to ensure their reliability. We adjust the diagram generation settings (e.g., arrow features) and observe that the results remained consistent ([§ E.2.2](#A5.SS2.SSS2)). Beyond diagram variations, we also test the robustness of our results by examining the consistency of LVLMs’ performance with different prompting templates, specifically under the in-context learning (ICL) setting. The findings indicate that ICL does not improve the models’ ability to identify or reason about relations ([§ E.2.3](#A5.SS2.SSS3)). These results further confirm the reliability of our conclusions. Thus, we can summarize our observation as follows:

###### Observation 2 (Ability to understand relations) .

LVLMs can barely identify both implicit and explicit relations, and they are unable to reason about them, even with CoT prompting.

This observation contradicts the remarkable success that LVLMs have demonstrated in understanding complex diagrams. To further investigate their failures, we evaluate them on real diagrams to determine whether they can effectively comprehend more complex, real-world scenarios.

###### Observation 2 (Ability to understand relations) .

### 3.3 Do LVLMs Understand Real Diagrams?

Synthetic diagrams are used to evaluate the basic abilities of models. Next, we move to real diagrams to double-check how these well-trained models perform in practical diagram understanding.

##### Preparation.

The evaluation follows the similar settings of that on synthetic diagrams. We first provide the performance on
KR
questions for both entities and relations in real diagrams.
Example questions are given above ([Tab. 1](#S2.T1)), and the question design for diagrams in each domain is in [§ D.1](#A4.SS1).
Prompt templates and examples can be found in [Figs. 26](#A6.F26), [27](#A6.F27), [28](#A6.F28) and [29](#A6.F29) in [§ F.2](#A6.SS2).

**Table 5: Performance of LVLMs on KR questions for real diagrams. Results indicate that models continue to recognize and reason about entities effectively, and they struggle with reasoning about relations. However, surprisingly, they are able to recognize relations in real diagrams.**
| Accuracy (%) | Entity | Relation |  |  |
| --- | --- | --- | --- | --- |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ |  |
| GPT-4V (ZS / CoT) | 84.30 / 88.89 | 49.41 / 78.77 | 77.48 / 78.68 | 42.10 / 59.89 |
| GPT-4o (ZS / CoT) | 87.90 / 93.10 | 61.40 / 82.29 | 81.60 / 84.10 | 50.40 / 72.89 |
| Gemini (ZS / CoT) | 87.29 / 84.99 | 53.49 / 68.35 | 80.90 / 80.50 | 51.39 / 57.69 |
| Average (ZS / CoT) | 86.49 / 88.99 | 54.77 / 76.47 | 79.99 / 81.09 | 47.96 / 63.49 |

##### Results.

[Tab. 5](#S3.T5) presents the performance of LVLMs on real diagrams.
We observe that models continue to perform well in recognizing and reasoning about entities, with GPT-4o achieving $93.10\%$ accuracy in recognition and $82.29\%$ accuracy in reasoning. Besides, models still struggle to reason about relations in real diagrams, similar to their performance on synthetic diagrams. Notably, though, we find that models can recognize relations quite well in real diagrams, with an average accuracy of $81.09\%$, compared to only $68.00\%$ on synthetic diagrams ([Tab. 4](#S3.T4)).
From these results, we can obtain the following observation:

###### Observation 3 (Performance on real diagrams) .

LVLMs struggle to recognize relations in simple synthetic diagrams, yet they can effectively recognize relations in complex real diagrams.

We substantiate that LVLMs cannot understand relations in synthetic diagrams, yet this finding reveals a contradiction. While the models do not inherently possess the ability to recognize relations, they are able to do so in real diagrams. This outcome contradicts our initial intuition ([Intuition 1](#Thmintuition1)). Therefore, we further investigate these counterintuitive findings by examining the key difference between synthetic and real diagrams: the role of background knowledge.

###### Observation 3 (Performance on real diagrams) .

## 4 Knowledge As Shortcuts: Quantitative Analysis

Given that rich knowledge has been encoded into LVLMs during various training stages, a possible hypothesis emerges: these models may not truly understand diagrams but instead rely on their stored knowledge as shortcuts to provide answers. In this section, we explore the impact of knowledge on the models’ ability to answer questions.

### 4.1 Knowledge Grounding of Diagrams Improves Relation Recognition

To determine the effect of knowledge, we first compare the models’ QA performance on diagrams with and without embedded knowledge. Our focus is on the explicit relations in synthetic diagrams. We construct relations that incorporate semantic knowledge to simulate practical conditions where models might use this knowledge as shortcuts. If our hypothesis is correct, we would expect to see an increase in accuracy on QA tasks for these specially constructed diagrams.

##### Preparation.

To construct diagrams grounded with semantic knowledge, we generate relations with real meaning behind them. Specifically, for each entity, we get its Word2Vec embedding (Mikolov et al., ) based on the text attribute, and use cosine similarity implemented by spaCy (Honnibal et al., ). If the similarity between entity text is larger than $0.5$, we regard that there exists a relation. We construct a semantic graph on all the entities from Lu et al. () with $377$ entities and $1901$ relations. We randomly generate diagrams following the settings mentioned in [§ 2.3](#S2.SS3.SSS0.Px1), but we constrain the generated diagram as the subgraph in the constructed semantic graph.(^3^33Under such strict constraints, we can only generate $885$ (instead of $1,000$) distinct diagrams for evaluation.) See [Figs. 24](#A6.F24) and [25](#A6.F25) in [§ F.1.5](#A6.SS1.SSS5) for the prompting templates and demonstration examples.

**Table 6: Performance of LVLMs on synthetic diagrams without and with semantic knowledge (denoted as “Vanilla” and “Knowledge-Grounded”). The annotated accuracy change reveals that models can better identify relations in knowledge-grounded diagrams while still struggling to reason about them effectively.**
| Accuracy (%) | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}})$ |  |  |
| --- | --- | --- | --- | --- |
| Diagram Type | Vanilla | Knowledge-Grounded | Vanilla | Knowledge-Grounded |
| GPT-4V (ZS / CoT) | 57.59 / 61.60 | $74.17_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{16.58}\uparrow}$ / $74.39_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{12.79}\uparrow}$ | 49.62 / 59.51 | $55.79_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{6.17}\uparrow}$ / $57.87_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{1.64}\downarrow}$ |
| GPT-4o (ZS / CoT) | 61.81 / 76.58 | $77.92_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{16.11}\uparrow}$ / $82.78_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{6.20}\uparrow}$ | 57.60 / 70.15 | $60.65_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{3.05}\uparrow}$ / $72.92_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{2.77}\uparrow}$ |
| Gemini (ZS / CoT) | 60.97 / 68.52 | $72.19_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{11.22}\uparrow}$ / $72.19_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{6.37}\uparrow}$ | 69.58 / 70.15 | $68.06_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{1.52}\downarrow}$ / $64.81_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{5.34}\downarrow}$ |
| Average (ZS / CoT) | 60.13 / 68.00 | $75.57_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{15.44}\uparrow}$ / $\definecolor{dynamiccolor}{RGB}{169, 0, 85}\hbox{\pagecolor{dynamiccolor!40}76.45}_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{8.45}\uparrow}$ | 58.94 / 66.60 | $61.50_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{2.56}\uparrow}$ / $\definecolor{dynamiccolor}{RGB}{128, 0, 126}\hbox{\pagecolor{dynamiccolor!40}65.20}_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{1.40}\downarrow}$ |

##### Results.

[Tab. 6](#S4.T6) presents the evaluation accuracies on diagrams without and with semantic knowledge. For comparison, we include the performance changes relative to the original results shown in [Tab. 4](#S3.T4).
Overall, models exhibit improved relation recognition in diagrams containing semantic knowledge. The average accuracy improvement for
NR
questions is $15.44\%$ with zero-shot (ZS) prompting and $8.45\%$ with CoT prompting. However, for
NC
questions, the performance remains largely unchanged.
The findings are consistent with our hypothesis: models utilize knowledge in diagrams as relation recognition shortcuts.
With these results, we have below observation:

###### Observation 4 (Knowledge in diagrams helps) .

LVLMs are more effective at recognizing relations in diagrams that relate to some background knowledge, as they can use it as a shortcut.

This observation indicates that even when questions do not explicitly require background knowledge to answer them, the presence of knowledge in the diagram can still enhance the performance of relation recognition. Next, we investigate whether questions that require models to actively use knowledge could further improve their performance.

###### Observation 4 (Knowledge in diagrams helps) .

### 4.2 Relation Recognition Questions Requiring Knowledge Show Improvements

We then evaluate the models using questions that do not require background knowledge (
KF
questions) and compare their performance with questions that do require such knowledge (
KR
questions).

**Table 7: The results indicate that models perform similarly on entity questions, regardless of whether or not they require background knowledge.**
| Accuracy (%) | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ |
| --- | --- | --- | --- | --- |
| GPT-4V (ZS / CoT) | 91.90 / 91.50 | $84.30_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{7.60}\downarrow}$ / $88.89_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{2.61}\downarrow}$ | 38.59 / 75.24 | $49.41_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{10.81}\uparrow}$ / $78.77_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{3.53}\uparrow}$ |
| GPT-4o (ZS / CoT) | 88.30 / 93.60 | $87.90_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{0.40}\downarrow}$ / $93.10_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{0.51}\downarrow}$ | 45.89 / 79.05 | $61.40_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{15.51}\uparrow}$ / $82.29_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{3.24}\uparrow}$ |
| Gemini (ZS / CoT) | 88.10 / 82.00 | $87.29_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{0.81}\downarrow}$ / $84.99_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{2.99}\uparrow}$ | 58.38 / 75.46 | $53.49_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{4.90}\downarrow}$ / $68.35_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{7.12}\downarrow}$ |
| Average (ZS / CoT) | 89.43 / 89.03 | $86.49_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{2.94}\downarrow}$ / $\definecolor{dynamiccolor}{RGB}{214, 0, 40}\hbox{\pagecolor{dynamiccolor!40}88.99}_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{0.04}\downarrow}$ | 47.62 / 76.58 | $54.77_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{7.14}\uparrow}$ / $\definecolor{dynamiccolor}{RGB}{169, 0, 85}\hbox{\pagecolor{dynamiccolor!40}76.47}_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{0.11}\downarrow}$ |

##### Evaluation on entity questions

We follow the same settings as in previous experiments. [Tab. 7](#S4.T7) illustrates the role of knowledge in questions related to entities. The results show that the gap between
KF
and
KR
questions for entities in real diagrams is negligible. Specifically, the average accuracy decrease in recognition is only $0.04\%$ under the CoT setting, while the decrease in reasoning is similarly minimal at $0.11\%$.

**Table 8: Model performance for questions on relations in real diagrams. The accuracy gap suggests that models perform better on NR questions when these questions require knowledge to answer. However, there is no significant improvement in accuracy when it comes to NC questions.**
| Accuracy (%) | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ |
| --- | --- | --- | --- | --- |
| GPT-4V (ZS / CoT) | 66.99 / 69.69 | $77.48_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{10.49}\uparrow}$ / $78.68_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{8.99}\uparrow}$ | 38.30 / 57.09 | $42.10_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{3.81}\uparrow}$ / $59.89_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{2.80}\uparrow}$ |
| GPT-4o (ZS / CoT) | 69.20 / 73.90 | $81.60_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{12.39}\uparrow}$ / $84.10_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{10.20}\uparrow}$ | 49.97 / 65.00 | $50.40_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{0.43}\uparrow}$ / $72.89_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{7.89}\uparrow}$ |
| Gemini (ZS / CoT) | 68.00 / 66.30 | $80.90_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{12.90}\uparrow}$ / $80.50_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{14.20}\uparrow}$ | 53.39 / 61.18 | $51.39_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{2.00}\downarrow}$ / $57.69_{\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}{3.49}\downarrow}$ |
| Average (ZS / CoT) | 68.07 / 69.96 | $79.99_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{11.93}\uparrow}$ / $\definecolor{dynamiccolor}{RGB}{186, 0, 68}\hbox{\pagecolor{dynamiccolor!40}81.09}_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{11.13}\uparrow}$ | 47.22 / 61.09 | $47.96_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{0.75}\uparrow}$ / $\definecolor{dynamiccolor}{RGB}{121, 0, 133}\hbox{\pagecolor{dynamiccolor!40}63.49}_{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{2.40}\uparrow}$ |

##### Evaluation on relation questions.

Similarly, [Tab. 8](#S4.T8) illustrates the impact of knowledge on questions related to relations. The results show that when questions require knowledge, models are better at recognizing relations. However, their reasoning ability remains largely unchanged. Specifically, the average accuracy gap in recognition is $11.13\%$ under the CoT setting, while the gap in reasoning is only $2.4\%$.
Thus, we have the observation below:

###### Observation 5 (Knowledge shortcuts help) .

LVLMs are better at recognizing relations in real diagrams when the question requires them to use background knowledge. However, whether or not a question requires knowledge does not significantly impact the models’ performance in entity recognition, entity reasoning, or reasoning about relations.

These two observations ([Observations 4](#Thmobservation4) and [5](#Thmobservation5)) lead to conclusions that are entirely contrary to our initial intuition ([Intuition 1](#Thmintuition1)). In this section, we use quantitative analysis to support our hypothesis that knowledge acts as a shortcut for recognizing relations. In the next section, we provide qualitative analysis to further substantiate the validity of this hypothesis.

###### Observation 5 (Knowledge shortcuts help) .

## 5 Knowledge Shortcuts: Qualitative Analysis

We provide further evidence to confirm that LVLMs can only recognize and reason about entities but not relations in real diagrams ([§ 5.1](#S5.SS1)). The ability to recognize relations in real diagrams appears to be an illusion driven by knowledge shortcuts rather than genuine understanding ([§ 5.2](#S5.SS2)).

### 5.1 LVLMs Cannot Recognize and Reason About Relations in Real Diagrams

##### Preparation.

We use the number of entities in a diagram as an indicator of its complexity, with the answers to ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ providing the entity count, as introduced in [Tab. 1](#S2.T1). We then divide all real diagrams into five bins based on their entity count, ensuring that each bin contains more than $100$ diagrams (detailed statistics are provided in [Fig. 8](#A5.F8)). For each subset of diagrams, we report the average accuracy for all questions under the CoT setting.

Figure: (a) Accuracies on entity questions

##### Results.

[Fig. 2](#S5.F2) present the average accuracy of three models on all questions (detailed results are in [Fig. 9](#A5.F9) in [§ E.3](#A5.SS3)). Overall, the performance on entity questions remains consistent across different levels of diagram complexity, while there is a noticeable decline in accuracy for relation questions as diagram complexity increases. These results further support that LVLMs can understand entities but struggle with understanding relations. Additionally, they suggest that the models’ apparent success in recognizing and reasoning about relations in real diagrams is largely due to their performance on simpler diagrams, rather than a true comprehension of complex relational structures.

### 5.2 LVLMs Hallucinate Relations in Real Diagrams (Case Study)

Next, we provide an intuitive and vivid case study to demonstrate that LVLMs hallucinate when interpreting relations in diagrams. Using the example diagram from [Fig. 1](#S1.F1), we construct two special cases for comparison: in the first case, we remove all relations from the diagram, and in the second case, we replace the relations with random ones. The goal is to test the GPT-4o’s response when there is either no relational information or when the relations presented conflict with background knowledge. For evaluation, we pose an annotated question (${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$) and a complex reasoning question involving food chain counting.

Figure: Figure 3: The model response on the example diagram and its variants. Results suggest that the model relies on background knowledge as a shortcut rather than accurately recognizing and reasoning about relations.
Refer to caption: /html/2410.00193/assets/images/casestudy/image.png

In [Fig. 3](#S5.F3), when comparing the responses in the left subfigure (vanilla) with those in the middle subfigure (w/o relation), we observe that even in the absence of explicit relational information, the model still identifies the correct predator. Additionally, for the food chain counting question, the model continues to provide the original answers. This indicates that the model has pre-existing knowledge and it can use the knowledge as a shortcut for answering questions.
Similarly, when comparing the responses in the left subfigure (vanilla) with those in the right subfigure (with random relations), we find that the model provides the same answers despite the introduction of random relations. The new correct predator could be “A)” or “C)”, and the new correct food chain count is $4$. This demonstrates that the model relies on learned knowledge rather than parsing the diagram itself.
With this additional evidence, we can reasonably conclude the following finding:

Finding: Current LVLMs can recognize and reason about entities in diagrams but struggle with understanding relations. However, they manage to answer diagram-related complex questions by identifying entities and leveraging relevant learned knowledge as a shortcut.

## 6 Conclusion

We evaluate three LVLMs on diagram understanding using our test suite, including synthetic and real diagrams. Our findings reveal that while these models can perfectly recognize and reason about entities depicted in the diagrams, they struggle with recognizing the depicted relations. Furthermore, we demonstrate that the models primarily rely on knowledge shortcuts when answering complex diagram reasoning questions. These results suggest that the apparent successes of LVLMs in diagram reasoning tasks create a misleading impression of their true diagram understanding capabilities.

## Appendix A Ethical Considerations

Our paper focuses on evaluation using synthetic and public datasets, and we thereby do not foresee any ethical issues originating from this work. The license of FoodWeb (Krishnamurthy et al., ) and AI2D (Kembhavi et al., ) is BSD-2-Clause and Apache-2.0 respectively.

##### Reproducibility

We have tried our best to ensure the reproducibility of our work.
The datasets and models that we use in this paper are publicly available. We present all the details about our evaluation and data including the annotation in the Appendix. At the same time, we have uploaded our evaluation code, generated synthetic diagram data, annotated real diagram data, as well as the responses of LVLMs in supplementary files. Every stage of our work ranging from code to results is introduced thoroughly and can be easily reproduced.

## Appendix B Limitations

Our work has two main limitations. First, we focus exclusively on diagrams that depict various entities and relationships. There may be other specialized types of diagrams that are not well-suited to this representation. We encourage future research to explore and analyze model performance on such diagrams.
Second, while we demonstrate that LVLMs have limited diagram understanding capabilities and that their strong performance is largely due to knowledge shortcuts, we do not offer insights on how to address this issue. Future work could focus on developing strategies to enhance LVLMs’ true diagram understanding abilities.

## Appendix C Related Work

##### Evaluation of LVLMs.

Various benchmarks have been introduced recently to evaluate the performance of multimodal models (e.g., LVLMs) focusing on different aspects like cognition and perception  (Fu et al., ), knowledge (Yu et al., ; Li et al., ; Yue et al., ), hallucination  (Li et al., ; Liu et al., ; Wang et al., ) and reasoning with images (Yue et al., ). These benchmarks include a wide range of images, some from natural image datasets  (Li et al., ; Wang et al., ) and others newly gathered or generated specifically for these studies (Liu et al., ; Fu et al., ; Yue et al., ; Zhang et al., ). The benchmarks are used either by employing a generative task of image captioning, where the model is prompted to describe the image  (Fu et al., ; Wang et al., ), or by visual question answering (VQA), where the model is asked several questions related to the image (Yue et al., ; Lu et al., ; Zhang et al., ). These benchmarks evaluate model performances qualitatively and quantitatively to report on their strengths and weaknesses.

##### Diagram Question Answering (DQA).

Diagram question-answering is a specialized task within the umbrella of VQA, including abstract representations of real-world knowledge instead of natural images (Lu et al., ). These diagrams typically relay information using the graphic space, diagram constituents, and a set of relationships which is far more complex than standard natural images where object detection might be sufficient to reason about the image Kembhavi et al. ().

Prior research on the DQA problem has been diverse, focusing on various types of diagrams and questions. Some studies have addressed spatial reasoning challenges, by generating diagrams with basic shapes like in NLVR Suhr et al. () and ShapeWorld Kuhnle & Copestake () or abstract representations of real-life scenarios Antol et al. (); Zhang et al. (). Others have gathered text-book diagrams, such as FoodWebs Krishnamurthy et al. (), AI2D Kembhavi et al. () and TQA Kembhavi et al. (), containing diagrams from the grade-school science domain with more emphasis on relationships between diagram entities and complex concepts like procedures, taxonomies, and labeling of components. Although these datasets contain complex diagrams, most of the questions can be answered with commonsense knowledge, without visual reasoning. In contrast, IconQA Lu et al. () encompasses simple diagrams but requires a spatial and property-level understanding of the diagram.

##### Capabilities and Limitations of LVLMs.

A common point on the success of LVLMs is optimistic. The great performance on various benchmarks is interpreted as the strong understanding abilities of large models on diagrams, such as identification ability (Chen et al., ) and reasoning ability (Lu et al., ). Besides diagrams, large language models are found powerful to solve various complex question-answering tasks requiring strong reasoning ablities (Qin et al., ; Bang et al., ). Large visual models are also shown to perform reasoning about images pretty well (Gupta & Kembhavi, ; Chen et al., ).

Another line of work focuses on the limitations of LVLMs. People find that large models act like stochastic parrots which only mindlessly mimic humans (i.e., memorizing pertaining corpus without any understanding) for both text reasoning and visual reasoning (Yang et al., ; Zecevic et al., ; Hu et al., ; Mao et al., ).
Meanwhile, a rising number of evaluations on LVLMs exposing their limitations on more basic reasoning tasks such as counting (Fu et al., ) and perception (Yue et al., ). This line of work indicates that LVLMs perform diagram understanding via illusion, i.e., relying on the learned knowledge instead of real-time reasoning of the diagram.

## Appendix D Supplementary Test Suite Details

We provide additional details about our test suite here.

### D.1 Question Annotation Details

For questions on synthetic diagrams, we annotate them as described in [Tab. 1](#S2.T1). Regarding real diagrams, we introduce their annotation details concerning the $6$ domains respectively. For each domain, we choose a representative diagram to show how we annotate, as shown in [Fig. 4](#A4.F4).

For each question, we provide four options. We first annotate the correct option. The three incorrect options are sampled uniformly without replacement from the pool of correct answers for questions of the same type, excluding the current correct answer. Furthermore, we manually verify (and edit if necessary) the sampled negative options to ensure that they are not inadvertently correct answers. The four options are randomly shuffled before feeding into LVLMs.

Figure: Figure 4: The representative diagrams of 6 domains.
Refer to caption: /html/2410.00193/assets/x3.png

#### D.1.1 Ecology - Food Web; Food Chain

Food web diagrams illustrate predatory relationships among animals (and between animals and plants or detritus) in the same environment (e.g. prairie, forests, sea, etc.).
Any possible path from a plant to a top animal is a single food chain.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is in the diagram?”. For [Fig. 4](#A4.F4)a, the correct option is “Sparrow”.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many entities are in the diagram?”. For [Fig. 4](#A4.F4)a, the correct option is “$10$”.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

The annotation of the question type is determined by the number of producers in the diagram:
(a) If the diagram contains three or fewer producers, the question type is annotated as “Which producer is in the diagram?”.
(b) If the diagram includes more than three producers, the annotation changes to “Which producer is not in the diagram?”.
(c) Only in cases where the food web diagram contains no producers (i.e. in the detritus environment) do we annotate the question as “Which consumer is not in the diagram?”.
[Fig. 4](#A4.F4)a contains two producers, “Grass” and “Blueberry Bush”, which is applicable to category (a). The correct option we annotate is “Blueberry Bush”. The (a)(b)(c) question subtypes account for $85.71\%$, $10.60\%$, and $3.68\%$ respectively.
As the LVLM must comprehend the concepts of producer and consumer to correctly answer these questions, this question type is classified as requiring knowledge.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many distinct consumers are there in the food web?”, which represents a subset of ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$. For [Fig. 4](#A4.F4)a, the correct option is “$8$”.
Since the LVLM needs to comprehend the meaning of consumer, this question type is classified as requiring knowledge.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

Depending on the number of arrows linked to an entity, for an entity with numerous connections, the question is annotated as “Which entity is not connected to Entity?” For an entity with fewer connections, the question is annotated as “Which entity is connected to Entity?”. These two subtypes account for $48.05\%$ and $51.95\%$ respectively.
For [Fig. 4](#A4.F4)a, we ask “Which entity is connected to Shrew”, and the annotated correct option is “Great Horned Owl”.
This question type only involves recognizing the relations between an entity of interest and others, and thus no knowledge is required.

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many arrows are linked to Entity in the diagram?”. For each diagram, we randomly select an entity that has both an arrow pointing to it and an arrow pointing out of it as the entity of interest. For example, in [Fig. 4](#A4.F4)a, we annotate “How many arrows are linked to Skunk in the diagram?” with the correct option “$3$”
This question type only involves counting the relations (represented by arrows) between an entity of interest and other entities, and thus no knowledge is required.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

Corresponding to ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$, we translate “Which entity is not connected to Entity?” to “Which is not the predator of Entity?” or “Which is not the prey of Entity?”. Similarly, we translate “Which entity is connected to Entity?” to “Which is the predator of Entity?” or “Which is the prey of Entity?”. Entity of interest is the same as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$. These four question subtypes account for $3.98\%$, $38.94\%$, $30.75\%$, and $26.33\%$ respectively.
Refer to our annotation of [Fig. 4](#A4.F4)a ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$, for this question type, we annotate question as “Which is the predator of Shrew?” with the correct option “Great Horned Owl”.
Since the LVLM needs to comprehend the meaning of prey (arrow points to the entity) and predator (arrow points out of the entity), this question type requires knowledge.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many types of prey are consumed by Entity in the foodweb?” with the same entity of interest as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$. Accordingly, for [Fig. 4](#A4.F4)a, we annotate “How many types of prey are consumed by Skunk in the foodweb?” with the correct option “$2$”.
Similarly, since the LVLM needs to comprehend the meaning of prey, this question type requires knowledge.

#### D.1.2 Biology - Life Cycle

In contrast to [D.1.1](#A4.SS1.SSS1), this domain delineates the developmental morphology of a single species across its life stages (e.g., for insects in diagrams like [Fig. 4](#A4.F4)b, stages such as egg, pupa, and adult are depicted). These morphological stages are interconnected via arrows, forming a directed cyclic graph. Notably, those diagrams include arrows pointing from the mature form (adult) to its offspring (egg, embryo, or seed). We claim that the mature form represents the final stage of the species’ life cycle, while the offspring represents the first stage.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is in the diagram?”. For [Fig. 4](#A4.F4)b, the correct option is “Larva’.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many entities are in the diagram?”. For [Fig. 4](#A4.F4)b, the correct option is “$4$”.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is annotated as “What is the [first/last] life stage for the creature in the diagram?”, where we choose one of two words manually, resulting in $47.80\%$ for “first” and $52.20\%$ for “last”. For [Fig. 4](#A4.F4)b, we annotate “first” with the correct option “Egg”.
Determining the first and last stage of a lifecycle requires background knowledge in biology.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many life stages are after the stage Entity in the diagram?”. For [Fig. 4](#A4.F4)b, we replace Entity with “Larva” and the correct option is “$2$”
Since the LVLM needs to comprehend the meaning of life stage and the arrow direction, this question type requires knowledge.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is connected to the Entity?” For [Fig. 4](#A4.F4)b, we replace Entity with “Larva” and the correct option is “Egg”.

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many arrows are in the diagram?”. In [Fig. 4](#A4.F4)b, the correct option is “$4$”.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which stage is after the Entity stage in the diagram?”. For [Fig. 4](#A4.F4)b, we replace Entity with “Larva” and the correct option is “Pupa”.
This question type requires LVLMs to understand the meaning of stage and arrow directions, and thus knowledge is required.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many stages can the creature change in the diagram?” Same as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$, in [Fig. 4](#A4.F4)b, the correct option is “$4$”.
Similarly, since the LVLM needs to comprehend the meaning of stage and creature, this question type requires knowledge.
A few diagrams use the same lifecycle paradigm to describe multiple species, e.g. egg $\rightarrow$ tadpole/chick $\rightarrow$ frog/chicken. The correct answer to ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ in this case is 3 while that to ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ is 6.

#### D.1.3 Physics - Circuit

This domain contains simple middle-to-high-school circuit diagrams, typically containing only power, switches, wires, and a few appliances (e.g., light bulbs), and does not contain circuit diagrams for complex electronics that require knowledge beyond high school. Circuit diagrams can be abstracted as undirected cyclic graphs. Note that in this domain, we do not regard wires as entities even if there is explicit text in the diagram. Instead, we consider them to be the representations of relations.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is in the diagram?”. For [Fig. 4](#A4.F4)c, the correct option is “Bulb”.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many entities are in the diagram?”. For [Fig. 4](#A4.F4)c, the correct option is “$3$”.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is annotated as “Which electronic component is in the diagram?”. For [Fig. 4](#A4.F4)c, the correct option is “Battery”.
Knowledge is required to understand the meaning of electronic component.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many bulbs in the diagram will glow when the switch is closed?” In [Fig. 4](#A4.F4)c, the correct option is “$1$”
Since the LVLM needs to understand the function of switch as well as determine whether it is a closed circuit, this question type is classified as requiring knowledge.
All the diagrams contain switches, but some do not contain light bulbs or they are intentionally short-circuited. In such cases, the correct answer is 0 0.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is connected to the Entity by the line?”. For [Fig. 4](#A4.F4)c, we replace Entity with “Bulb” and the correct option is “Battery”.

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many line segments are in the diagram?”. For [Fig. 4](#A4.F4)c, the correct option is “$3$”.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which electronic component is connected to the Entity by the wire?” Same as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$, in [Fig. 4](#A4.F4)c, we replace Entity with “Bulb”and the correct option is “Battery”.
This question type requires LVLMs to understand the meaning of electronic component and wire, and thus knowledge is required.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many wires are in the diagram?”. Same as ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$, in [Fig. 4](#A4.F4)c, the correct option is “$3$”.
Similarly, since the LVLM needs to comprehend the meaning of wire, this question type requires knowledge.

#### D.1.4 Astronomy - Solar System; Satellite System

The subject of the diagrams in this domain encompasses seasonal changes caused by the Earth’s revolution around the Sun, moon phase changes caused by the rotation of satellites around the planets, and planetary revolutions in the solar system. Diagrams describing phase changes can be regarded as directed cyclic graphs. A diagram depicting the solar system uses relative positions to express the relation between astronomical objects without using arrows.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is in the diagram?”. For [Fig. 4](#A4.F4)d, the correct option is “Sun”.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many entities are in the diagram?”. For [Fig. 4](#A4.F4)d, the correct option is “$5$”.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is annotated as “Which astronomical object is in the diagram?”. For [Fig. 4](#A4.F4)d, the correct option is “Sun”.
Knowledge is required to understand the meaning of astronomical object.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many planets or satellites are in the diagram?” For [Fig. 4](#A4.F4)d, the correct option is “$4$”.
Since the LVLM needs to understand the meaning of planets and satellites, this question type requires background knowledge.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is connected to the Entity in the diagram?”. For [Fig. 4](#A4.F4)d, we replace Entity with “September” and the correct option is “December”.

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many arrows are in the diagram?”. For [Fig. 4](#A4.F4)d, the correct option is “$5$”.
Some of the diagrams rely on relative positions rather than arrows to represent relations between entities, in which case the answer is 0 0.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “What is the next phase after the Entity in the diagram?”. For [Fig. 4](#A4.F4)d, we replace Entity with “Earth in Summer” and the correct option is “Earth in Fall”
This question type requires LVLMs to understand the meaning of phase, and thus knowledge is required.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many times that the planets or satellites can change in the diagram?”. For [Fig. 4](#A4.F4)d, the correct option is “$4$”.
Similarly, since the LVLM needs to comprehend the meaning of planets and satellites, this question type requires knowledge.

#### D.1.5 Chemistry - Water Cycle; Carbon Cycle

This domain includes topics such as the water cycle and the carbon cycle where various plants and animals participate, and photosynthesis and transpiration of a specific plant. Water or carbon enjoy multiple pathways to transfer between two phases, so that substantial diagrams can be viewed as directed multigraphs. Other diagrams that describe a single cyclic pathway (such as photosynthesis in a single plant) can be viewed as directed cyclic graphs.

For example, the visual entities in the [Fig. 4](#A4.F4)e are Sun, House Emissions, Carbon Dioxide, Cow, Ground (Soil), Tree, Worm, and Cloud. Among these, Smoke, Cow, Ground, Tree, and Worm are actively involved in the depicted carbon cycle. These entities are called cycle stages. Entities such as soil, lakes, and forests are, as per image segmentation principles, considered as single entities despite their spatial extent.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is in the diagram?”. For [Fig. 4](#A4.F4)e, the correct option is “Cow”.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many entities are in the diagram?”. As mentioned in the domain summary, there are “$8$” visual entities in [Fig. 4](#A4.F4)e.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is annotated as “Which cycle stage is described in the diagram?”. For [Fig. 4](#A4.F4)e, the correct option is “Animal”.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many different cycle stages are in the diagram?” As mentioned before, there are “$5$” cycle stages in [Fig. 4](#A4.F4)e.
Since the LVLM needs to understand the function of cycle stages, this question type requires knowledge.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which entity is connected to the Entity by the arrow?”. For [Fig. 4](#A4.F4)e, we replace Entity with “Cow” and the correct option is “Carbon Dioxide”

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many arrows are in the diagram?”. In [Fig. 4](#A4.F4)e, there are “$1$” arrows.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which cycle stage will happen after the Entity in the diagram?”. For [Fig. 4](#A4.F4)e, we replace Entity with “Animal” and the correct option is “Carbon Dioxide”
This question type requires LVLMs to understand the meaning of cycle stage, and thus knowledge is required.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many different processes of transitions are in the diagram?” For [Fig. 4](#A4.F4)c, the correct option is “$5$”.
Similarly, since the LVLM needs to comprehend the meaning of processes of transitions, this question type requires knowledge.

#### D.1.6 Geology - Planet Structure; Star Structure

Diagrams in this domain depict the geological structure of the Earth or other astronomical objects, showing the relationship between geological strata through their relative positions (typically nested structures) rather than explicit arrows.

##### Q ​ ( V | KF , NR ) 𝑄 conditional 𝑉 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which layer is included in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is “Crust”.

##### Q ​ ( V | KF , NC ) 𝑄 conditional 𝑉 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many layers are in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is “$4$”.

##### Q ​ ( V | KR , NR ) 𝑄 conditional 𝑉 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is annotated as “Which stratification that is outside the core is included in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is ‘Crust”
Understanding the meaning of stratification and determining the containing relations between stratigraphic layers requires the use of background knowledge.

##### Q ​ ( V | KR , NC ) 𝑄 conditional 𝑉 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many stratifications are outside the core in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is “$2$” (“Mantle” and “Crust”). Same as above, answering this question also requires background knowledge.

##### Q ​ ( E | KF , NR ) 𝑄 conditional 𝐸 KF NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which layer is next to the Layer in the diagram?”. For [Fig. 4](#A4.F4)f, we replace Layer with “Crust” and the correct option is “Mantle”.
This question type only involves recognizing the adjacency between a layer of interest and others, and thus no knowledge is required.

##### Q ​ ( E | KF , NC ) 𝑄 conditional 𝐸 KF NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many layer boundaries are in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is “$3$”.
We regard the boundary between layers as the representation of relation. This question only involves counting the number of boundaries, so no background knowledge is required.

##### Q ​ ( E | KR , NR ) 𝑄 conditional 𝐸 KR NR {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}) .

This question type is consistently annotated as “Which stratification is the next outside layer of the Layer in the diagram?”. For [Fig. 4](#A4.F4)f, we replace Layer with “Mantle” and the correct option is “Crust”.
This question type requires LVLMs to understand the meaning of stratification and the containing relations between layers, hence knowledge is required.

##### Q ​ ( E | KR , NC ) 𝑄 conditional 𝐸 KR NC {\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KR}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}}) .

This question type is consistently annotated as “How many transition zones of the structure are in the diagram?”. For [Fig. 4](#A4.F4)f, the correct option is “$3$”.
Similarly, since the LVLM needs to comprehend the meaning of transition zones, this question type requires knowledge.

## Appendix E Supplementary Results

We introduce the details of the models we evaluate and provide additional results in this section.

### E.1 Model Configurations

Generally, we spend around $800\$$ for all experiments.
The LVLM models are provided with the system message: “You are a visual assistant answering multiple choice questions about diagrams. Read the question, inspect the diagram, and answer with the correct choice in the following format: ”A) 0”.”

##### GPT-4V.

Model is used with the key gpt-4-vision-preview with the OpenAI API, Chat Completions. The temperature parameter is set to 0 to ensure deterministic outputs and a seed is given to the model to help with reproducibility. The max_tokens is limited to 600.

##### GPT-4o.

Model is used with the key gpt-4o with the OpenAI API, Chat Completions. The temperature parameter is set to 0 to ensure deterministic outputs and a seed is given to the model to help with reproducibility. The max_tokens is limited to 600.

##### Gemini.

Model is used with the key gemini-1.5-pro with the VertexAI API, Generative Models. The temperature parameter is set to 0 to ensure deterministic outputs. The max_output_tokens is limited to 600.

### E.2 Synthetic Diagram

The additional results on the synthetic diagrams are given in this subsection.

#### E.2.1 Entity Position and Spatial Relation

**Table 9: The template and example of questions for the evaluation of entity position and spatial relation in synthetic diagrams. The text with underline (e.g., top row) is specific and varies across diagrams.**
| Synthetic Diagram | Question | Question Example (with Answer Options) |
| --- | --- | --- |
| Entity Position | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | Which one of the text labels exists in the top row of the diagram? |
| ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ | How many text labels are there in the top row of the diagram? |  |

##### Preparation.

We generate synthetic diagrams similar to the previous settings ([Tab. 1](#S2.T1)).
We introduce a grid structure to describe the absolute positions of entities. Specifically, we use a $3\times 3$ grid and gridlines to define the compartments in the canvas. The entities are placed in the center of them with generated arrows connecting them.
We represent the entity via text.
As depicted in the example in [Tab. 9](#A5.T9), we use “top/center/bottom row/column” to describe the entity location.

**Table 10: Performance of LVLMs on entity position QA. LVLMs can capture part of the position information and struggle with entity identification and reasoning.**
| Accuracy (%) | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$ | ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NC}$}}})$ |
| --- | --- | --- |
| GPT-4V (ZS/CoT) | 75.25 / 77.67 | 41.75 / 64.02 |
| GPT-4o (ZS/CoT) | 78.07 / 89.54 | 50.30 / 79.52 |
| Gemini (ZS/CoT) | 63.58 / 64.79 | 66.80 / 73.96 |
| Average (ZS/CoT) | 72.30 / 77.33 | 52.95 / 72.50 |

##### Results.

LVLMs start to struggle with identifying the entity’s position attribute ([Tab. 10](#A5.T10)). Even with CoT prompting, the average score of
NR
questions is $77.33\%$, which is worse compared to the entity text recognition accuracy in [Tab. 3](#S3.T3) (i.e., $95.02\%$). Since LVLMs only identify the position partially, the average accuracy on
NC
questions (i.e., $72.50\%$ with CoT prompting) is also worse than that of text entity (i.e., $98.46\%$), where Gemini performs much worse than the other two models.

Figure: (a) ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}_{{\scalebox{0.8}{\mbox{$\displaystyle S$}}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$

##### Analysis.

We further analyze the results for more insights by visualizing the performance of LVLMs with entities in different compartments. Results ([Fig. 5](#A5.F5)) show that LVLMs can answer
NR
and
NC
questions much better if the entities are not in the center area, where this phenomenon is more obvious for two GPT models.

#### E.2.2 Consistency: Diagram Variation

##### Preparation.

We change the relation attributes, i.e., the arrow features in synthetic diagrams, to see if LVLMs can understand relations better.
Specifically, we randomly change the arrowhead size to its $1.5$, or $2$ times, change the line width to its $0.5$, $2$, or $4$ times, and the arrow color to black, red, or blue. Other settings remain the same as in [§ 2.3](#S2.SS3.SSS0.Px1). Then, we ask the same questions on these newly generated diagrams to observe how performance changes. For simplicity, we only consider the CoT prompting setting since it achieves better results.
The prompting templates as well as demonstration examples are shown in [Figs. 20](#A6.F20) and [21](#A6.F21).

Figure: (a) ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}},[{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}}])$

##### Results.

We visualize the results in [Fig. 6](#A5.F6) comparing with the results in [Tab. 4](#S3.T4). We find that changing the relation attributes does not yet improve the accuracy of QA for both
NR
questions and
NC
questions. The maximum improvement is only $1\%$, while the average accuracies remain roughly the same ($1.74\%$ lower for
NR
questions and $0.3\%$ higher for
NC
questions).
These results further support that our findings on relations are valid and that LVLMs indeed do not understand relations even with different types of relations.

#### E.2.3 Consistency: Prompt Variation

##### Preparation.

We follow the settings in Li et al. () to construct ICL prompts. We randomly select $4$ examples and concatenate these diagrams as well as questions and answers as few-shot examples (represented by image). Then, we modify the template to adapt to the ICL examples under the setting of CoT prompting for evaluation.
See [Figs. 22](#A6.F22) and [23](#A6.F23) for the prompting templates and demonstration examples.

Figure: (a) ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle E$}}},[{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},\textbf{{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}}}])$

##### Results.

We observe that ICL does not help with the relation identification ([Fig. 7(a)](#A5.F7.sf1)) and reasoning ([Fig. 7(b)](#A5.F7.sf2)) at all. Overall the average scores decrease, dropping $3\%$ for
NR
questions and dropping $6\%$ on
NC
questions. The findings further support that LVLMs can neither identify nor reason about relations even when provided a few examples with answers in context.

### E.3 Real Diagram

We provide the supplementary results for evaluations on real diagrams.

#### E.3.1 Diagram Distribution on Entity Number

Figure: Figure 8: The distribution of the number of diagrams with respect to the number of entities in them.

For our $1,001$ real diagrams, we annotate diagrams with the number of entities in them. Thus, we visualize the distribution in [Fig. 8](#A5.F8). We can find that it is similar to the long-tail distribution, and most diagrams have $3-10$ entities.

#### E.3.2 Supplementary Results for Statistical Analysis

Figure: (a) ${\scalebox{0.9}{\mbox{$\displaystyle Q$}}}({\scalebox{0.8}{\mbox{$\displaystyle V$}}}|{\scalebox{0.8}{\mbox{$\displaystyle\text{KF}$}}},{\scalebox{0.8}{\mbox{$\displaystyle\text{NR}$}}})$

We provide all the detailed accuracies for three models on QA with respect to the number of entities. Generally, all these three models have similar tendencies, and the accuracy tendencies are similar to their average as in [Fig. 2](#S5.F2).

## Appendix F Examples of Prompts and Responses

### F.1 Synthetic Diagrams

#### F.1.1 Entity

Prompting templates and demonstration examples for text entities and visual entities are shown in [Figs. 12](#A6.F12), [13](#A6.F13), [10](#A6.F10), [11](#A6.F11), [14](#A6.F14) and [15](#A6.F15).

#### F.1.2 Relation

Prompting templates and demonstration examples of explicit relations
$\displaystyle E$
(i.e., arrows) and implicit relations ${\scalebox{0.8}{\mbox{$\displaystyle E$}}}_{P}$ (i.e., spatial relations) are shown in [Figs. 16](#A6.F16), [17](#A6.F17), [18](#A6.F18) and [19](#A6.F19).

Figure: Figure 10: An existence question example for text entity.
Refer to caption: /html/2410.00193/assets/images/text/text_59.png

Figure: Figure 11: A count question example for text entity.
Refer to caption: /html/2410.00193/assets/images/text/text_39.png

Figure: Figure 12: An existence question example for the visual entity.
Refer to caption: /html/2410.00193/assets/images/icon/icon_73.png

Figure: Figure 13: A count question example for the visual entity.
Refer to caption: /html/2410.00193/assets/images/icon/icon_96.png

Figure: Figure 14: An existence question example to evaluate the position of entities.
Refer to caption: /html/2410.00193/assets/images/position/position_27.png

Figure: Figure 15: A count question example to evaluate the position of entities.
Refer to caption: /html/2410.00193/assets/images/position/position_12.png

Figure: Figure 16: An existence question example from the Spatial Relation subtask.
Refer to caption: /html/2410.00193/assets/images/spatial_rel/spatial_rel_58.png

Figure: Figure 17: A count question example from the Spatial Relation subtask.
Refer to caption: /html/2410.00193/assets/images/spatial_rel/spatial_rel_7.png

Figure: Figure 18: An existence question example from the Explicit Relation subtask.
Refer to caption: /html/2410.00193/assets/images/schematic_rel/schematic_rel_91.png

Figure: Figure 19: A count question example from the Explicit Relation subtask.
Refer to caption: /html/2410.00193/assets/images/schematic_rel/schematic_rel_2.png

#### F.1.3 Diagram Generation

Prompting templates and demonstration examples of synthetic diagrams with different arrow features are shown in [Figs. 20](#A6.F20) and [21](#A6.F21).

Figure: Figure 20: An existence question example from the Explicit Relation subtask for a different arrow feature.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_different/23.png

Figure: Figure 21: A count question example from the Explicit Relation subtask for a different arrow feature.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_different/5.png

#### F.1.4 In-Context Learning (ICL) Prompting Template

Four-shot ICL Prompting templates and demonstration examples are shown in [Figs. 22](#A6.F22) and [23](#A6.F23).

Figure: Figure 22: An existence question example from the Explicit Relation subtask with few-shot examples.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_icl/icl_4shots.jpg

Figure: Figure 23: A count question example from the Explicit Relation subtask with few-shot examples.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_icl/icl_4shots.jpg

#### F.1.5 Diagrams with Semantic Knowledge

Prompting templates and demonstration examples of synthetic diagrams containing semantic knowledge are shown in [Figs. 24](#A6.F24) and [25](#A6.F25).

Figure: Figure 24: An existence question example for a diagram generated from a knowledge graph.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_knowledge/162.png

Figure: Figure 25: A count question example for a diagram generated from a knowledge graph.
Refer to caption: /html/2410.00193/assets/images/schematic_rel_knowledge/84.png

### F.2 Real Diagrams

Prompting templates and demonstration examples of real diagrams (Astronomy - Solar System; Satellite System) for different question types are shown in [Figs. 26](#A6.F26), [27](#A6.F27), [28](#A6.F28), [29](#A6.F29), [30](#A6.F30), [31](#A6.F31), [32](#A6.F32) and [33](#A6.F33).

Figure: Figure 26: A knowledge-required entity recognition question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 27: A knowledge-required entity reasoning question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 28: A knowledge-free relation recognition question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 29: A knowledge-required relation reasoning question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 30: A knowledge-free entity recognition question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 31: A knowledge-free entity reasoning question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 32: A knowledge-free relation recognition question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

Figure: Figure 33: A knowledge-free relation reasoning question example for a real diagram.
Refer to caption: /html/2410.00193/assets/images/real_diagrams/astronomy_2757.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could it be possible to make the synthetic dataset public so that the reviewers have a better sense of its content?
2. The authors classified the diagram complexity based on the number of entities. Was there a reason they did not consider the number of relationship to measure complexity?
soundness: 3
strengths: The template of questions used in the paper is quite extensive. The authors also carefully evaluated each template separately showing a holistic evaluation framework. Specifically, the key observation noticed under Q1 seems interesting to me. The ability of VLM to understand and reason well about entities while struggling with relationships shows that relationships are hard to decode in general. The performance gap between real and synthetic datasets is also interesting.
summary: The paper demonstrates an in-depth evaluation if Vision Language models (VLM) can understand visual digram. The authors show the results both on their synthetically generated dataset as well as real visual diagrams curated from real datasets. They curate an extensive list of possible questions to evaluate VLM’s separately on each question category.
weaknesses: 1. The paper write-up could perhaps be revisited. It is difficult to read Table 6 the first time — the relative improvement could perhaps be presented more intuitively.
2. The motivation behind using ‘knowledge as a shortcut' in sections 4 and 5 was not clearly stated. Was there a chance for choosing to construct a separate knowledge graph of textual content rather than labeling the visual entities in the original diagram? Providing the rationale behind this construction would be interesting. Since ‘knowledge’ is a quite generic word, it might be useful to define more precisely what kind of knowledge they are referring to early in the paper. 

3. There are some relevant works from Chart, Graph, and 3D scene understanding that will be worth mentioning in the related works section:

        - ChartQA: https://arxiv.org/abs/2203.10244

        - Talk like a graph: Encoding graphs for large language models: https://research.google/blog/talk-like-a-graph-encoding-graphs-for-large-language-models/

        - CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning: https://openaccess.thecvf.com/content_cvpr_2017/html/Johnson_CLEVR_A_Diagnostic_CVPR_2017_paper.html

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - "Furthermore, we demonstrate that the models primarily rely on knowledge shortcuts when answering complex diagram reasoning questions." I am uneasy with the use of "primarily". The difference between KR and KF is at most ~15%. Saying "primarily" seems wrong to me. Saying knowledge is a shortcut LVLMs exploit seems reasonable, but could the authors justify why they said "primarily"?
soundness: 3
strengths: - The writing is very clear. As a reader, I feel that the hypotheses, experiment designs, and results are all clearly conveyed. I especially like how the authors layer experiments based on previous results to provide deeper and deeper insights.
- Experiments are overall well-designed. The tasks seem reasonable. The combination of synthetic and real diagrams is a strength that allows controllability and real-world validity. I like how the authors pay attention to details like varying the sizes and colors of arrows in the diagrams.
summary: This paper evaluates three LVLMs on diagram understanding. The authors curated a synthetic + real diagram dataset and investigated LLM performance on entity and relation recognition. Results suggest that LVLMs may not be truly understanding diagrams.
weaknesses: 1. Making counterfactual variations of diagrams and asking LVLMs about them is certainly interesting, but I think it is not surprising that this should degrade LVLM performance. When strong prior is present and the diagram contradicts that, the LVLM could simply get confused. In such cases I think it is important to test if explicitly including instructions to ignore prior knowledge and solely answer using the information in the diagram improves the performance. Take Section 5.2 as an example. When no link is present (middle panel), asking "How many food chains are there?" sounds more like the goal is to test relevant ecology knowledge instead of diagram reading, and I think the LVLM's decision to hallucinate connections is actually warranted. In other words, this case in particular feels like an unfair trick question to me.
2. This paper tested 3 LVLMs. Perhaps testing a few more would be helpful, e.g., Claude 3.5 Sonnet.
3. Even though I appreciate the inclusion of a real-world dataset, it is exclusively focused on science. A broader scope would be better.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: The main text does not state how the real-world diagram set was curated. Specifically, it would be interesting which filtering criteria were applied. Did you filter diagrams only by their domain? Did you apply a limit for the maximum number of nodes/relations/edge crossing?
soundness: 4
strengths: - The paper is written well, and the figures help the reader understand the main ideas. 
- The categorization of questions is intuitive and exemplified, and the test suite allows for further research in the direction of diagram understanding. An extensive appendix supports the main text by providing additional information about the curated test suite and the research methodology.
- The findings are novel and counter-intuitive. The semantic background leakage is well-induced and confirmed by extensive experiments. It highlights a surprising weakness of state-of-the-art LVLMs that should be considered when using them.
- The findings bear opportunities for future research. Semantic background leakage may also appear in other areas besides diagram understanding. Furthermore, it should be researched how LVLMs can be trained to be more robust and effective in diagram understanding.
summary: The presented work examines to what degree large vision-language models (LVLM) understand diagrams. A test suite is developed that includes diagram-understanding questions for a synthetic and real-world dataset. The questions are distributed into four categories: Every question is either a recognition or reasoning question, and every question is either knowledge-required or knowledge-free. The questions are either applied to entities in the diagram or to relations between them. While LVLMs can recognize and reason about entities in synthetic diagrams, they show poor performance for entities on real-world data. Surprisingly, their performance improves on more complex, real-world diagrams. The authors provide evidence that this performance leap originates from semantic background knowledge that the LVLM bears. In a case study, the authors show that the LVLMs hallucinate some answers due to their semantic background knowledge.
weaknesses: The case study (Section 5.2) could be more extensive, and its design includes some shortcomings: 
- The evaluation of the middle figure needs to be revised, as the correct answers are not included in the answer options. I propose to include an additional experiment, possibly in the appendix, with correct answer options. It would be highly interesting to observe if LVLMs even deviate from correct reasoning due to their inherent semantic background knowledge.
- Another important aspect that needs to be considered in the case study is the spatial positioning of the entities. To exclude the influence of spatial positionings on the observed hallucinations, I propose to rerun the case study with swapped spatial positions. Is the "fish" consistently predicted if swapped with the other entities?

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Additionally, beyond the overall evaluation results, I would like to see specific results across different domain and topics (as shown in Table 2). This could lead to new discoveries and spark more valuable discussions (involving different knowledge systems). 

The Chain of Thought approach mentioned by the authors is relatively simple, as it only involves adding prompts like “think Step-by-Step.” Are there any more effective ways to use Step-by-Step prompts for this kind of chart? I suggest that the authors explore this in greater depth. In the authors’ experiments, it is mentioned that under certain circumstance, the test of LVLMs do not use any knowledge shortcuts. However, for some simple relational charts, how can we be certain that these large models have not encountered similar charts during training? This point remains uncertain.

By the way, in this work, beyond the evaluations conducted by the authors, I would like to see a model proposed by the authors specifically for understanding basic relational charts.
soundness: 3
strengths: The overall contribution of the paper lies in the comprehensive nature of the experiments, including a relatively thorough consideration of chart understanding (both explicit and implicit) and other relevant aspects.
summary: This paper proposes an evaluation framework to assess LVLMs' capability in understanding diagrams. The results show that models like GPT-4V and Gemini rely primarily on their knowledge base rather than reasoning abilities when understanding relationships.
However, contributions of this paper are not particularly prominent. Is the core contribution the proposed test suite, the evaluation dataset, or the insights gained? In fact, some of these insights have already been revealed, such as “models consistently perform well on entity-related questions”, “models exhibit significant difficulty in identifying relationships”. I suggest that authors review the existing work and clearly highlight the differences between this paper and previous work, including the evaluation methods and insights gained.
Additionally, beyond the overall evaluation results, I would like to see specific results across different domain and topics (as shown in Table 2). This could lead to new discoveries and spark more valuable discussions (involving different knowledge systems). 

The Chain of Thought approach mentioned by the authors is relatively simple, as it only involves adding prompts like “think Step-by-Step.” Are there any more effective ways to use Step-by-Step prompts for this kind of chart? I suggest that the authors explore this in greater depth. In the authors’ experiments, it is mentioned that under certain circumstance, the test of LVLMs do not use any knowledge shortcuts. However, for some simple relational charts, how can we be certain that these large models have not encountered similar charts during training? This point remains uncertain.

By the way, in this work, beyond the evaluations conducted by the authors, I would like to see a model proposed by the authors specifically for understanding basic relational charts.
weaknesses: However, contributions of this paper are not particularly prominent. Is the core contribution the proposed test suite, the evaluation dataset, or the insights gained? In fact, some of these insights have already been revealed, such as “models consistently perform well on entity-related questions”, “models exhibit significant difficulty in identifying relationships”. I suggest that authors review the existing work and clearly highlight the differences between this paper and previous work, including the evaluation methods and insights gained.

Meta-review: The paper presents an evaluation of whether VLMs can understand visual diagrams. While VLMs recognize and reason about entities, they struggle with reasoning about relations in synthetic diagrams. Surprisingly, their performance on understanding relations improves on more complex, real-world diagrams. The authors provide evidence that this performance leap might not be due to visual understanding, but due to pre-trained knowledge baked into these language models 

Most Reviewers found the paper to be clearly written, experiments well-designed with useful insights. However, some concerns were raised about sensitivity to prompts. One of the suggestions was to conduct a more thorough investigation of prompt-engineering approaches on a dev-set, before proceeding to the whole corpus. During the reviewer discussion period, reviewers felt that the paper can benefit from advanced prompting techniques, or fine-tuning open-weights model to be considered a full paper. While proposing a new model maybe out of scope, additional experiments with open-weights model as well as fine-tuning on synthetic dataset would have significantly increased the contributions of the paper. It would help answer questions about how much data is needed to improve diagram understanding, and what kind of data is required. Such insights will be useful to the broader LLM/ VLM community.

Decision: Reject